What we have learned from building production Azure data platforms at enterprise scale — the patterns that work and the ones that cost you later.
What we have learned from enterprise Azure deployments
Building production Azure data platforms at enterprise scale is different from what the documentation describes. The architectural decisions that look correct in a whitepaper often have failure modes that only become visible under real workloads, with real security requirements, and real data volumes.
This is our honest account of what actually works — and what costs you later.
Get your landing zone right before you build anything else
The landing zone is the foundation: network topology, identity configuration, resource organisation, and governance policies. Every Azure data platform decision you make after this point is constrained by the landing zone you establish at the start.
The most expensive mistake we see is starting with an ad-hoc resource deployment and retrofitting governance later. Access controls, private endpoint configurations, network segmentation between production and development — these are orders of magnitude harder to add retroactively than to build correctly from day one.
The Azure Cloud Adoption Framework landing zone accelerators are a reasonable starting point, but they are a starting point — not a finished product. Review the default security policies against your organisation's actual compliance requirements before you deploy a single resource.
The medallion architecture is correct — apply it rigorously
Bronze to Silver to Gold is not just a naming convention. It is a data quality contract. Bronze contains raw, unmodified source data. Silver contains cleansed, conformed, and integrated data. Gold contains business-ready aggregates for specific analytical purposes.
The failure mode we see most often: teams that merge Silver and Gold because they will just clean it in the reporting layer. This creates dashboards that contain business logic embedded in DAX or Tableau calculated fields that nobody documents and everybody depends on. When the logic needs to change, there is no single place to change it.
Enforce the medallion separation in your storage account hierarchy, in your Delta Lake schemas, and in your access controls. Data in Bronze should be immutable after ingestion. Data in Gold should change only when the business logic that defines it changes.
Azure Data Factory vs Databricks vs Synapse Pipelines
This is the most common architecture question in Azure data projects and it is routinely answered wrong because the answer depends on what you are optimising for.
**Azure Data Factory** is the right choice for orchestration of well-defined ETL pipelines with clear source-to-destination movement. It is excellent for ingestion from the 90+ native connectors and for scheduling and dependency management. It is a poor choice for complex data transformation logic that requires code.
**Databricks** is the right choice when your transformation logic is complex, when you need machine learning capabilities alongside your data engineering, or when your data volumes require distributed compute. It is a higher operational overhead than ADF but significantly more capable for sophisticated processing.
**Synapse Pipelines** is best understood as ADF with Synapse Analytics integration. If your analytical workload is primarily SQL-based and Synapse is your compute layer, Synapse Pipelines minimises the integration surface area. If you are using Databricks for compute, the Synapse Pipelines integration adds complexity without benefit — use ADF instead.
The common mistake: using ADF for complex transformations by embedding long SQL blocks in pipeline activities. This creates untestable, unversionable transformation logic buried in pipeline JSON. If your transformation logic is complex enough to require extensive SQL, move it to a dedicated compute layer.
Delta Lake is non-negotiable for production workloads
If you are building on ADLS Gen2 without Delta Lake, you are building with infrastructure that cannot reliably support concurrent writes, schema evolution, or time travel — all of which you will need eventually.
Delta Lake's transaction log provides ACID compliance on top of object storage. This is what allows multiple pipeline processes to write to the same table without corruption, what allows you to roll back to a previous table version when a bad pipeline run corrupts data, and what allows schema changes without breaking downstream consumers.
Delta Lake also enables the OPTIMIZE and ZORDER operations that dramatically improve query performance on large tables — by compacting small files and co-locating related data. Running OPTIMIZE ZORDER BY (date, customer_id) on a large fact table is frequently the highest-impact single change you can make to analytical query performance in a Databricks environment.
Unity Catalog before you have a governance problem
Unity Catalog is Azure Databricks' data governance layer. It provides a centralised metastore, fine-grained access controls at the table, row, and column level, and lineage tracking across notebooks, jobs, and dashboards.
The architectural recommendation: implement Unity Catalog before you need it. Retrofitting governance onto an unmanaged Databricks workspace is painful — there is no clean migration path for external Hive metastore tables, and establishing consistent permissions across an existing data estate is far harder than designing them correctly from the start.
The access control model should mirror your organisational structure. Data domains map to catalogs. Teams map to schemas. Row-level security enforced in Unity Catalog is more reliable and auditable than row-level security implemented in individual notebooks.
Monitoring and observability are not optional
Every production Azure data platform needs monitoring that can answer three questions within minutes of a failure: what failed, why it failed, and what downstream processes are affected.
The baseline monitoring stack: Azure Monitor for infrastructure metrics, Log Analytics workspace for logs aggregated from all data services, and Azure Data Factory monitoring for pipeline execution history. Add custom alerts on pipeline failure, extract refresh failure, and data quality check failures.
The pattern that distinguishes mature data platforms from immature ones: data quality monitoring treated as a first-class engineering concern. Run automated data quality checks after every significant pipeline run — null rate, row count variance, referential integrity. Alert immediately when quality metrics fall outside expected bounds. Catching a bad pipeline run in minutes is an entirely different problem than discovering it three weeks later when an executive questions a dashboard number.
Common architectural mistakes we see repeatedly
Private endpoints added retroactively. Network security requirements typically mandate that data services are not accessible over public internet. Adding private endpoints to a platform that was built without them requires significant networking rework. Build with private endpoints from day one.
No separation between development and production environments. Development engineers with write access to production data is a governance risk and a data quality risk. Separate subscriptions for dev, test, and production with automated deployment pipelines between them is the baseline.
Ignoring Synapse Analytics for SQL workloads in favour of Databricks. Databricks is powerful but expensive for workloads that are primarily SQL-based with no machine learning requirement. Synapse Analytics dedicated SQL pools are frequently the more economical choice for large-scale relational analytical workloads.
The decision that matters most
Every Azure data platform architecture eventually comes down to one decision: where does business logic live?
If business logic lives in your analytical layer — in DAX, in Tableau calculated fields, in ad-hoc SQL queries — you have a platform that is hard to test, hard to version, and impossible to audit. The same business logic will be implemented differently in different places, and the numbers will never fully reconcile.
If business logic lives in your data platform — in governed transformations in your Silver and Gold layers, enforced through Unity Catalog and documented in your data catalog — you have a platform that can scale with the organisation.
That decision shapes every architectural choice that follows. Get it right at the start.
For the conceptual distinction between architecture and engineering decisions — which matters when building an Azure data team — data architecture vs data engineering covers where the two disciplines begin and end.
Our Azure data engineering consulting team has built production Azure data platforms at enterprise scale. We are happy to review your current Azure data architecture — or help you design a new one from first principles. Book a free 30-minute audit and we will give you an honest assessment of where you are and what needs to change.
A former Microsoft data architect audits your data foundation, identifies your top priorities, and sends you a written plan. Free. No pitch.
Book a Call →