Pattern 03 · Data Architecture

The Lakehouse, without the swamp

A production pattern for one governed copy of your data on open table formats — serving BI, ML, and streaming from the same ACID foundation.

governed copy of the data, not N pipeline copies

zones — bronze raw, silver validated, gold curated

proprietary format lock-in with open tables

01 · PROBLEM

Warehouses are expensive; lakes are swamps

The classic trap: the data warehouse bill grows with every workload while the data lake accumulates ungoverned files nobody trusts. Teams copy data between systems for BI, ML, and streaming — each copy drifting, each pipeline owned by nobody, each audit a archaeology project.

The lakehouse pattern puts ACID transactions on cheap object storage via open table formats (Delta Lake, Apache Iceberg, Apache Hudi), so one governed dataset serves SQL analytics, machine learning, and streaming consumers without copies.

02 · WHEN TO USE

Fit and anti-fit

Warehouse compute costs are growing faster than business value.
The same data must serve BI dashboards, ML training, and event-driven apps.
You need schema evolution, time travel, and auditability on large datasets.
Regulatory or cost pressure favors open formats over proprietary storage.

Don't use it for small, stable, purely SQL workloads where a managed warehouse is already cheap and loved — or as an excuse to skip governance. A lakehouse without contracts and quality gates is a swamp with ACID.

03 · ARCHITECTURE

Reference architecture

Medallion zones enforce a one-way discipline: bronze is immutable raw, silver is validated and conformed, gold is curated for consumption. The catalog — not tribal knowledge — is how anyone finds anything.

04 · COMPONENTS

What each piece does

Component	Responsibility	Build/buy notes
Ingest (batch + stream)	Land data reliably: CDC from OLTP, event streams, file drops, SaaS extracts	Kafka/Kinesis for streams; Fivetran/Airbyte or Spark for batch — match to team skill
Open table format	ACID transactions, schema evolution, time travel on object storage	Standardize on ONE format org-wide; Iceberg for engine breadth, Delta for Spark depth
Bronze zone	Immutable raw landing — exactly as arrived, with ingestion metadata	Append-only; never mutate bronze, reprocess from it
Silver zone	Validated, deduplicated, typed, conformed to canonical keys	This is where data quality lives — don't skip it
Gold zone	Domain-curated models: star schemas, aggregates, feature tables	Owned by domain teams; versioned like code
Governance catalog	Schema registry, lineage, quality scores, access policy	Unity Catalog, Apache Polaris, or cloud-native — pick one and enforce it
Serving	BI, ML feature store, APIs — all reading the same gold tables	No extracts into shadow marts; serve from the lakehouse

05 · FLOW

How data moves through the zones

Land raw. Every source writes to bronze with ingestion timestamp, source ID, and batch/stream markers. Bronze is append-only and immutable.

Validate into silver. Deduplicate, type-cast, conform keys, apply data contracts. Failed records go to quarantine with reasons — never silently dropped.

Curate gold. Domain teams build modeled, documented, versioned datasets. Gold tables carry SLAs and named owners.

Serve everything from gold. Dashboards, ML training, and APIs read the same tables. New consumers don't create new copies.

Govern continuously. Lineage, quality scores, and access audits update with every write — the catalog is a runtime system, not documentation.

06 · TRADEOFFS

Decisions with real costs

Decision	Option A	Option B	Our default
Table format	Delta — deepest Spark/Databricks integration	Iceberg — broadest multi-engine support	Iceberg unless you're standardized on Databricks
Compute	One engine (Spark) — simpler ops	Multi-engine (Spark + Trino + Flink) — best tool per job, more skill needed	Start single-engine; add Trino for interactive SQL when needed
Streaming	Micro-batch — simpler, minute-level latency	True streaming — sub-second, exactly-once complexity	Micro-batch default; true streaming only for proven latency SLAs
Small files	Ignore — query performance degrades silently	Compaction jobs — operational overhead	Automated compaction with monitoring; it's non-optional at scale

07 · SECURITY

Non-negotiables

Encryption: at rest (SSE with customer-managed keys where required) and in transit; credentials via secret managers, never in pipelines.
Fine-grained access: row/column-level security and dynamic masking in the serving layer — analysts see masked PII, models train on tokenized features.
Environment separation: production data never lands in dev; lower environments use masked or synthetic subsets.
Network: private endpoints for storage and catalog; no public ingress to the data plane.

08 · GOVERNANCE

Running it like a system, not a swamp

Data contracts: producers declare schemas and SLAs; breaking changes are versioned, not surprise deployments.
Quality gates: expectation suites run on silver promotion; failures quarantine data and page owners.
Lineage: every gold table traces to bronze sources — impact analysis and audits are queries, not interviews.
Retention & lifecycle: tiered storage and defined retention per zone; bronze isn't an infinite attic.

09 · COST

Where the money goes

Storage is cheap; compute isn't. Object storage costs are usually trivial — the bill is Spark/Trino clusters and always-on streaming.
Separation is the lever: scale compute to zero when idle; the classic warehouse mistake is paying for idle capacity.
Compaction and vacuuming are real compute costs — budget them, and monitor file counts like you'd monitor any SLO.
Data transfer across regions/accounts adds up; co-locate compute with storage.

10 · IMPLEMENTATION

A phased path that de-risks

One domain, end to end (4–6 weeks). Pick a bounded domain with a willing owner. Bronze → silver → gold, cataloged, with one BI consumer. Prove the discipline before the platform.

Platform hardening (6–8 weeks). Standardize the format, automate compaction, enforce contracts, stand up the catalog as a runtime system.

Domain expansion (ongoing). Onboard domains one at a time with the same zone discipline; federate ownership, centralize platform.

Advanced serving (ongoing). Feature store, streaming gold tables, cross-region replication — only after the foundation is boring.

See it in motion: the SaaS CDC Lakehouse reference blueprint walks this pattern through a concrete scenario, and our Engineering practice builds lakehouses as production platforms.

11 · RELATED

Keep exploring

SaaS CDC Lakehouse

Reference blueprint: change-data-capture into a medallion lakehouse, end to end.

Read blueprint →

Engineering

Our data engineering practice: platforms, pipelines, and lakehouses that hold up.

Explore practice →

Modernization

Moving off legacy warehouses and databases onto open, cloud-native foundations.

Explore practice →

Start here

Talk to an Architect

Bring your hardest AI, data, or modernization problem. We'll tell you plainly whether we can help — and what it takes.

Talk to an Architect Start with a Discovery Kit

FAQ

Questions we hear

If your workloads are primarily SQL BI with strong existing warehouse investment, a warehouse may stay. Choose the lakehouse when you need the same data to serve BI, ML, and streaming without copies, or when warehouse compute costs are growing faster than value.

All three deliver ACID on object storage. Iceberg has the broadest engine support today; Delta is strongest inside the Databricks/Spark ecosystem; Hudi excels at upsert-heavy streaming. Pick based on your engine commitments, then standardize — format sprawl is the new silo.

Skipping the silver layer. Teams land raw data and build gold models directly on it, recreating the swamp with better marketing. Validation, deduplication, and conformed keys are the whole point.