Data Pipelines That Don't Wake You Up: Reliability Engineering for Data Teams

01 · The 2 AM problem

Why data pipelines fail (and why it's always at night)

Data pipelines fail at night because they run at night — batch windows, off-peak loads, and the accumulated consequences of daytime changes. The failure taxonomy is depressingly consistent:

Source changes. An upstream API adds a field, renames a column, or changes a timestamp format. The pipeline that assumed stability breaks. Nobody told the data team because nobody knew the data team depended on it.

Data quality drift. Null rates creep up, distributions shift, a new product category appears that breaks every downstream CASE statement. The pipeline ran green; the data is wrong. These are the failures stakeholders discover, which is worse than the pipeline discovering them.

Scale surprises. The Black Friday load, the viral launch, the backfill someone ran without telling anyone. The pipeline that handled 10M rows daily chokes on 200M. Compute limits, timeouts, and out-of-memory errors at 2 AM.

Dependency chains. Pipeline C waits for B waits for A; A is delayed by an upstream outage; the SLA breach cascades. Nobody owns the end-to-end timeline, so nobody sees the cascade coming.

Each of these has an engineering answer. None of them is solved by a more heroic on-call rotation.

02 · Contracts

Data contracts: stop failures at the boundary

A data contract is a formal agreement between producers and consumers: schema, semantics, freshness SLAs, and quality guarantees — enforced at the pipeline boundary, versioned, and tested in CI. The principle: breaking changes should be caught at the boundary, not discovered in a broken dashboard.

What goes in a contract: the schema (fields, types, nullability), semantic definitions (what does "active user" mean — the definition, not the column name), freshness commitments (data available by 6 AM daily, or within 15 minutes for streaming), quality rules (this field is never null, values within this range), and the change policy (additive changes flow; breaking changes require version bump and consumer sign-off).

Enforcement is what separates contracts from documentation: schema validation at ingestion (reject or quarantine non-conforming records), contract tests in CI (producer changes run against consumer expectations before merge), and schema registries for event streams. Start with your five most critical pipeline boundaries — the ones whose failure pages someone. Expand from proven value, not from a master plan.

03 · Quality gates

Test data like code

Software engineering learned decades ago that untested code fails in production. Data engineering is learning it now. The testing layers:

Schema tests. Columns exist, types match, nullability respected. Cheap, fast, catches the most common source-change failures. Every pipeline gets these; there's no excuse for skipping them.

Data quality tests. Uniqueness on keys, referential integrity, accepted value ranges, freshness thresholds, row-count bounds. Tools like dbt tests, Great Expectations, or Soda make these declarative. The art is in the thresholds: too tight and every normal variation pages someone; too loose and real problems pass through. Tune from production history.

Distribution checks. The sophisticated layer: has the distribution of this column shifted significantly? New categories appeared? Null rate tripled? These catch the subtle drift that schema tests miss — the "pipeline green, data wrong" class. Run them on silver/gold layers where business semantics live.

Quarantine, don't just alert. Failed records go to a quarantine area with full context — not silently dropped, not passed through. Someone triages quarantine daily. The quarantine table is also your best source of contract-improvement ideas: patterns in quarantined records reveal what the contract missed.

04 · Idempotency

Design for the retry

Pipelines will fail mid-run. The question is whether a retry is safe. Idempotent pipelines — re-running produces the same result — turn failures into retries instead of corruption events.

The patterns: merge/upsert on stable business keys instead of blind appends (re-running an append duplicates; re-running a merge converges). Deterministic transformations — same input, same output, no timestamps or random values baked into results. Watermark-based incremental loads — track what's been processed and resume from the watermark, never reprocess-or-skip ambiguously. And transactional writes where the platform supports them — Delta Lake and Iceberg give you atomic commits; use them instead of multi-step writes that can half-complete.

Backfills deserve the same discipline: a backfill is just a pipeline run over a historical window, and it should use the same idempotent logic. The backfill that someone runs manually with different logic is how you get two versions of history. One code path, all windows.

05 · Observability

Monitor four things, not one

Most teams monitor pipeline success/failure and call it observability. That's 25% of the job:

Freshness. Did data arrive when expected? SLA tracking per pipeline with alerting on breach — and SLAs negotiated with consumers, not invented by engineers. A pipeline that's "usually done by 7" needs a defined SLA before you can alert on missing it.

Volume. Row counts and byte volumes within expected bounds. Sudden drops mean upstream problems; sudden spikes mean duplicates or scope changes. Both are worth waking someone for — after tuning the bounds from history.

Quality. The test results from the quality gates, trended. Null rates, validation failure rates, quarantine volumes. Quality degradation is usually gradual — trending catches what threshold alerts miss.

Cost. Compute per pipeline per run. The pipeline that silently doubles its Snowflake credits or DBU consumption is a reliability problem wearing a FinOps costume. Cost anomaly detection belongs in the data observability stack.

Lineage ties it together: when something breaks, you need to answer "what's downstream of this?" in seconds, not in a Slack thread. Automated column-level lineage (from dbt, from your catalog) turns incident response from archaeology into navigation.

06 · Culture

Reliability is a team property

The technical patterns only work inside a culture that treats data reliability as engineering. That means: pipelines as code (versioned, reviewed, tested — no click-ops production pipelines), blameless postmortems for data incidents (every 2 AM page produces a prevention item, not a hero story), SLAs as contracts with stakeholders (negotiated, published, measured), and on-call that's sustainable (if the rotation burns people out, the system is unreliable by definition — the humans are part of the system).

Start where the pain is: pick the three pipelines that page most often, apply contracts, gates, and idempotency to those, and measure the page-rate drop. Reliability compounds — each stabilized pipeline frees the attention to stabilize the next. The goal isn't zero incidents; it's incidents that are caught by gates, retried safely, and learned from systematically. That's what "pipelines that don't wake you up" actually means.

07 · Incidents

Data incident response: the runbook

Even reliable pipelines fail. The difference is response quality. The data incident runbook:

Assess blast radius first. Before fixing anything: what's downstream of the failure? Lineage answers this in seconds — which dashboards, which models, which decisions consumed the bad data. The instinct is to fix the pipeline; the discipline is to scope the impact first, because the fix for "one table is stale" differs from "three days of bad data fed the forecast."

Decide: fix forward or backfill. Stale data usually means fix-forward (repair the pipeline, let the next run catch up). Wrong data means backfill (correct the historical records with the idempotent pipeline logic — never manual SQL heroics that create a second version of history). Communicate which one you're doing; stakeholders can tolerate either, but not ambiguity.

Communicate early and specifically. "The revenue dashboard is showing stale data as of yesterday; the pipeline failed at 2:14 AM on a schema change from the billing API; fix expected by noon; no other dashboards affected." Compare with: "We're looking into a data issue." Specificity builds trust; vagueness destroys it.

Postmortem everything that pages. Blameless, written, with prevention items that have owners and dates. The pattern to watch: the same root cause twice means the prevention item didn't work or wasn't done. Data incidents that don't produce systemic fixes are just scheduled surprises.

Track the metrics that matter: mean time to detect (are gates catching it or are stakeholders?), mean time to scope (does lineage work under pressure?), and repeat-incident rate (is the postmortem process working?). Reliability you can't measure is reliability you don't have.

08 · Bottom line

Reliability compounds

Data pipeline reliability isn't a project with an end date — it's a capability that compounds. Each contract you enforce prevents a class of failures. Each quality gate catches problems before stakeholders do. Each idempotent pipeline turns a future 2 AM page into a silent retry. The teams that sleep well didn't get lucky; they built the boring infrastructure of reliability one pipeline at a time.

Start with the three pipelines that page most often. Measure the page-rate drop. Use the credibility to fund the next three. Within a year, the on-call rotation is uneventful — and uneventful is the highest compliment infrastructure can earn. The goal was never heroics. It was pipelines so reliable that nobody tells stories about them.