All writing

When it breaks

Schema drift is always noticed too late

Upstream changes rarely break anything immediately. They break something three weeks later, in a report, in front of somebody senior.

1 minute read

A column widens. A nullable field starts arriving null. A type changes from integer to string in a source system because somebody added a code with a letter in it. None of these fail a pipeline. They all change a number quietly, and the number is noticed at the far end, by a person who cannot see the cause and has to be convinced there was one.

Why pipelines do not catch it

Most transformations are written against the columns they use, not against the shape they expect. A `select` of five columns from a table that grew a sixth is unaffected. A cast that silently coerces is unaffected until a value will not coerce. The pipeline is designed to be tolerant, which is correct for throughput and exactly wrong for detection.

Detection is a diff, not a test

The reliable approach is to snapshot the shape of the estate — columns, types, nullability, and the statistics you can get cheaply — and compare it to the previous snapshot. Changes then arrive as a list, with the objects downstream of each one, before anybody has written a ticket. The point is not to block the change; it is to be the first to know about it rather than the last.

What a useful drift report contains

  • What changed, in the vocabulary of the source: the column, the old type, the new type.
  • When it first appeared, which is usually not when it was noticed.
  • What is downstream of it, to column level where the lineage supports it.
  • Whether anything downstream actually reads the changed column, which decides whether this matters at all.

That last line is what separates a useful drift report from an unread one. An estate of any size drifts constantly; most of it is irrelevant. A tool that reports all of it has moved the problem rather than solved it.

Run this against your own estate

Gecko reads metadata and query history — never the rows in your tables — and tells you what it found.

Get started