When it breaks
Schema drift is always noticed too late
Upstream changes rarely break anything immediately. They break something three weeks later, in a report, in front of somebody senior.
1 minute read
A column widens. A nullable field starts arriving null. A type changes from integer to string in a source system because somebody added a code with a letter in it. None of these fail a pipeline. They all change a number quietly, and the number is noticed at the far end, by a person who cannot see the cause and has to be convinced there was one.
Why pipelines do not catch it
Most transformations are written against the columns they use, not against the shape they expect. A `select` of five columns from a table that grew a sixth is unaffected. A cast that silently coerces is unaffected until a value will not coerce. The pipeline is designed to be tolerant, which is correct for throughput and exactly wrong for detection.
Detection is a diff, not a test
The reliable approach is to snapshot the shape of the estate — columns, types, nullability, and the statistics you can get cheaply — and compare it to the previous snapshot. Changes then arrive as a list, with the objects downstream of each one, before anybody has written a ticket. The point is not to block the change; it is to be the first to know about it rather than the last.
What a useful drift report contains
- What changed, in the vocabulary of the source: the column, the old type, the new type.
- When it first appeared, which is usually not when it was noticed.
- What is downstream of it, to column level where the lineage supports it.
- Whether anything downstream actually reads the changed column, which decides whether this matters at all.
That last line is what separates a useful drift report from an unread one. An estate of any size drifts constantly; most of it is irrelevant. A tool that reports all of it has moved the problem rather than solved it.
More on when it breaks
Run this against your own estate
Gecko reads metadata and query history — never the rows in your tables — and tells you what it found.
Get started