Column-level lineage, and what it is actually for

Most lineage tooling is sold on the impact-analysis story: change this table, see what breaks. That is useful about once a quarter, during a migration. The question teams actually have weekly is narrower and nastier — this figure moved, where did it come from — and table-level lineage cannot answer it.

The difference in practice

A mart has forty columns and eleven upstream tables. Table lineage tells you the figure could have come from any of the eleven. Column lineage tells you that this one column is derived from two of them, through one join and one case expression, and that the case expression was edited last week. One of those is a starting point; the other is an afternoon.

Why it is harder to produce

• It requires parsing SQL rather than reading a dependency list, and every warehouse has its own dialect.
• Expressions fan in: one output column frequently depends on several inputs through logic that has to be followed, not just recorded.
• `select *` destroys it at the point of use, and it is everywhere.
• Dynamic SQL and macros mean the text in the repository is not the text the warehouse ran.

Where it earns its keep

Three places, consistently: tracing a number that changed, scoping a schema change to only the models that genuinely read the altered column, and finding columns that are computed and then never read — which is dead work hiding inside a live table, and is invisible at table level because the table itself is busy.