Stop alerting. Start fingerprinting

A data team's alerting channel is usually a record of one problem, repeated. The same model fails at the same step every night, and every night it produces a message identical to the last one except for the timestamp. After a week the channel is noise; after a month it is muted; after a quarter nobody notices the new failure arriving among the old ones.

An alert is an event. An incident is a thing

The fix is not fewer alerts, it is a different object. Take the error — the model, the step, the exception shape with its variable parts stripped out — and hash it. Every occurrence with the same fingerprint joins the same incident and increments a count. Thirty nights of the same breakage becomes one row that says it has happened thirty times and when it started, which is a far more actionable sentence than any one of the thirty messages.

What has to be stripped to make it work

• Timestamps, run ids and query ids — present in nearly every error, different every time.
• Row counts and byte counts, which vary with the data rather than the fault.
• Temporary object names, which often carry a run identifier.
• Absolute paths that include a workspace directory created per run.

Strip too little and every occurrence is a new incident, which is where you started. Strip too much and two genuinely different faults collapse into one, which is worse — it hides the second behind the first one's count. The rule that works is to strip what varies per run and keep what varies per cause.

What it changes

Three things, immediately. A long-running failure becomes visibly long-running rather than perpetually new. A new failure stands out, because it has a count of one. And "how often does this happen" becomes a field instead of an afternoon in the logs — which is the number you need to decide whether it is worth fixing properly.