Read on its own, each post solves one problem. Read in order, they follow the path a data team walks as it matures: understand the problem, assess the risk, define the fix, build the controls, monitor for what slips through, and respond calmly when something breaks anyway.

Here is the whole set, with the single idea behind each one and the one thing to do about it.

1. Start by naming the problem

“The dashboard is wrong” is not a diagnosis. It is a feeling, and it hides three different problems: a value that is wrong (quality), a value that is correct but late (reliability), and a change upstream that no rule was watching for (observability). Each one has a different owner and a different fix.

So when a number is disputed, ask three questions before you open a query. Is a value wrong or missing? That is quality. Are the values fine but late? That is reliability. Did something change that no rule saw? That is observability. The label is the fastest way to stop three teams from investigating the same ghost.

The takeaway: name the problem before you touch it, and the fix is already half done.

Read the full post: Data Quality, Data Reliability, and Data Observability: Telling the Three Apart

2. Rank issues by risk, not by row count

Profile a messy table and the errors with the biggest counts grab all the attention. They are almost never the ones that matter. Forty duplicate invoices feeding the revenue number will hurt you far more than twelve thousand harmless trailing spaces.

The fix is two columns, not a new tool. Next to every finding, write who feels it downstream and how often it happens, then sort by those instead of by count. Impact times likelihood is the whole model, and it quietly floats your real priorities to the top while the loud, harmless piles sink.

The takeaway: the most dangerous issue is rarely the one with the biggest number.

Read the full post: Count Is Not Risk: Ranking Data Issues by What They Actually Cost

3. Turn “clean” into a rule you can test

“Just make the data clean” cannot be tested, so it can never be met. A usable requirement has three parts: a rule a check can answer yes or no, a threshold that says how much deviation is acceptable, and a named owner accountable for the definition. That is the difference between a wish and a data contract.

Say each expectation out loud and hold it against three questions. Can I write it as a check that returns a number? Have I decided what number passes? Can I name the person who owns it? Three yeses and you have a requirement. A shrug on any of them and you have a preference.

The takeaway: if it cannot fail a test, it is not a requirement.

Read the full post: “Clean Data” Is Not a Requirement: Writing Rules People Can Act On

4. Put controls where they can actually stop the flow

You can have the right check and still catch the problem too late. A validation that runs after publication is a smoke detector wired to go off the morning after the fire.

There are only three places a control can live, and each catches something different. Ingestion stops bad rows at the front door. Transformation catches the errors your own joins and calculations create. The release gate is the last block before anyone downstream reads the data. Put the control at the earliest point that can still stop the problem, decide whether it blocks or only warns, and quarantine what fails so the good data keeps moving.

The takeaway: a check that cannot stop the flow is just a log line.

Read the full post: Where Should a Data Quality Check Live? Gates, Controls, and Quarantine

5. Watch the signals your tests will never catch

Row-level rules only catch the failures you already imagined. They cannot see a feed that arrived stale, a row count that collapsed, a column that changed type, or a value mix that quietly shifted. Those four signals, freshness, volume, schema, and distribution, are where the surprises live.

You cannot spot abnormal without first defining normal. For your most important table, write down four things: when it should arrive, how many rows is normal, what its schema should be, and the usual mix of values in the column you care about. That is a baseline, and a baseline is most of monitoring. Then alert on the size of the change, not the fact of it, so a two percent wobble stays quiet and a sixty percent drop wakes someone.

The takeaway: rules protect you from the mistakes you imagined, monitoring protects you from the ones you did not.

Read the full post: It Passed Every Test and Still Broke: Four Signals Validation Never Catches

6. Respond to a data incident without panic

A wrong number in a live meeting makes everyone want to start changing things immediately. That instinct is how a one-hour fix becomes a three-day cleanup.

Triage first, because you cannot size a response until you know the blast radius. Contain before you fix, because cleaning while the tap runs is endless. Trace to the source instead of patching the number, or it comes back next week. Verify the fix, tell the people who trusted the bad value, and close with a short blameless review so the same break does not return.

The takeaway: changing the data is the middle of an incident, not the end.

Read the full post: You Just Found Bad Data in Production. Now What?

How the six fit together

Read as a set, they build on each other, because you cannot rank a problem you have not named, define a fix for one you have not ranked, or control one you have not defined. Understanding comes first, response comes last, and monitoring is the quiet layer that keeps the whole thing honest between incidents.

Here is the entire guide in one line each, if you ever need to find the right idea fast.

Start at the top, or jump to whichever problem is on fire today. Each post also carries a link to its companion Pluralsight course if you would rather see the idea built hands-on.

In short

Data quality and observability is not one skill but six connected moves. Name the problem before you chase it, because “the dashboard is wrong” hides three different failures: a wrong value, a late value, and a silent change. Rank issues by impact and likelihood, not by row count. Turn vague requests like “make it clean” into a rule, a threshold, and an owner. Place controls where they can stop bad data, not merely log it. Monitor the signals row-level tests miss: freshness, volume, schema, and distribution. And when something breaks anyway, respond calmly rather than fast. Read the six posts in order and you have a working playbook for trustworthy data, from the first disputed number to the last blameless review.

Reference: Pinal Dave (https://blog.sqlauthority.com/), X

First appeared on Data Quality and Observability: A Field Guide in Six Parts

I have been the person the room turned to look at. The instinct is to start changing things immediately, to be seen doing something. Resist it. Panicked changes are how a small data problem becomes a big one, and how a one-hour fix becomes a three-day cleanup.

Slow down for the first few minutes so you can move fast for the rest. Here is the playbook I wish someone had handed me.

The data incident response playbook, step by step

Six steps, in order. The first two are the ones people skip, and skipping them is what turns the one-hour fix into the three-day cleanup.

Step one: triage before you touch anything

Before you change a single value, answer three questions. What exactly is wrong? How far has it spread? Who is already acting on it?

That last one gets skipped and later regretted. A wrong number already sitting in a report someone sent to a customer is a very different incident than a number only your team has seen. You cannot size the response until you know the blast radius.

Resist the urge to fix. Diagnose.

Step two: contain the spread

Now that you know roughly what broke, your job is still not to fix it. It is to stop it getting worse. If a bad feed is pumping duplicate rows into a table every few minutes, pausing that feed matters more than cleaning the rows already there. Clean up while the tap is still running and you will be cleaning forever.

Containment buys the one thing you actually need: time to fix the real thing properly instead of frantically.

Step three: find the source, do not just patch the symptom

Here the temptation peaks. You can see the wrong number. You could just update it. Set the value to what it should be, and the meeting is happy.

Then it comes back next week, because you treated the symptom.

The value you can see is the end of a chain, not the start. Walk it backward. The dashboard read a published table, which was built by a transformation, which read a source feed. Follow the lineage upstream until you find the actual break. This is where good lineage and metadata earn their keep, turning a whole-pipeline mystery into a single suspect stage. Fix it there and it stays fixed.

Step four: fix, and then verify the fix

The fix is not done when you make the change. It is done when you have proven the change worked and broke nothing else. Two different things, and people declare victory after the first one constantly.

Re-run the check that caught the problem. Confirm the number is right, not merely different. Then glance at the neighbors, because the classic incident-inside-an-incident is the well-meaning fix that quietly breaks the table next door.

Verification is boring. Verification is the job. A fix you have not verified is a hopeful edit.

Step five: tell the people who trusted the bad number

Somebody made a decision, sent a report, or built a forecast on that value. They need to hear it from you, in plain language, before they hear it from someone else. Say what was wrong, what is right now, and whether they need to do anything.

It is an uncomfortable message, and it is the difference between a team people trust and a team people quietly stop believing. Nobody expects perfection. Everybody remembers who told them straight.

Step six: the short, blameless review

Once the fire is out, spend fifteen minutes on the question that stops the next one. What broke, why did it break, and what single control would have caught it earlier?

Keep it blameless. The moment it becomes a hunt for who to blame, people stop telling you what really happened, and you lose the only information that prevents a repeat. And keep it short. A wandering two-hour meeting produces nothing. Fifteen focused minutes and one follow-up action beats it every time.

Write it on one page. What happened, the timeline, the cause, the fix, and the one thing you are changing so it does not happen again. A one-page record beats a heroic memory, because heroic memories leave the company and take the knowledge with them.

The real measure of a data team

I used to think a good data team was one that never had incidents. I do not believe that anymore. Data breaks. Sources change, feeds fail, someone fat-fingers a value.

The teams I trust are not the ones who never have a bad day. They are the ones who can take a wrong number in a live meeting and close it out calmly, quickly, without drama, then quietly make sure that exact break never happens again.

Changing the data is the middle of an incident, not the end. The end is when the fix is verified, the right people know, and you have learned enough to prevent the next one. Get there calmly and you become the person everyone is glad they turned to look at.

If you want the full walkthrough, including recovery verification and the post-incident review, I cover the whole response process in my Pluralsight course, Respond to Data Quality Incidents.

Reference: Pinal Dave (https://blog.sqlauthority.com/), X

First appeared on You Just Found Bad Data in Production. Now What?

This confused me for an entire afternoon once. Every rule I had written was passing, because every rule asked about the contents of a row, and every row was fine. What none of them could tell me was that yesterday’s file had never arrived. The table was full of perfectly valid data that was exactly one day stale.

My checks were guarding the rooms while someone walked off with the building.

Validation asks “is this row ok.” Observability asks “did the feed change.”

Validation takes a record and checks it against a rule you wrote in advance. Is the region present? Is the amount positive? Is the id unique? Good questions, and they catch plenty. But notice what they share. You had to know to ask them. Validation only catches the failures you already imagined.

Observability watches the behavior of the whole feed over time and asks something different. Is this feed acting the way it usually does? That question catches the failures nobody thought to write a rule for, and those are the ones that hurt.

The four signals that watch behavior, not rows

None of these can be caught by staring at a single row.

Signal one: freshness

Did the data arrive on time, or is a valid-looking table quietly stale? This is the one that got me. Every value can be correct and the table can still be wrong, because it is yesterday’s correct. Freshness watches the clock, not the contents.

-- Freshness as a check. How stale is the newest row?
SELECT MAX(loaded_at) AS last_load,
       DATEDIFF(hour, MAX(loaded_at), GETDATE()) AS hours_stale
FROM sales_orders;

Not one value is inspected there. The query only asks when the table last moved. Alert when hours_stale crosses the window you expect, and you have caught the failure that ate my afternoon. A row-level rule never would, because the rows sitting there look great.

Signal two: volume

Feeds have a rhythm. A daily load that usually brings in fifty thousand rows and today brings five hundred is screaming that something upstream broke, probably a partial load. A load that suddenly brings five million is screaming just as loudly in the other direction, maybe a duplicated source or a runaway join. Every individual row can be flawless. The count is the tell.

Signal three: schema drift

Upstream systems change without asking your permission. A column that held a whole number starts holding a decimal. A field named total becomes total_amount. A column vanishes entirely, and your pipeline, being agreeable, fills it with nulls and carries on. Schema drift catches that on the day it happens, instead of three weeks later when someone asks why a chart went flat.

Signal four: distribution shift

The subtle one, and my favorite, because it is so sneaky. Every value is still legal, but the mix has moved. Yesterday, ninety percent of your orders were in the “completed” status. Today, sixty percent are. No rule was broken. Both values are perfectly valid. But the shape of reality shifted, and it usually means something real happened upstream, like a status field that stopped updating.

You cannot spot abnormal without defining normal

All four signals need the same thing. To know that today is weird, you have to know what a normal day looks like. That is a baseline: the usual arrival time, the usual row count, the usual schema, the usual mix of values.

The good news is that normal is mostly just history. Your feed has been telling you what it usually does for months. You only have to start listening and write it down.

A quick word on not drowning in alerts

Set up monitoring for the first time and the temptation is to alert on everything. Within a week half the alerts are noise and all of them are muted.

Alert on the size of the change, not the fact of it. A row count that drops two percent is Tuesday. A row count that drops sixty percent is an incident. Tune the thresholds so an alert means “come look now,” not “a number moved slightly, as numbers do.”

Start your data pipeline monitoring this week

Pick your most important table and answer four questions you have probably never written down. When should it arrive? How many rows is normal? What is its schema supposed to be? What is the usual mix of values in the column you care about most?

Write the answers somewhere. You just built a baseline, which is most of observability.

Rules protect you from the mistakes you imagined. Monitoring protects you from the ones you did not. And the feeds that take down a dashboard are almost always in that second group, sitting there green and confident and quietly wrong.

If you want to see these monitors built with real data, I walk through freshness, volume, schema, and distribution in my Pluralsight course, Monitor Data Health with Data Observability.

Reference: Pinal Dave (https://blog.sqlauthority.com/), X

First appeared on It Passed Every Test and Still Broke: Four Signals Validation Never Catches

The checks were not wrong. Every one of them fired. They fired at the very end of the pipeline, after the data was already published, which is a bit like wiring your smoke detector to go off the morning after the fire. Technically accurate. Not much help.

A check is only as useful as two things: where it runs, and whether it can stop anything. A brilliant check in the wrong place is just a very polite log line.

Detection is not control

A check that writes “this looks wrong” to a log is detection. A check that can stop the bad data moving forward is a control.

Detection is a witness. It saw the crime and wrote it down. Control is the locked door that stopped the crime. You want witnesses. You build the pipeline on locked doors. That proud team had a wall of excellent witnesses and not one locked door, so everything they detected still got through.

Three places data quality checks can live

Three natural checkpoints, each good at catching something different.

At ingestion, the front door

Raw data arrives from a source. Checks here are about basic fitness. Is the file the right shape? Are the required fields present? Are the ids the right format? A problem caught here is the cheapest possible outcome, because the bad data has not touched anything yet. Reject it or quarantine it before it gets comfortable.

In transformation, the workshop

Here you reshape, join, and calculate. Some errors can only appear here, because they are born here. A join that fans out and doubles your rows. A revenue calculation that mishandles a currency. A category mapping that silently drops the values it does not recognize. Transformation checks catch the problems your own logic creates, and every pipeline creates a few.

At the release gate, the last door before the public

The final block before anyone downstream reads the data. Your last chance to say “no, this batch is not going out.”

This is exactly what the proud team was missing. They had checks everywhere except the one place that could have stopped publication.

Block or warn

For each control, decide how loudly it speaks. Some failures should stop the pipeline cold. A duplicate in the field that feeds revenue. A load that is half its expected size. A required column gone missing. Publishing that is worse than publishing nothing, so the gate blocks.

Others should only raise a flag. A slightly higher rate of missing optional values deserves a warning, not a full stop. Make every check a hard block and you will be woken at 3 am by something cosmetic. By the third time you will start ignoring the alerts entirely, which defeats the whole point. Save the hard stop for failures that are genuinely worse than a delay.

Quarantine, or where bad rows go to think about what they did

When a check catches bad rows you have a choice. Throw out the whole batch, or set aside just the bad rows and let the good ones flow. The second option is quarantine, and it is usually the kinder one. Good data keeps moving, bad data waits in a holding area, and nothing is silently lost.

-- Good rows go on. Bad rows wait in quarantine.
INSERT INTO orders_clean
SELECT * FROM orders_staging WHERE region_code IS NOT NULL;

INSERT INTO orders_quarantine
SELECT * FROM orders_staging WHERE region_code IS NULL;

Quarantine comes with a rule everyone forgets. Somebody has to actually look at it. A quarantine table nobody reviews is not a safety mechanism. It is a junk drawer, and six months later it holds forty thousand rows and a faint smell. Schedule the failed-record review. Someone owns opening that drawer and deciding what gets fixed, released, or thrown out for good.

The placement question, in one line

Where does a new check belong? Ask what is the earliest point where this problem can exist, and whether a check there can actually stop it. Put the control at the earliest place that can both see the problem and block it. Earlier is cheaper. Blocking is what turns a check into a control.

Try this on your pipeline

Pick your most important check and answer two questions honestly. Where in the pipeline does it run? And if it fails right now, does anything actually stop, or does the data keep flowing while an alert quietly logs itself somewhere nobody reads?

If the honest answer is “it logs and moves on,” you have a witness where you needed a locked door. Moving a check earlier and giving it the power to stop the flow is usually a small change. It just happens to pay for itself the first time it catches a bad batch at the front door instead of the front page.

If you want to see a full control plan with validation, gates, and quarantine, I build the whole pipeline in my Pluralsight course, Design and Implement Data Quality Controls.

Reference: Pinal Dave (https://blog.sqlauthority.com/), X

First appeared on Where Should a Data Quality Check Live? Gates, Controls, and Quarantine

Two weeks later the report was wrong and we were in a meeting arguing about whether cleaning the data had ever been my job. Nobody could win that argument. “Clean” was never defined, so it could not be met, measured, or tested. It was a wish wearing the costume of a requirement.

A requirement you cannot test is not a requirement. It is a hope with better grammar.

What “clean” is actually hiding

The word feels specific until you try to check it. Clean how? Clean by when? Clean to what standard? Zero errors, or good enough? On every field, or only the ones that matter?

A usable requirement answers those questions with three parts. Get all three and a machine can enforce it and a person can own it. Miss one and it quietly stops working.

The three parts of a usable requirement

Write them in this order. Each part closes a gap the one before it leaves open.

Part one: the rule

The rule is what a check answers with a yes or a no. No opinions, no vibes.

“The data should be correct” is not a rule. “Every order has a region code” is a rule. You can run it.

-- The rule, written as a test. Zero is a pass.
SELECT COUNT(*) AS failing_rows
FROM sales_orders
WHERE region_code IS NULL;

Written that way, the rule can fail. And a requirement that can fail is one that can pass, which means it is finally real. If yours cannot fail a test, it was never a requirement. It was a mood.

Part two: the threshold

Here is where careful people quietly lose. They write the rule and silently assume the answer must be one hundred percent. Always.

Sometimes it must be. An order id that is supposed to be unique should be unique, full stop, because a duplicate directly overstates revenue. But for plenty of fields, chasing perfection is how you burn a month on the last stubborn half percent that nobody downstream would have noticed.

The threshold says how much deviation you accept. Ninety nine and a half percent of orders carrying a valid region might be fine for a territory report. One hundred percent of order ids being unique is the right call for anything that touches money. Decide the number on purpose and out loud, instead of accidentally promising perfection and then explaining every month why you missed it.

Part three: the owner

A rule and a threshold with no human behind them are decoration. The owner is a real person who can decide what the field means and approve a change to it. Not “the data team.” A name.

Why does this matter so much? Because the interesting failures are never “the value is missing.” They are “two teams define region differently and both think they are right.” Somebody has to be the tiebreaker. If the answer to “who owns the definition of region” is a shrug, the requirement dies the first time two people disagree, which is usually week one.

Do not forget the service expectation

There is a quiet fourth part. When must this data be ready, how fresh must it be, and how often is it checked? A correct value that lands an hour after the review is still a miss. Bake the timing into the requirement so nobody argues about it later.

Before and after

Watch a wish turn into a requirement.

Before: “The data should be correct.”

After: “order_id is unique across sales_orders, one hundred percent, owned by the sales data lead, checked on every load. Failures block the publish.”

The first version starts fights. The second one ends them. One is a feeling. The other is a line in a data contract that a machine enforces and a named person answers for. When the check fails there is no debate about whose job it is, what “correct” meant, or whether it is bad enough to matter. It is all right there in the sentence.

A quick test for your data quality rules

Take any expectation floating around your team right now. Say it out loud, then ask three questions. Can I write it as a check that returns a number? Have I decided what number counts as passing? Can I name the person who owns it?

Three yeses and you have a requirement. A shrug on any of them and you have a preference. And preferences do not survive contact with a busy quarter.

Clean is a feeling. A rule, a threshold, and an owner is a plan. Write the plan.

If you want to see rules, thresholds, and contracts come together into a full requirements pack, I build it step by step in my Pluralsight course, Define Data Quality Requirements.

Reference: Pinal Dave (https://blog.sqlauthority.com/), X

First appeared on “Clean Data” Is Not a Requirement: Writing Rules People Can Act On