After going through the checklist, I started thinking about what I would actually want to use the next time I perform a security review.

The DoD STIG for SQL Server is a great, solid starting point for establishing your security practices with SQL Server. In fact, it’s also a good template for your own STIG in your organization. So, you may want to create a custom checklist that makes sense from the perspective of your SQL Server environment.

The STIG is detailed, which is a good thing, but I found myself wanting something a little more practical for day-to-day DBA work. Something I could open, work through one item at a time, record what I found, and come back to later without having to navigate through the entire STIG document every time.

Questions to Ask

I want it to answer some simple questions:

• What am I checking?
• Why am I checking it?
• How can I verify it?
• What does PASS or FAIL look like?
• What evidence should I record?
• Does this requirement actually apply to this environment?
• If I can’t meet a requirement, how do I document the exception?

So in this post, I’ll walk through how I’m building the checklist, based on the DoD STIG as the baseline (and I can then organize it into something I can actually use as a DBA).

The main thing here is to understand what the STIG is asking you to verify.

Once I have the STIG requirements in front of me, the next step is to break each rule down into information that I can actually work with. Rather than simply copying the requirement into my checklist, I want to capture a few key details for each rule.

This gives me enough information to understand what I’m checking, how I can validate it, and what I should record when I’m done.

Again, this STIG is just a starting point. Capture the checklist in a separate document that you can share in your organization.

What to Capture?

For every rule, capture:

• STIG ID
• Severity
• Requirement
• Check
• Fix
• Instance or Database
• Applicable?
• Current status

Let’s Create a Checklist

Let’s open the STIG Viewer, at the Checklist section, click New to create a new checklist.

Let’s select to create a checklist for a SQL Server Instance.

You have to options: Add individual rules or add all the rules in the STIG. Click the search icon to pick the individual rules or the plus sign to add all the rules. For the purpose of this blog post, let’s add few rules.

Select the rule by clicking the Plus sign. Select all that you want added in your checklist.

To save the checklist, navigate to the burger menu in the top left-hand corner and select Save. Give the checklist file an intuitive name.

Once saved, you can close the checklist builder. The STIG Checklist options will then give you a couple of options to load or create another checklist. Let’s open the checklist that we just created.

Review the checklist and check it against a specific instance or database that you want to assess. Here you can note the instance/database information and mark each rule with three statuses:

Green Check Icon: Not a Finding (meaning instance/database pass the rule),

Red Exclamation: Open (for further review), or

Crossed Circle: Not Applicable

Default Grey Circle: Not reviewed

Keep clicking the gray button to cycle through the statuses.

You can also leave a comment and findings details in each of the rules in the designated boxes at the bottom.

You can then export the checklist if you want to keep a separate file for the assessment (html and then print it to a pdf file).

Here’s some screenshots of the the example html…

Your Checklist as Your Tool

Instead of reproducing the STIG’s structure, I’d create something a DBA can work through. I’d create a checklist for the following categories:

That and among other things that may be applicable to whichever environment I happen to be working on.

This is where your checklist becomes your own tool, rather than just another copy of the STIG.

Most Important Thing, Document

This is an important part of the checklist. I don’t want the process to be simply about finding a problem and immediately changing the configuration. Involve everyone and, needless to say, document.

The goal is to first understand the current state, evaluate whether the setting actually needs to be changed, make the appropriate remediation if necessary, and then validate the change. This gives me a more deliberate process instead of treating every finding as something that automatically requires a configuration change.

This process would look as simple as the following:

1. Check – Determine the current state.

2. Evaluate – Decide whether the configuration is appropriate for your environment.

3. Remediate – Make the change if necessary.

4. Validate – Run the check again.

5. Document – Record what changed.

That gives you a repeatable process:

       CHECK
         ↓
      EVALUATE
         ↓
     REMEDIATE
         ↓
      VALIDATE
         ↓
     DOCUMENT

And I’d emphasize that STIG compliance doesn’t automatically mean every setting should simply be changed without considering the environment.

If this was a real checklist for a SQL Server environment I am responsible for, this is my starting point. I expect the checklist to change as I use it, find gaps, and learn more about what works in an actual SQL Server environment.

The STIG gives me a solid baseline, but turning those requirements into something I can consistently use as a DBA makes the exercise much more useful for me.

My next step is to take these checks and start turning them into T-SQL scripts where possible, so instead of manually checking everything, I can let SQL Server help me identify what needs attention.

Have fun and enjoy securing your SQL Server!

The post Creating a Custom SQL Server Security Checklist Using the DoD STIG first appeared on SQL, Code, Coffee, Etc..

When I put together the invitation for T-SQL Tuesday #202, I wasn’t sure what kind of stories would come out of it.

The topic was simple:

That one SQL Server outage you’ll never forget.

I expected stories about bad queries, failed deployments, storage problems, or maybe a database that decided to have a very bad day. What I got was much more interesting.

There were stories about ransomware, corrupted databases, deleted storage, power and cooling failures, an identity column reaching its limit, and even a floppy disk. A server Meltdown too.

Some of these outages lasted hours. Others lasted days or even weeks.

And what I really enjoyed was that most of these stories weren’t just about what went wrong. They were about what people learned afterward.

Here are the stories I was able to collect.

Deborah Melkin: Sometimes a Floppy Disk Is All It Takes

Deborah Melkin shared a story from earlier in her career involving a server reboot that went very wrong. A bootable floppy disk had been left in the server. It contained an fdisk /mbr command. The server was rebooted, the command ran, and suddenly they had a much bigger problem than they expected.

The team had to find another server, reinstall SQL Server, and restore the databases from backup. The lesson seems obvious now, but that’s one thing I like about outage stories. Things that seem obvious after the fact aren’t always obvious when you’re standing in the middle of the problem.

Sometimes the lesson is simply to understand what is actually sitting in or connected to the server before you reboot it. And, of course, make sure your backups are somewhere other than the server you’re trying to recover.

Read Deborah’s full story

Andy Yun: The Outages That Live in Infamy

Andy Yun shares two non-SQL Server outages before getting to his SQL Server story. The first was when a backhoe literally dug up his company’s T1 line, leaving them without Internet for most of the day.

The second happened at a software company that supported market traders, where executives chose not to have a backup Internet connection after moving the office to VoIP. When the Internet went down, the entire office, call center, and data center were effectively offline, including their phones.

Both stories reinforced the same lesson: single points of failure hurt.

His SQL Server outage happened while Andy and another DBA were at PASS Summit, when a SAN administrator accidentally deleted the production transaction-log LUN for an instance with several hundred databases and several terabytes of data. Fortunately, their backups were good, and Andy used sp_restoregene to quickly generate the restore commands.

They initially tried four parallel restores, but the SAN couldn’t handle the load, so they stopped and worked with the business to prioritize the databases. The full recovery took three or four days.

What Andy realized afterward was that while they had tested restores for CHECKDB, they had never tested a full-instance restore at that scale. The experience also made him realize how important it was to understand the storage and infrastructure that your disaster recovery plan depends on.

Read Andy’s full story

Aaron Bertrand: When an INT Runs Out

Aaron Bertrand’s story is a good reminder that capacity problems aren’t always about disk space, memory, or CPU. In this case, an identity column reached the maximum value for an int. The result was an outage at Stack Overflow. The immediate solution wasn’t to change the column to bigint.

That would have been much more difficult to do during a live production outage. Instead, the identity was reseeded into the negative range, buying the team years of additional capacity. Problem solved. Until it happened again. A later operation involving IDENTITY_INSERT caused the identity value to move toward the positive limit again. So the team had another outage. And they reseeded it again.

I liked this story because it shows how an emergency fix can create another problem if the underlying issue isn’t eventually addressed. It also reminded me that we tend to think about capacity in terms of infrastructure. But data types have limits too. Sometimes the thing running out of room isn’t the disk.

Read Aaron’s full story

Rob Farley: When Corruption Leaves You With Very Few Options

Rob Farley wrote about an outage involving a bad disk controller that corrupted hundreds of database pages and even affected some backup files. This was one of those situations where the normal recovery path wasn’t enough. Rob and the customer’s CTO started looking at the database table by table, using clustered indexes, nonclustered indexes, DBCC PAGE, older backups, and other sources of information to reconstruct what they could.

One of the interesting parts of the story was discovering that a nonclustered index could still contain information that was missing from the corrupted clustered index. That meant even damaged parts of the database could potentially be useful during recovery.

Eventually, they were able to rebuild the tables and indexes and get the system back online. Reading this made me think about how different troubleshooting becomes when you’re no longer trying to find the best query plan or fix a blocking problem. When you’re dealing with serious corruption, you’re looking for anything that can help you recover the data.

Read Rob’s full story

Vlad Drumea: Two Weeks of Ransomware Recovery

Vlad Drumea shared probably one of the biggest incidents in this collection. His organization was hit by Ryuk ransomware in 2020. More than 60 SQL Server instances across more than 30 VMs were involved. Recovery took more than two weeks. This wasn’t simply a matter of restoring a database.

The environment itself had to be treated as compromised. Vlad described rebuilding VMs, recreating SQL Server directories, restoring system databases, restoring user databases, dealing with reinfection, fixing broken LSN chains, and rebuilding a VM from scratch. There was also a lot of automation involved using PowerShell, T-SQL, and dbatools. What really stuck with me was the human side of this story.

Recovery involved 16-hour workdays for more than two weeks, and Vlad talks about the burnout that followed. We spend a lot of time talking about backups, DR, security, and automation. Those things matter. But there are also people sitting in front of those computers at 2 AM trying to get a business back online. That’s part of the story too.

Read Vlad’s full story

Jeff Taylor: SQL Server on Fire, Literally!

Jeff Taylor shared two stories involving infrastructure problems. The first started with a power outage in an office that had effectively become a small data center. The servers had battery backup. The air conditioning didn’t. As the room heated up, the team started using fans and eventually began shutting servers down to keep the hardware from being damaged.

The temperature eventually approached 120°F. That incident resulted in a much larger infrastructure redesign, including better cooling, battery backup for the cooling, a generator, and fire suppression.

Then there was another incident involving a Dell server where Jeff was replacing memory and a drive. After powering it back on, he saw a flash. Then smoke. The problem apparently involved an iSCSI cable that had been damaged during the earlier heat incident and eventually shorted when the equipment was moved.

It’s a good reminder that SQL Server doesn’t operate in a vacuum. Power, cooling, storage, networking, and the physical environment are all part of keeping a database available.

Read Jeff’s full story

Edwin Sarmiento: When the Windows XP Image Wiped Out SQL Server

Edwin Sarmiento shared a story from a System Center deployment project where he was responsible for building a ConfigMgr environment for a healthcare company. To keep the deployment isolated, he placed ConfigMgr on its own network and set up a separate SQL Server.

During testing, he scheduled a Windows XP image deployment to run against some demo virtual machines. The problem was an incorrect IP address range. Instead of deploying the image to the demo VMs, it deployed to the SQL Server machine and wiped out the entire server.

Edwin actually discovered the problem after receiving a 6:15 AM alert that SQL Server was down, eventually logging in to find the SQL Server machine displaying the Windows XP desktop. Fortunately, this was an isolated environment and there was no disruption to the customer’s normal business operations.

What stood out to me about Edwin’s story is that his biggest lesson wasn’t about ConfigMgr or SQL Server. It was about the people involved, the decisions they make, and the importance of planning for what can go wrong before it does.

Read Edwin’s full story

Thomas Rushton: When the Server Room Gets Too Hot

The Lone DBA shared a story about a server room in an old Victorian mill building that overheated after the air conditioning failed during a hot summer weekend.

The servers were shut down, and after things cooled down, most of them came back online without any obvious problems. One server, however, kept crashing intermittently. They patched it, updated drivers, replaced the memory, HBAs, CPUs, and even the storage, but nothing fixed the problem.

Eventually, while replacing the motherboard, an engineer discovered that a daughterboard had partially melted during the overheating event, causing an intermittent short circuit. It’s a good reminder that a server room getting too hot isn’t just a temporary availability problem. It can cause physical hardware damage that may not show up until much later.

Read The Lone DBA’s full story

Andy Levy: When Your Database Time Travels

Andy Levy’s outage happened while he was away for the weekend and started with more than 250 notifications on his phone. His two-node SQL Server Failover Cluster Instance had repeatedly failed over, and both nodes had gone offline. When he checked the databases, he discovered that roughly two months of work had disappeared.

He shut down SQL Server to prevent things from getting worse and began working with his team on recovery. They ultimately restored the databases from backups to a cold spare SQL Server, getting the critical systems back online within a few hours. The postmortem revealed that a storage move two months earlier had left the two cluster nodes pointing to different copies of the virtual disks. Everything appeared fine until patching rebooted both nodes and caused the cluster to fail over to the VM connected to the older copy of the storage.

The databases had essentially “time traveled” two months into the past. What I liked about this story is how many small things had to line up for this to happen, and how weekly test restores, off-site backups, backed-up TDE certificates, and regular Export-DbaInstance scripts made the recovery possible.

Read Andy’s full story

Chad Callihan: What Time Is It?

Chad Callihan shared a deployment that appeared to go perfectly, only for users to start reporting later that morning that some data had the wrong times. Some records that should have been stored in local time were being saved in UTC, while other records were correct.

The problem wasn’t SQL Server itself, but another part of the release. Once identified, they were able to stop the problem, but cleaning up the incorrect data became the bigger challenge. Their environment had multiple time zones, with different tables, databases, logs, and error messages using different time references. Figuring out which records had the wrong time without changing the ones that were already correct made the cleanup particularly difficult.

The outage itself wasn’t very long, but Chad describes the cleanup as one of the worst parts of the experience. It’s a good reminder that deployments can appear successful while still leaving behind problems that may not show up until users start working with the data.

Read Chad’s full story

Rebecca Lewis: When SQL Slammer Took Down Both Data Centers

Rebecca Lewis takes us back to January 2003 and the SQL Slammer outbreak. Her organization had a primary data center in Chicago and a DR site in New York, but the worm came in through an approved VPN connection and quickly spread across the internal network. The resulting traffic overwhelmed their routers, eventually taking down both sites.

More than 400 physical servers had to be recovered, one at a time, while keeping clean machines isolated from infected ones. The IT and DBA teams worked through the weekend and had everything recovered by about 4 AM Monday, just in time for the market open. What stood out to me was that geographic redundancy didn’t help when the same problem could reach both locations.

The firewall had done its job, but the threat came through a legitimate connection that was already inside the network. It’s a great reminder that having a DR site doesn’t necessarily protect you from every kind of failure.

Read Rebecca’s full story

M G: A Comment That Deserved to Be Part of the Roundup

A person who goes by their initial M G don’t have a blog, but left a detailed comment on my invitation. I thought the story was too interesting to leave out.

The incident involved SQL Server 2016 and a SharePoint database with around 15 million rows and more than 850 GB of binary data. Three databases were repaired, but the fourth became the real problem. There were hundreds of suspect pages, a corrupted clustered index that was also the primary key, and a DBCC CHECKTABLE ... REPAIR operation that had been running for weeks and failing.

What caught my attention was the eventual workaround. Changing the database’s PAGE_VERIFY setting from CHECKSUM to OFF allowed the primary key to be dropped. That exposed another problem: SharePoint had accumulated roughly 90,000 duplicate elements. This is exactly the kind of kind of troubleshooting story that is difficult to forget because there isn’t necessarily a clean checklist that tells you what to do next. You investigate. You try something. You learn something new. Then you try again. And sometimes the solution comes from a place you weren’t expecting.

What I Took Away From These Stories

After reading through all of these, I noticed something. The actual cause of the outage was often not SQL Server itself. It was the environment around SQL Server.

A floppy disk. A SAN administrator deleting a LUN. An identity value reaching its limit. A bad disk controller. Ransomware. A lack of cooling. Corruption inside a SharePoint database.

These are very different problems, but they have something in common.

You don’t always know what the outage is going to look like until you’re already in it.

That’s probably why these stories are useful. You can study SQL Server performance. You can learn backup and restore. You can learn Availability Groups. You can learn PowerShell and dbatools. You can learn monitoring. But eventually, something unexpected is going to happen.

The best thing we can do is learn from people who have already been there.

And that’s what I really liked about this month’s T-SQL Tuesday. These weren’t polished success stories. They were stories about things going wrong.

And those are often the stories I remember the longest.

Thank you to everyone who took the time to participate in T-SQL Tuesday #202, whether you wrote a full post or shared your experience in the comments.

And thank you to Steve Jones for giving me the opportunity to host this month’s T-SQL Tuesday.

Until the next outage…

No, God forbids. It ould be yours, and hopefully the stories above give you the resolution route.

The post T-SQL Tuesday #202 SQL Server Outage You’ll Never Forget: A Roundup first appeared on SQL, Code, Coffee, Etc..

If you have been working with SQL Server for a while, chances are you have at least one outage that you still remember clearly. It might have happened years ago, and you probably still remember what time it happened, how you found out, what you were doing when the page came in, and what you had to do to get things back to normal.

I am excited to host T-SQL Tuesday for the first time. I want to hear about that one outage that stands out in your memory. We all have incidents that stay with us long after the servers are back up and things have returned to normal. I’m looking forward to hearing those stories and seeing what we can learn from each other.

That One SQL Server Outage You’ll Never Forget

Tell us about your most memorable SQL Server outage. It could have been a midnight page, a failed failover, a runaway query, a storage problem, a bad deployment, a server that simply would not come back up, or something else that brought production to a stop.

I’m interested in the whole story. What happened? How did you discover the problem? What did you check first? What did you try that worked, and what didn’t? How did you eventually get things back to normal?

Most importantly, what did you learn from the experience?

You don’t have to share the biggest outage you’ve ever dealt with. Maybe it was a relatively small incident that changed the way you approach backups, monitoring, failover, capacity planning, change management, or troubleshooting. Maybe it exposed a weakness in your environment that you didn’t know was there. Maybe it taught you something that you have carried with you throughout your career.

Share as much of the story as you can, including the things you wish you had known before the outage happened.

Since September is Labor Day month here in the US (September 7, 2026), I thought it would also be a good opportunity to recognize the people behind these systems. You can make your post as technical as you want, or you can focus more on the human side of the experience. After all, keeping databases online is not just about the technology. There are people behind those systems who have to respond when things go wrong, sometimes at the most inconvenient time.

My Own Outage Story

I have one of these stories myself, and it is an outage I don’t think I will ever forget because of the circumstances surrounding it.

I wrote about it in my blog post, Reflections on the Life of a DBA. It was a cold January evening, and I was at a black-tie party when the alerts started coming in. My phone was lighting up with Splunk On-Call alerts and Teams messages because an important SQL Server had gone down.

I had my work laptop with me, as I almost always did, so I found a corner in the busy kitchen, opened the laptop, and started working on the problem while everyone else continued with the evening. I still remember sitting there in a black suit with my laptop, troubleshooting SQL Server in the middle of a busy kitchen while a celebration was happening around me.

That is one of those moments from my DBA career that has stayed with me, and it is part of what inspired me to choose this month’s topic. We spend a lot of time talking about SQL Server features, performance tuning, architecture, and best practices, but some of the lessons that stay with us come from the times when something actually went wrong and we had to figure it out.

Now I’m curious about yours.

A Little T-SQL Tuesday History

T-SQL Tuesday started back in 2009 when Adam Machanic invited SQL Server bloggers to write about a common topic and publish their posts on the same day. What started as a simple way for the community to share different perspectives has become a long-running SQL Server tradition.

Today, Steve Jones coordinates the event, and the posts are collected in the T-SQL Tuesday archive. Thanks, Steve, for selecting me to host this month’s T-SQL Tuesday. If you have never gone through the archive, there is a lot of good SQL Server knowledge and real-world experience in there.

One of the things I like about T-SQL Tuesday is seeing how people from different backgrounds approach the same topic. You often learn something new, and sometimes you find a story that sounds very familiar.

The Rules

The rules for this month’s T-SQL Tuesday are pretty simple.

1. Write a blog post about the topic.

Write about your most memorable SQL Server outage and share the story, the recovery, and the lessons you took away from it.

2. Publish your post on Tuesday, September 8, 2026.

That’s the publishing date for T-SQL Tuesday #202.

3. Link back to this invitation.

Please include a link to this invitation in your post so readers can find the topic and discover the other posts participating in this month’s T-SQL Tuesday.

4. Add your post to the comments.

Once your post is published, leave the link in the comments below so I can find it and include it in the roundup.

5. Keep company and customer information confidential.

Please don’t include anything that shouldn’t be publicly shared, such as customer information, credentials, server names, IP addresses, or other sensitive details. Change the names and details as necessary. The goal is to share what we learned from the experience.

Now Tell Us Your Story

I’m looking forward to reading these because outages are where a lot of our best lessons come from. The technical details are important, but I’m also interested in what happened around the technical problem, how you approached the situation, what decisions you had to make under pressure, and what you changed afterward.

Most of us have had that one SQL Server incident that made us learn something the hard way.

Maybe you were at home. Maybe you were in the office. Maybe you were asleep. Maybe, like me, you were at a party sitting in a kitchen with a laptop.

What’s yours?

Write about it, share it with the SQL Server community, and let’s see what we can learn from each other’s outage stories.

The post T-SQL Tuesday #202 Invitation: That One SQL Server Outage You’ll Never Forget first appeared on SQL, Code, Coffee, Etc..

The stored procedure runs in milliseconds. It has been running in milliseconds for the last two years. Then one morning, it suddenly takes four minutes to complete. No code deployment. No configuration change. Nothing obvious changed. Restart the SQL Server instance, and it goes back to being fast — at least until later in the day.

You already know what the first response in the incident channel will be: “It’s parameter sniffing.”

And technically, that answer may be correct. But it does not tell you enough to fix the problem.

Parameter sniffing is not a single issue. There are different ways this can bite you, but two of the most common ones are easy to confuse.

The first is a bad plan choice. SQL Server compiles a plan based on one set of parameter values, but that same plan performs poorly when reused for a different set of values. For example, SQL Server may choose an index seek with hundreds of thousands of key lookups when a scan would have been the better option.

The second is a bad memory grant. SQL Server estimates it only needs enough memory for a small number of rows, but the actual query returns hundreds of thousands of rows. The result can be spills to tempdb, poor performance, and unnecessary memory pressure.

These two problems can look similar from the outside, but they require different troubleshooting approaches.

This is especially important with newer SQL Server features like Memory Grant Feedback. It can help correct inaccurate memory grants, but it does not change a fundamentally bad plan choice. If you do not identify which problem you actually have, you can apply a fix that was never designed to solve the issue.

The goal of this post is to separate these two behaviors, show how they are different, and walk through a demo you can reproduce yourself.

Sniffing is a feature

Before getting into the problem, it is important to set the right context. A lot of discussions around parameter sniffing make it sound like a SQL Server defect that Microsoft should have fixed. That is not really the case.

When SQL Server compiles a parameterized query, it uses the parameter values that caused the compilation to estimate the number of rows and build a plan. It looks at the statistics, checks the histogram, estimates the expected cardinality, and creates a plan based on that information. That plan is then stored in cache and reused for future executions, even if those executions use very different parameter values.

The first part of this process is actually what makes SQL Server perform well. Without parameter sniffing, SQL Server would have to create plans based on generic estimates instead of the actual values being searched. You would end up with a plan that is average for everyone instead of a plan that is optimized for the majority of cases.

The problem is not parameter sniffing itself. The problem is reusing a plan when the data distribution does not match the values that plan was originally optimized for.

This is where data skew comes into play. If values in a column are evenly distributed, most parameter values will produce similar row counts, and the cached plan will usually work well. Parameter sniffing only becomes a problem when some values return a small number of rows while others return a significantly different number of rows.

So the first question should not be, “How do I disable parameter sniffing?”

The better question is, “How is the data distributed, and how much skew exists in this column?”

SELECT   TOP (20) CustomerID, Rows = COUNT_BIG(*)
FROM     Sales.OrderLines_or_whatever
GROUP BY CustomerID
ORDER BY Rows DESC;

SELECT   TOP (20) CustomerID, Rows = COUNT_BIG(*)
FROM     Sales.OrderLines_or_whatever
GROUP BY CustomerID
ORDER BY Rows DESC;

If the difference between the highest and lowest values is only within an order of magnitude, then data skew is probably not your problem. There is likely something else causing the bad plan choice.

Building a demo; thanks, claude

I wanted to demonstrate both failure modes using actual query behavior, which means I needed data with a noticeable skew. The challenge is that standard sample databases are not always useful for demonstrating these types of problems.

For this test, I used Claude Code to help create the setup script and build the test scenario. WideWorldImporters, Microsoft’s sample database, was used as the starting point because it contains real order data. However, the data distribution is fairly uniform. Order lines are spread across customers, dates, and stock items. That makes sense for a sample database, but it does not create the conditions needed to demonstrate parameter sensitivity.

The skew in this demo is intentional and documented. The script creates Demo.OrderLinesSkewed using WideWorldImporters’ approximately 231,000 existing order lines, then adds another 500,000 rows tied to a single customer. The goal is to create a simple scenario where one customer has a large percentage of the data while most other customers have significantly fewer rows.

The customer values are not hardcoded. The script identifies the large customer and the smaller customers from the data and prints them out. This makes the demo more portable because WideWorldImporters installations may not all contain identical data.

I prefer demos that are transparent about how the test conditions were created. The important part is not the test data itself. The important part is understanding the SQL Server behavior we are trying to demonstrate.

There are three important details in the setup that make this demo work. Each one represents a common reason why performance demos like this can produce misleading results:

The index is narrow on purpose. A single non-clustered index on CustomerID, no INCLUDE columns:

CREATE NONCLUSTERED INDEX IX_OrderLinesSkewed_CustomerID
    ON Demo.OrderLinesSkewed (CustomerID);

CREATE NONCLUSTERED INDEX IX_OrderLinesSkewed_CustomerID
    ON Demo.OrderLinesSkewed (CustomerID);

TThe goal is to force SQL Server to make a real choice. It can either use the index and perform key lookups for each row, or decide that scanning the clustered index is the better option. Where SQL Server draws that line is the plan shape side of the problem.

If the index is covering, that decision goes away. You may still see a memory grant issue, but you will not see the plan change between executions. The result is a demo that only shows one side of the problem and misses how parameter sensitivity can affect plan selection.

The rows are intentionally wide. There is a char(200) filler column, and the stored procedure includes that column in the output. Memory grants are calculated using estimated rows multiplied by estimated row width. If the rows are too narrow, the memory grant behavior is not very interesting.

There is also no TOP and no ROW_NUMBER() in this demo. This is an important detail because many demos around this topic accidentally hide the memory grant problem.

For example, a query like SELECT TOP (50) ... ORDER BY UnitPrice DESC introduces a Top N Sort. The memory grant for a Top N Sort is based on the number of rows being returned, in this case 50, instead of the total number of rows flowing through the sort. Filtering a ROW_NUMBER() value against a constant can have a similar issue because the optimizer may rewrite it into a Top.

In both cases, the memory grant no longer scales with the actual workload. The demo may still run, but it is no longer showing the behavior we are trying to analyze.

If you build your own version of this test, check the execution plan and make sure the operator is a Sort and not a Top N Sort. The demo scripts capture the plan after each execution and flag Top N Sort because it changes the behavior being tested.

Here’s the procedure. It is deliberately boring:

CREATE OR ALTER PROCEDURE Demo.usp_CustomerLinesByPrice
    @CustomerID int
AS
BEGIN
    SET NOCOUNT ON;

    SELECT  ol.OrderLineID, ol.OrderID, ol.CustomerID, ol.StockItemID,
            ol.Description, ol.Quantity, ol.UnitPrice, ol.OrderDate,
            ol.Filler
    FROM    Demo.OrderLinesSkewed AS ol
    WHERE   ol.CustomerID = @CustomerID
    ORDER BY ol.UnitPrice DESC, ol.Description;
END

CREATE OR ALTER PROCEDURE Demo.usp_CustomerLinesByPrice
    @CustomerID int
AS
BEGIN
    SET NOCOUNT ON;

    SELECT  ol.OrderLineID, ol.OrderID, ol.CustomerID, ol.StockItemID,
            ol.Description, ol.Quantity, ol.UnitPrice, ol.OrderDate,
            ol.Filler
    FROM    Demo.OrderLinesSkewed AS ol
    WHERE   ol.CustomerID = @CustomerID
    ORDER BY ol.UnitPrice DESC, ol.Description;
END

The test is simple by design. One equality predicate against a skewed column. One sort operation where no index can fully support it. Those two things are enough to reproduce the behavior we want to analyze.

Failure mode one: sniff small, run big

Compile the procedure for the minnow. Then call it for the whale.

EXEC sys.sp_recompile N'Demo.usp_CustomerLinesByPrice';
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Minnow;  -- compiles here
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Whale;   -- suffers here

EXEC sys.sp_recompile N'Demo.usp_CustomerLinesByPrice';
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Minnow;  -- compiles here
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Whale;   -- suffers here

The plan compiled for the minnow is a good plan for the minnow: seek the non-clustered index, look up the handful of matching rows in the clustered index, sort them in a memory grant barely above the minimum. For a few hundred rows that’s exactly right.

Then the whale arrives, and the same plan does it 500000 times.

Two separate things have now gone wrong, and from here on I’m going to insist on naming them separately.

The plan shape is wrong. Key lookups are fine in the hundreds and catastrophic in the hundreds of thousands. The logical read count tells the story: 1519167 reads to return 500000 rows. A clustered index scan would have read the table roughly once.

The memory grant is wrong, and this is the part people find surprising. The grant is not recalculated per execution. It is baked into the cached plan at compile time, computed from the estimated row count and the estimated row width. Runtime reality does not get a vote. So the sort gets a grant sized for the minnow — 1 MB — while the engine’s own after-the-fact assessment of what it should have had is 0.53 MB.

When a sort doesn’t have enough memory, it spills to tempdb. Not a warning, not a retry — it writes sort runs to disk and merges them, and your query goes from memory-speed to disk-speed while holding its locks the whole time. In the actual execution plan you’ll see a warning triangle on the Sort operator. In the Extended Events output you’ll see sort_warning fire.

The gap between GrantMB and IdealMB is the fingerprint. Learn to read it.

Failure mode two: sniff big, run small

Now the mirror image, which most write-ups skip, and which is the more interesting half.

EXEC sys.sp_recompile N'Demo.usp_CustomerLinesByPrice';
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Whale;   -- compiles here
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Minnow;  -- wastes memory here

EXEC sys.sp_recompile N'Demo.usp_CustomerLinesByPrice';
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Whale;   -- compiles here
EXEC Demo.usp_CustomerLinesByPrice @CustomerID = @Minnow;  -- wastes memory here

The plan compiled for the whale is a clustered index scan with a memory grant sized for half a million wide rows. Reused for the minnow, it returns a few hundred rows and finishes quickly.

Nothing spills. Nothing is slow. This query will never appear in your “top ten by duration” report. It is not broken in any way a duration-based monitor can see.

It is, however, greedy. It asked for 127.31 MB of workspace memory and touched 0.22 MB of it.

Here’s why you should care about memory a query didn’t use:

• A memory grant is reserved for the lifetime of the query, used or not. It is not lazily allocated and it is not shared.
• Workspace memory is a finite, instance-wide pool. There is only so much of it.
• When the pool is exhausted, incoming queries queue on RESOURCE_SEMAPHORE waits — they sit there, having compiled successfully, waiting for permission to start.

So one procedure with a badly sniffed grant, called from enough sessions concurrently, will stall queries that have nothing to do with it. The victim is never the culprit. That’s what makes this one hard to trace back, and it’s why duration is a bad detector for half of all parameter sniffing problems.

The detector that does work is a comparison, not a threshold:

SELECT  qs.execution_count,
        GrantMB = qs.last_grant_kb      / 1024.0,
        UsedMB  = qs.last_used_grant_kb / 1024.0,
        IdealMB = qs.last_ideal_grant_kb/ 1024.0,
        st.text
FROM    sys.dm_exec_query_stats AS qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) AS st
WHERE   qs.last_grant_kb > 1024
  AND   qs.last_grant_kb > qs.last_used_grant_kb * 2
ORDER BY qs.last_grant_kb DESC;

SELECT  qs.execution_count,
        GrantMB = qs.last_grant_kb      / 1024.0,
        UsedMB  = qs.last_used_grant_kb / 1024.0,
        IdealMB = qs.last_ideal_grant_kb/ 1024.0,
        st.text
FROM    sys.dm_exec_query_stats AS qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) AS st
WHERE   qs.last_grant_kb > 1024
  AND   qs.last_grant_kb > qs.last_used_grant_kb * 2
ORDER BY qs.last_grant_kb DESC;

Granted far above used means memory reserved and wasted. Ideal far above granted means the query spilled. Same three columns, two different diseases.

Two problems, not one

This is the pivot of the whole post, so here it is in one table:

Everything below refers back to this.

OPTION (RECOMPILE)

The blunt instrument, and the one that always works.

    ...
    ORDER BY ol.UnitPrice DESC, ol.Description
    OPTION (RECOMPILE);

    ...
    ORDER BY ol.UnitPrice DESC, ol.Description
    OPTION (RECOMPILE);

Both columns of that table go green. The plan is built against the actual parameter value on every single execution, so the shape is right and the grant is right, every time, by construction. You get a bonus, too: because the value is a literal constant at compile time, the optimizer can do things it can’t do for a cached plan — fold constants, eliminate whole branches of a query, simplify predicates it would otherwise have to keep general.

In the demo, scenario D calls the recompiling version with the minnow, then the whale, then the minnow again, and the grant tracks the actual row count in both directions: 1, 127.35, 127.34 MB.

Now the cost, stated honestly, because “just add RECOMPILE” is bad advice delivered confidently.

You pay a compilation on every execution. Compilation is CPU-expensive. On a procedure called four thousand times a minute, you have traded an intermittent memory problem for a permanent CPU problem, and the second one is harder to notice because it doesn’t spike — it just raises your floor.

You lose the plan cache as a diagnostic surface. After a recompiling statement runs, there’s nothing in sys.dm_exec_cached_plans to look at. Your monitoring gets thinner exactly where you were having trouble.

Statement-level, not procedure-level. OPTION (RECOMPILE) on one statement recompiles that statement. CREATE PROCEDURE ... WITH RECOMPILE recompiles every statement in the procedure on every call, including the nine that were fine. It is almost never what you want. If you inherited a procedure with WITH RECOMPILE in the header, that’s usually someone’s decade-old shotgun fix for a problem in one statement.

The rule of thumb worth internalizing: RECOMPILE is priced per execution. A reporting procedure called forty times an hour should probably just use it and stop thinking about this. A hot OLTP path called forty times a second should not.

And note something for later: after scenario D there is no cached plan at all. Hold that thought.

Memory grant feedback

Now the feature everyone wants to talk about, and the one whose limits are routinely oversold.

Memory grant feedback is a learning loop. After a query executes, the engine compares the memory it granted against the memory the query actually used. If the query spilled, the grant was too small — write a bigger number onto the cached plan for next time. If the grant was more than twice what was used, it was too big — write a smaller one. Over a handful of executions the grant converges on reality without anybody recompiling anything.

What you need for it:

That compatibility level column is where most people get stuck. A database restored from an older instance keeps its old compatibility level forever; WideWorldImporters ships at 130. You can be running SQL Server 2022 and getting none of this.

Scenario C in the demo sets the loop up to succeed: compile for the minnow, then call the whale six times in a row with nothing recompiling in between. That last part matters. Feedback is written onto the cached plan, so anything that evicts the plan throws away everything the engine learned. This is also why the adjustment always lands on the following execution — execution n discovers the grant was wrong, execution n+1 benefits.

The trajectory:

That State column is IsMemoryGrantFeedbackAdjusted from the cached plan’s XML, and it’s the cleanest way to watch the loop work: it moves from NoFirstExecution through YesAdjusting to YesStable.

Now the three caveats, which are the actual reason this section exists.

It fixes the grant. It never fixes the plan shape. Look at the PlanShape column across all six of those executions in the demo output. It does not change. It cannot change — memory grant feedback adjusts a number attached to an existing plan; it does not trigger a recompilation and it has no opinion about operators. Those 1519167 logical reads from failure mode one are still there on execution six. The query stops spilling and gets faster. It does not get good. If you go into this expecting feedback to solve parameter sniffing, this is where you’ll be disappointed, and it won’t be the feature’s fault.

It’s a learning loop, so somebody has to do the learning. The first caller always eats the bad grant. On SQL Server 2019 and earlier, so does the first caller after any cache eviction — a plan flush, memory pressure, a stats update, a failover. SQL Server 2022’s Query Store persistence exists precisely to stop throwing that lesson away, and it’s a good reason to have Query Store on.

It gives up if you make it thrash. A workload that genuinely alternates between tiny and enormous will push the grant up, then down, then up again. Rather than oscillate forever, the engine notices the instability and switches feedback off for that query. There’s an Extended Event for it — memory_grant_feedback_loop_disabled. Percentile grant feedback in SQL Server 2022 is the answer to this case: instead of chasing the last execution, it sizes the grant from a percentile of recent executions, which is far more stable across a genuinely bimodal workload.

Why RECOMPILE and memory grant feedback don’t combine

This falls straight out of the two sections above, and it’s the question that sent me down this path in the first place.

Memory grant feedback writes its correction onto a cached plan. OPTION (RECOMPILE) doesn’t leave a cached plan. There is nothing for the feedback to attach to.

You can watch this in the demo: after scenario D runs three times, query the cached plan view and you get nothing back. Compare with scenario C, where the plan is sitting right there accumulating adjustments.

This is not a conflict you need to resolve, and it isn’t a bug. RECOMPILE already produces an accurate grant on every execution by construction — there’s nothing left for a feedback loop to improve. But it does mean the two are alternatives, not layers. Don’t reach for RECOMPILE while imagining that feedback is also working quietly underneath, and don’t diagnose the absence of feedback on a recompiling statement as something being broken.

Parameter Sensitive Plan optimization

Which leaves the gap in that table from earlier: memory grant feedback never fixes plan shape, and RECOMPILE fixes plan shape but charges you per execution. Is there anything that fixes the shape without the compile?

On SQL Server 2022, yes. Parameter Sensitive Plan optimization caches multiple plan variants for a single statement and dispatches between them based on the cardinality the incoming parameter implies. The minnow gets the seek-and-lookup plan, the whale gets the scan, neither one triggers a compilation, and both come out of cache.

It’s on by default at compatibility level 160. Its limits are worth knowing: equality predicates only, at most three of them, and the column has to be skewed enough for the engine to consider it worth the trouble — PSPO is not applied to every parameterized query, only to ones where the optimizer sees a genuine sensitivity.

The most convincing thing I can say about PSPO is not an argument, it’s a confession about the demo: the setup script has to turn PSPO off. On a 2022 instance at compatibility level 160, scenarios A and B don’t fail. The engine handles them. I had to explicitly disable the feature to show you the classic behaviour at all:

ALTER DATABASE SCOPED CONFIGURATION
    SET PARAMETER_SENSITIVE_PLAN_OPTIMIZATION = OFF;

ALTER DATABASE SCOPED CONFIGURATION
    SET PARAMETER_SENSITIVE_PLAN_OPTIMIZATION = OFF;

Scenario E turns it back on so you can see the contrast: same procedure, same two parameters, no recompilation between them, two different plan shapes.

Things that look like fixes

A tour of the mitigations you’ll find in older Stack Overflow answers, and what they actually cost.

OPTIMIZE FOR UNKNOWN. Compiles using the density vector — the average rows per distinct value — instead of the histogram. You’ve traded a plan that is excellent for most callers and terrible for a few, for a plan that is mediocre for everyone. Sometimes that really is the right trade, especially when the terrible case is bad enough to cause an outage. But it should be a decision, not a reflex.

OPTIMIZE FOR (@CustomerID = 12345). You’ve pinned the plan to a magic constant. It works until the data distribution moves, at which point it fails silently and nobody remembers that number is in there.

Assigning parameters to local variables. The folk-remedy version of OPTIMIZE FOR UNKNOWN — the optimizer can’t sniff a local variable, so you get the density estimate. Same trade-off, but now it’s invisible, and the next developer will “clean up” the pointless variable assignment and reintroduce the bug.

Updating statistics, or rebuilding the index. This evicts plans, so the symptom goes away, so it looks like a fix. It will look like a fix again next week, and the week after, forever. This is the single most common way a parameter sniffing problem survives for years: it is permanently one maintenance job away from being invisible.

Restarting SQL Server. Same mechanism, more downtime, and it usually happens at 3am while someone types “resolved” into a ticket.

Splitting the procedure into branches — an IF that routes big customers to one procedure and small ones to another, so each gets its own plan. This one genuinely works. It also hardcodes today’s understanding of your data into control flow, and it ages badly. Reach for it when PSPO isn’t available and RECOMPILE is too expensive, and leave a comment explaining why.

So what should you actually do?

1. Confirm the column is skewed. Group by the predicate column, compare the top and bottom. Within an order of magnitude? It isn’t parameter sniffing. Go look somewhere else.
2. Work out which problem you have. Compare last_grant_kb, last_used_grant_kb, and last_ideal_grant_kb against last_logical_reads. Wrong grant, wrong shape, or both.
3. Grant only, on 2019 or later, with steady traffic — check your compatibility level is 150+ and let memory grant feedback handle it. Turn on Query Store if you’re on 2022, so the lesson survives an eviction.
4. Shape wrong, on 2022 — check whether PSPO is on before you write any code. You may not have a problem.
5. Shape wrong, low call rate — OPTION (RECOMPILE) on the statement. Measure the compile cost afterwards rather than assuming it’s fine.
6. Shape wrong, high call rate, no PSPO available — branch the procedure, and write down why in a comment, because in three years the reason will not be obvious.

The thing I’d most like you to take away is the second step. Almost everything written about parameter sniffing collapses the two failure modes into one story, and once you’ve separated them the modern features stop looking mysterious. Memory grant feedback isn’t under-delivering — it’s doing exactly the one job it claims to do, and PSPO is the feature that does the other one.

Run it yourself

Download the sql files from my repo (see link below):

01-setup.sql     run once, builds the skewed table and the procedure
02-demo.sql      run the whole file, all five scenarios, records its own evidence
03-cleanup.sql   restores everything it changed

Two things before you do. Turn on Query Options → Results → Grid → Discard results after execution in SSMS, because the procedure returns half a million wide rows about ten times over and you want to be timing the server rather than the grid. And know that 01-setup.sql raises your database compatibility level and turns PSPO off — both are recorded before they’re changed and restored by the cleanup script, but point it at a scratch instance, not production.

The demo captures every measurement into Demo.DemoResults as it goes, so you don’t have to sit and read execution plans between executions. The summary at the bottom flags spills and wasted grants for you.

Dowload the demo scripts from my github repo.

The post Demo for Parameter Sniffing and Memory Grant Feedback first appeared on SQL, Code, Coffee, Etc..

#temp tables can be useful in so many ways. I hadn’t really given them much thought beyond using them when a situation called for it, until I saw the invitation for T-SQL Tuesday #201 Invitation: Temp Tables, Friend or Foe? This month’s T-SQL Tuesday is hosted by Jeff Taylor (b).

For this post, I needed to do some research because there is more to #temp tables than simply creating one and using it. As I started digging into the topic, I realized there are a lot of discussions around #temp tables and @table variables, especially around when one might be a better choice over the other.

One of the common arguments is that #temp tables go to disk while @table variables stay in memory. Another argument is that this is a myth. That got me curious about what the actual differences are and when they really matter. Discussions like this almost always starts with where the data is stored (disk vs memory).

Both #temp tables and @table variables use tempdb. Both allocate pages that are managed by the buffer pool. Depending on the size of the data, memory availability, and other factors, those pages may or may not ever be written to disk. If that’s the case, then maybe “memory versus disk” isn’t really the question we should be asking.

Let’s take a look at a couple of scenarios.

Test SQL: SQL Server 2022 CU25 (16.0.4255.1) container (Orbstack on Macbook pro)
CPU: 8 (schedulers)
MAXDOP: 0
Cost Threshold for Parallelism: Default 5 (for parallelism test scenario)

The difference is statistics

One of the biggest differences between #temp tables and @table variables is statistics. #temp tables can have statistics created and maintained by SQL Server, which gives the optimizer more information when generating an execution plan. @table variables do not have column statistics, although newer versions of SQL Server have improved their cardinality estimates through deferred compilation.

#temp tables can have statistics. Those statistics include histograms that help the optimizer understand how the data is distributed and make better estimates.

A table variable does not get column statistics. Even with newer versions of SQL Server and improvements like deferred compilation, the optimizer still does not have the same level of information that it has with a #temp table.

This is where many of the differences start to show up. Things like indexes, constraints, recompiles, and parallelism are often related to the fact that a #temp table is a temporary table with metadata that the optimizer can use, while a table variable has different behavior.

The rule I am starting to use from this research is simple:

Use a #temp table when the optimizer needs more information about the data. Use a table variable when the amount of data is small, the usage is simple, or you specifically need the behavior that a table variable provides.

Let’s take a look at a simple test.

I wanted to keep this as fair as possible. Both objects have the same structure, the same data, and the same nonclustered index. All the test scripts can be found in the Github repo (see link at the bottom of this post).

CREATE TABLE #temptable 
(
    id int NOT NULL, 
    grp int NOT NULL, 
    INDEX ix_grp NONCLUSTERED (grp)
);

DECLARE @tablevariable TABLE 
(
    id int NOT NULL, 
    grp int NOT NULL, 
    INDEX ix_grp NONCLUSTERED (grp)
);

CREATE TABLE #temptable 
(
    id int NOT NULL, 
    grp int NOT NULL, 
    INDEX ix_grp NONCLUSTERED (grp)
);

DECLARE @tablevariable TABLE 
(
    id int NOT NULL, 
    grp int NOT NULL, 
    INDEX ix_grp NONCLUSTERED (grp)
);

The data is intentionally skewed. grp = 1 has 9,000 rows, while grp = 2 has only 10 rows.

When I checked the estimated row counts, the difference was obvious.

The #temp table had statistics, so the optimizer had information about the data distribution. The estimates were close to the actual row counts.

The table variable was different. Even though it had the same index, the optimizer did not have the same information available. The index gave it something to seek on, but it did not tell the optimizer how the data was distributed.

That distinction matters because estimates influence the rest of the execution plan. Row estimates affect join choices, memory grants, and other optimizer decisions. If SQL Server estimates 100 rows but the query actually returns 9,000 rows, the optimizer may choose a plan that isn’t optimal for the actual workload. That can also lead to memory spills to tempdb if the memory grant is too small.

One thing I noticed while testing this is that the estimate for a table variable is not always the same. For example, removing the index can change the estimate. The important part is that the optimizer still has limited information about the data inside a table variable.

So what about SQL Server 2019 and table variable deferred compilation?

Deferred compilation helps because SQL Server can see the table variable row count before compiling the statement. This improves cardinality estimates in many scenarios.

However, it does not create statistics. Knowing that a table variable has 10,000 rows is different from knowing how those 10,000 rows are distributed. A row count can help, but it does not replace a histogram.

Table variables migh have a problem with parallelism

Note: It is quite complicated to test paralellism on my test environment. Some tests results returned “inconclusive”. Your mileage varies (depending on your config basically).

I also looked at how #temp tables and @table variables behave when it comes to parallelism.

A common statement is: “@table variables are always serial.” That may not be completely accurate. But again, I have not tested this extensively. FYI.

The restriction is related to modifying the table variable, not reading from it. For example, a regular SELECT or a query that reads from a table variable can still use parallelism. The limitation shows up when SQL Server is inserting into or modifying the table variable. I need to test this with bigger workload when I get the chance.

Again, this is not conclusive.

Download the test scripts to see how this looks in your environment. Link to my Github repo is at the bottom of this post.

The impact really depends on how much data you are working with.

If you are inserting a small number of rows into a table variable and reading from it later, the lack of parallelism during the insert probably does not matter. The difference between a serial and parallel insert for a small dataset is not something you will likely notice.

Where this becomes important is when you start loading a large amount of data. A table variable that is used as a staging area for thousands or millions of rows can limit the insert operation to a serial plan when a #temp table could take advantage of parallelism.

For small amounts of data, use whichever option makes the code easier to understand. For larger data loads, a #temp table is usually worth considering.

The test scenario used in this post covers only a couple of the differences between #temp tables and @table variables. If you want to experiment with the scenarios yourself, you can grab the test scripts from my GitHub repository and run them in your own environment.

https://github.com/MarlonRibunal/sqlserver-demos/tree/main/tempdb

The post When Do You Reach For A #temp Table? first appeared on SQL, Code, Coffee, Etc..