TSQL Tuesday

The second Tuesday of the month is upon us once again. That means it is time for another group blog party called TSQLTuesday. This party that was started by Adam Machanic has now been going for long enough that changes have happened (such as Steve Jones (b | t) managing it now). For a nice long read, you can find a nice roundup of all TSQLTuesdays over here.

This month, just about every SQL Server Data professional is being asked to get out of their comfort zone and talk about Linux (there are some out there that are very comfortable with Linux, but the vast majority are not ).

This is a topic that is near and dear to the heart of our organizer Tracy Boggiano (b | t). In Tracy’s invite, she says “While I know it takes a while to adopt new technologies; I was wondering what it would take for people to adopt SQL on Linux.  Alternating I’m offering up for you to blog about what everyone should know when working with SQL on Linux or anything else related to SQL running on Linux.”

That pretty much leaves the door wide open, right?

Event Files

For the most part, things work the way you might expect them to work in windows – except it is on Linux. Sure some things are different, but SQL Server itself, is largely the same. That similarity, for the most part, boils all the way down into Extended Events (XEvents) as well. There is one significant divergence, however. And that is when it comes to specifying your event_file target for a new session. Let’s dive into that just a bit.

Let’s take a common setup for an XEvent session.

USE master;
GO
-- Create the Event Session
/* this method will fail. The session will create but the session cannot be started due to a non-existent directory 
However, if the directory exists in linux format, and permissions are in place, the session will start.
*/
IF EXISTS
	(
		SELECT
				  *
			FROM  sys.server_event_sessions
			WHERE name = 'PREEMPTIVE_OS_PIPEOPS'
	)
	DROP EVENT SESSION PREEMPTIVE_OS_PIPEOPS ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE\';
GO

CREATE EVENT SESSION [PREEMPTIVE_OS_PIPEOPS]
	ON SERVER
	ADD EVENT sqlos.wait_info_external(
		 WHERE ([wait_type] = 'PREEMPTIVE_OS_PIPEOPS')
		)
	ADD TARGET package0.event_file
		(SET filename = N'C:\Database\XE\PREEMPTIVE_OS_PIPEOPS.xel')
	WITH
		(
			MAX_MEMORY = 4096KB
		  , EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
		  , MAX_DISPATCH_LATENCY = 5 SECONDS
		  , TRACK_CAUSALITY = ON
		  , STARTUP_STATE = ON
		);
GO
ALTER EVENT SESSION Preemptive_os_pipeops ON SERVER
STATE = START;
GO

This will fail before the query really even gets out of the gate. Why? The proc xp_create_subdir cannot create the directory because it requires elevated permissions. The fix for that is easy enough – grant permissions to write to the Database directory after creating it while in sudo mode. I will get to that in just a bit. Let’s see what the errors would look like for now.

Msg 22048, Level 16, State 1, Line 15
xp_create_subdir() returned error 5, ‘Access is denied.’
Msg 25602, Level 17, State 23, Line 36
The target, “5B2DA06D-898A-43C8-9309-39BBBE93EBBD.package0.event_file”, encountered a configuration error during initialization. Object cannot be added to the event session. The operating system returned error 5: ‘Access is denied.
‘ while creating the file ‘C:\Database\XE\PREEMPTIVE_OS_PIPEOPS_0_132072025269680000.xel’.

Let’s resolve the folder issue. I will create the Database folder (from a console session on the server), and then I will take ownership of that folder.

$ sudo mkdir /Database/

$ sudo chown -R $User:$User /Database/

$ sudo chgrp mssql /Database/

$ chmod g=rwx /Database/

From there, it is also advisable to grant permissions to this folder to the SQL group via the chgrp command. Once done, re-running the entire session creation will magically work – including that windows based create subdir proc.

Alternative #2

Maybe we just want to do things via the default method. If so, we can do this and it just works.

USE master;
GO
-- Create the Event Session
/* this method succeeds but uses the default log directory */
IF EXISTS
	(
		SELECT
				  *
			FROM  sys.server_event_sessions
			WHERE name = 'PREEMPTIVE_OS_PIPEOPS2'
	)
	DROP EVENT SESSION PREEMPTIVE_OS_PIPEOPS2 ON SERVER;
GO

CREATE EVENT SESSION [PREEMPTIVE_OS_PIPEOPS2]
	ON SERVER
	ADD EVENT sqlos.wait_info_external(
		 WHERE ([wait_type] = 'PREEMPTIVE_OS_PIPEOPS')
		)
	ADD TARGET package0.event_file
		(SET filename = N'PREEMPTIVE_OS_PIPEOPS.xel')
	WITH
		(
			MAX_MEMORY = 4096KB
		  , EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
		  , MAX_DISPATCH_LATENCY = 5 SECONDS
		  , TRACK_CAUSALITY = ON
		  , STARTUP_STATE = ON
		);
GO

Alternative #3

And of course, there is always this option. Unfortunately, this means keeping multiple session scripts in source control in order to maintain the differences between Windows and Linux as illustrated in these two examples.

USE master;
GO
-- Create the Event Session
/* this method succeeds but requires the permissions to be modified just the same as in the first example */
IF EXISTS
	(
		SELECT
				  *
			FROM  sys.server_event_sessions
			WHERE name = 'PREEMPTIVE_OS_PIPEOPS3'
	)
	DROP EVENT SESSION PREEMPTIVE_OS_PIPEOPS3 ON SERVER;
GO

CREATE EVENT SESSION [PREEMPTIVE_OS_PIPEOPS3]
	ON SERVER
	ADD EVENT sqlos.wait_info_external(
		 WHERE ([wait_type] = 'PREEMPTIVE_OS_PIPEOPS')
		)
	ADD TARGET package0.event_file
		(SET filename = N'/Database/XE/PREEMPTIVE_OS_PIPEOPS3.xel') --this is correct and matches the same pattern that ssms uses when using linux filesystem
	WITH
		(
			MAX_MEMORY = 4096KB
		  , EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
		  , MAX_DISPATCH_LATENCY = 5 SECONDS
		  , TRACK_CAUSALITY = ON
		  , STARTUP_STATE = ON
		);
GO

With all three sessions now on the server, I can query my target paths to confirm the correct paths have been used.

SELECT ses.name AS SessionName, CASE WHEN ISNULL(xs.name,'') = ''
		THEN 'No' ELSE 'Yes' END AS ISStarted
		, xs.total_bytes_generated
		,REPLACE(CONVERT(NVARCHAR(128),sesf.value),'.xel','*.xel') AS targetvalue, ses.event_session_id
				FROM sys.server_event_sessions ses
					INNER JOIN sys.server_event_session_fields sesf
						ON ses.event_session_id = sesf.event_session_id
					LEFT OUTER JOIN sys.dm_xe_sessions xs
						ON xs.name = ses.name
				WHERE sesf.name = 'filename'
					AND ses.name LIKE 'PREEMPTIVE_OS_PIPEOPS%';

This produces the following results for me.

Perfect, everything looks to be working and properly configured. Well, except for that session that is using the default directory – gag.

Wrapping it Up

Extended Events is a powerful tool with plenty of ease of use and flexibility. This flexibility allows the DBA to capably monitor the server for any issue be it small or large. This article demonstrated how to create an Event Session on SQL Server that is running on Linux. If you are in need of a little tune-up for your XE skiils, I recommend reading a bit on Extended Events to get up to date. For other basics related articles, feel free to read here.

Does anybody know who or what is using this database? Better yet, is the database being used by anything at all? Have you ever asked those types of questions of your staff in an environment you have inherited?

As silly as it might sound, these are legitimate questions. It is not uncommon to have a relic database hanging around after an application has long since been retired. The database just continues to live on via life support in the dark recesses of your database server. Everybody has forgotten about it yet it consumes precious resources to do little more than exist.

But how do you go about discovering if the database is truly required or if it is just chewing up space and cpu cycles? The answer is to start by discovering if there is any activity in the database at all. Extended Events is an ideal method to audit your database and discover if there is truly activity on the database or not.

DB In Use?

Depending on the version of SQL Server you are using, the method is going to be a little bit different. Each method I use still relies on Extended Events (XE), they just require some slight changes and subtle nuances (particularly for those pesky 2008 and R2 instances out there). Let’s focus on the 2008 method this time around and we will revisit the updated version in a later article.

2008/R2

Let’s first start with the XE session that works well for discovering the frequency of use a database may be encountering.

/* 2008R2 */
USE master;
GO
-- Create the Event Session
IF EXISTS (SELECT * FROM sys.server_event_sessions WHERE name = 'DBinUse')
	DROP EVENT SESSION DBinUse ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE';
GO

CREATE EVENT SESSION DBinUse
	ON SERVER
	ADD EVENT sqlserver.database_transaction_begin
		( --Occurs when a database transaction begins. Count of transactions at the database level that "start"
		 ACTION
			 (
				 sqlserver.database_id
			   , sqlserver.database_context --action is deprecated in later versions
			 )
		)
	ADD TARGET package0.asynchronous_bucketizer
		(SET source_type = 1, source = N'sqlserver.database_id') --2008r2 requires that the package name be specified with the action
	WITH
		(
			MAX_DISPATCH_LATENCY = 5 SECONDS
		  , TRACK_CAUSALITY = ON
		  , STARTUP_STATE = ON
		);


/* start the session */
ALTER EVENT SESSION DBinUse ON SERVER STATE = START;
GO

Inline with the script, I left some interesting notes. First, the note in the target section is of particular interest because it lays out a requirement for this kind of target in 2008/R2. When using the action source type, then the package name must be specified along with the action in two-part naming format.

Next, in the actions, I have a note about sqlserver.database_context. This particular action is deprecated in later versions. Do I need it in this session? No. I put it in there for additional troubleshooting/exploration.

Lastly, I have a note about the event name. I chose database_transaction_begin because this is fired when a database transaction starts. I don’t really care if the transaction completes. I just want to know if an attempt was made to use that database and the data therein.

If I comment out the deprecated action (database_context), I could actually use this session on later versions (as constructed) of SQL Server. It doesn’t matter that the asynchronous_bucketizer has been replaced by the histogram target, the session will still create and work properly. SQL Server knows to automatically update the deprecated targets with the appropriate target when creating an XE Session on a server.

Since this is 2008/R2, we have to use TSQL in order to parse the data. The following query will do that for us!

WITH dbusepreprocess AS
	(SELECT
		 CAST([target_data] AS XML) AS target_data
	   , xes.event_name
		 --INTO #xmlpreprocess
		 FROM sys.dm_xe_session_targets			 AS xt
			 INNER JOIN sys.dm_xe_sessions		 AS xs
				 ON xs.address = xt.event_session_address
			 INNER JOIN sys.dm_xe_session_events xes
				 ON xes.event_session_address = xt.event_session_address
		 WHERE xs.name = N'DBInUse'
			   AND xt.target_name = N'asynchronous_bucketizer')
	SELECT
		d.name AS DBName
	  , summary.dbuse_count
	  , summary.event_name
		FROM
			(
				SELECT
					hist.target_data.value('(value)[1]', 'bigint')	AS database_id
				  , hist.target_data.value('(@count)[1]', 'bigint') AS dbuse_count
				  , pp.event_name
					FROM dbusepreprocess									   pp
						CROSS APPLY target_data.nodes('BucketizerTarget/Slot') AS hist(target_data)
			)						 summary
			RIGHT OUTER JOIN sys.databases d
				ON summary.database_id = d.database_id;

After executing that query, I will see results listing the activity of each database since the session was started. Here’s an example.

Here we can see there are indeed some databases that are still in use on this server. If we are looking to retire the instance, or migrate relevant databases to a new server, we have better information about how to go about planning that work. Based on this data, I would be able to retire the ProtossZealot database but would be required to do something with the ZergRush and BroodWar databases. In the case of the ProtossZealot database that is not being used, we now have evidence to present back to the team that the database is not used. It is now up to those insisting on keeping it around to justify its existence and document why it must remain in tact.

Wrapping it Up

Extended Events is a powerful tool with plenty of ease of use and flexibility. This flexibility allows the DBA to capably monitor the server for any issue be it small or large. This article demonstrates how to use Extended Events to determine if a database is being used by someone or something. If you are in need of a little tune-up for your XE skiils, I recommend reading a bit on Extended Events to get up to date. For other basics related articles, feel free to read here.

In addition, through the power of XE, we were able to capture a quick glimpse into some internals related to the database engine. In this case, we were able to see autoshrink in action and discover the frequency that it runs on when enabled.

There are multiple ways of retrieving data from a table when you have a list of values. Sometimes, the method is straight-forward and sometimes we just can’t see the right solution.

Some of the methods to retrieve data from a table (based on a list of values) might include one or multiple of the following: join to another table, throwing an in clause at it, dumping the values into a temp table, sub-queries, and a table value constructor.

Let’s focus on the table value constructor. This one seems to trip a few people up from time to time. First off, what is it?

Table Value Constructor

The table value constructor is basically like a virtual table not too different from a CTE or a subquery (in that they are all virtual tables of sorts). The table value constructor however can be combined with either of those other types and is a set of row expressions that get put into this virtual table in a single DML statement.

So, this virtual table contains a set of values that we define. Let’s go ahead and take a look at an example set of data that we can cultivate into a table value constructor.

VALUES
		(14)
	, (15)
	, (18)
	, (20)
	, (102)
	, (103)
	, (104)
	, (105)
	, (106)
	, (107)
	, (108)
	, (109)
	, (110)
	, (111)
	, (112)
	, (113)
	, (115)
	, (116)
	, (117)
	, (118)
	, (128)
	, (129)
	, (130)
	, (131)
	, (132)
	, (133)
	, (134)
	, (135)
	, (152)
	, (153)
	, (170)
	, (171)
	, (172)
	, (173)
	, (175)
	, (176)
	, (177)
	, (178)

Now, let’s say that I want to get all of the trace events that might correlate to those specific values, all I need to do is add this list of values to a query in the appropriate place. First, let’s see what it looks like when I add it via a CTE.

WITH traceevents (EventID) AS
	(SELECT
			 *
		 FROM
			 (
				 VALUES
					 (14)
				   , (15)
				   , (18)
				   , (20)
				   , (102)
				   , (103)
				   , (104)
				   , (105)
				   , (106)
				   , (107)
				   , (108)
				   , (109)
				   , (110)
				   , (111)
				   , (112)
				   , (113)
				   , (115)
				   , (116)
				   , (117)
				   , (118)
				   , (128)
				   , (129)
				   , (130)
				   , (131)
				   , (132)
				   , (133)
				   , (134)
				   , (135)
				   , (152)
				   , (153)
				   , (170)
				   , (171)
				   , (172)
				   , (173)
				   , (175)
				   , (176)
				   , (177)
				   , (178)
			 ) AS it (EventID) )
	SELECT
					   te.EventID
					 , t.name  AS EventName
					 , tc.name AS Category
		FROM		   traceevents			te
			INNER JOIN sys.trace_events		t
				ON t.trace_event_id = te.EventID
			INNER JOIN sys.trace_categories tc
				ON t.category_id = tc.category_id;

In this case, I have thrown the table value constructor into a CTE. There could be a few good reasons to do it this way such as it being easier to read or maybe some corporate style policy. For me, I like the readability and the tight grouping of the join clauses without needing to scroll. Running the query in this case would produce results similar to the following.

Now, what if I really prefer to not use a CTE? Easy enough, let’s look at this next example. I will use the same list of values as I search for various wait_types.

SELECT
	mv.name
  , mv.map_key
  , mv.map_value
	FROM
		(
			VALUES
				(14)
			  , (15)
			  , (18)
			  , (20)
			  , (102)
			  , (103)
			  , (104)
			  , (105)
			  , (106)
			  , (107)
			  , (108)
			  , (109)
			  , (110)
			  , (111)
			  , (112)
			  , (113)
			  , (115)
			  , (116)
			  , (117)
			  , (118)
			  , (128)
			  , (129)
			  , (130)
			  , (131)
			  , (132)
			  , (133)
			  , (134)
			  , (135)
			  , (152)
			  , (153)
			  , (170)
			  , (171)
			  , (172)
			  , (173)
			  , (175)
			  , (176)
			  , (177)
			  , (178)
		)								it (itemid)
		INNER JOIN sys.dm_xe_map_values mv
			ON mv.map_key = it.itemid
	WHERE mv.name = 'wait_types';

Notice here that I have a little bit of a different query. I only have one table to join to after the table value constructor. That said, it works just the same as the CTE example. Once the data is constructed, I alias it like a table, then I define the column names. This query will now return results similar to the following.

Both of these examples have been pretty easy in building a single attribute array for the table value constructor. What if we had the need for multiple columns in our list of data?

SELECT
	mv.name
  , mv.map_key
  , mv.map_value
  , it.WaitStatus
	FROM
		(
			VALUES
				(14,'Active')
			  , (15,'Active')
			  , (18,'Active')
			  , (20,'Active')
			  , (102,'Active')
			  , (103,'Active')
			  , (104,'Active')
			  , (105,'Active')
			  , (106,'Active')
			  , (107,'Active')
			  , (108,'Benign')
			  , (109,'Benign')
			  , (110,'Active')
			  , (111,'Active')
			  , (112,'Active')
			  , (113,'Active')
			  , (115,'Active')
			  , (116,'Active')
			  , (117,'Active')
			  , (118,'Active')
			  , (128,'Active')
			  , (129,'Benign')
			  , (130,'Benign')
			  , (131,'Benign')
			  , (132,'Benign')
			  , (133,'Active')
			  , (134,'Active')
			  , (135,'Active')
			  , (152,'Active')
			  , (153,'Active')
			  , (170,'Active')
			  , (171,'Active')
			  , (172,'Active')
			  , (173,'Active')
			  , (175,'Active')
			  , (176,'Active')
			  , (177,'Active')
			  , (178,'Active')
		)								it (itemid,WaitStatus)
		INNER JOIN sys.dm_xe_map_values mv
			ON mv.map_key = it.itemid
	WHERE mv.name = 'wait_types';

Notice the addition of an extra value (separated by a comma) within each set of parenthesis and the additional column definition added after the alias for the table value constructor. This is a pretty simple modification. Upon execution, I can expect results such as the following.

The Wrap

Being able to take a list of data and construct it into a virtual table for further querying and data analysis is a pretty cool thing to do. The setup and syntax are fairly simple – it just takes a bit of practice and effort.

This article demonstrated three different examples of using a table value constructor to join a list of data to other tables. See what other kinds of examples you can come up with and share them.

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

Recently, I covered the need to understand job failure frequency and knowing the value of your SQL Agent jobs. You can read the specifics in the article – here.

Sometimes, just knowing the frequency of the job failure is good enough. Sometimes, more information is helpful. Having more information is particularly helpful when meeting with the business to discuss the validity of the job in question.

What do you do in times like this? The most basic answer to that question is – get more data. But that barely scratches the surface. The real question being asked there is how do you go about gathering that data?

There are two methods to gather the data – the hard way and the easy way. Do you like to work hard? Or would you rather work more efficiently?

Efficiency Matters

As was discussed in the previous article, I prefer to do things just a little bit less manually where possible. The consistency of a script matters, but it also is just so much faster than doing things the hard, manual, iterative way. So let’s build a little bit on the script from the previous article.

WITH jobhist AS
	(SELECT
			   jh.job_id
			 , jh.run_date
			 , jh.run_status
			 , jh.step_id
			 , ROW_NUMBER() OVER (PARTITION BY jh.job_id, jh.run_status ORDER BY jh.run_date DESC) AS rownum
		 FROM  dbo.sysjobhistory jh
		 WHERE jh.step_id = 0)
   , agglastsixty AS
	(SELECT
						sjh.job_id
					  , sjh.run_status
					  , COUNT(*) AS RunCount
		 FROM			dbo.sysjobhistory sjh
			 INNER JOIN jobhist			  jh
				 ON jh.job_id = sjh.job_id
					AND jh.run_status = sjh.run_status
					AND jh.step_id = sjh.step_id
		 WHERE			CONVERT(DATE, CONVERT(VARCHAR, sjh.run_date)) > DATEADD(
																				   d
																				 , -60
																				 , CONVERT(DATE, CONVERT(VARCHAR, jh.run_date))
																			   )
						AND jh.rownum = 1
		 GROUP BY		sjh.job_id
					  , sjh.run_status)
   , aggtotal AS
	(SELECT
						sjh.job_id
					  , sjh.run_status
					  , COUNT(*) AS RunCount
		 FROM			dbo.sysjobhistory sjh
			 INNER JOIN jobhist			  jh
				 ON jh.job_id = sjh.job_id
					AND jh.run_status = sjh.run_status
					AND jh.step_id = sjh.step_id
		 WHERE			jh.rownum = 1
		 GROUP BY		sjh.job_id
					  , sjh.run_status)
	SELECT
							j.name										 AS JobName
						  , sc.name										 AS CategoryName
						  --	  , sp.name										 AS OwnerName
						  --	  , j.owner_sid
						  , j.date_created
						  , j.enabled
						  , CONVERT(DATE, CONVERT(VARCHAR, oa.run_date)) AS RunDate
						  , CASE oa.run_status
								WHEN 0
									 THEN 'Failed'
								WHEN 1
									 THEN 'Succeeded'
								WHEN 2
									 THEN 'Retry'
								WHEN 3
									 THEN 'Canceled'
								WHEN 4
									 THEN 'In Progress'
							END											 AS run_status
						  , als.RunCount								 AS Last60StatusCount
						  , agt.RunCount								 AS TotalStatusCount
						  , js.subsystem								 AS JobStepSubsystem
						  , js.command
						  , CASE
								WHEN js.subsystem = 'SSIS'
									 AND js.command LIKE '%DECRYPT%'
									 THEN LTRIM(RTRIM(SUBSTRING(
																   js.command
																 , CHARINDEX('/DECRYPT', js.command, 1) + 9
																 , CHARINDEX('/', js.command, CHARINDEX('/DECRYPT', js.command, 1) + 1)
																   - CHARINDEX('/DECRYPT', js.command, 1) - 9
															   )
													 )
											   )
								ELSE 'N/A'
							END											 AS PkgPassword
						  , CASE
								WHEN js.subsystem = 'SSIS'
									 AND js.command LIKE '%FILE%'
									 THEN LTRIM(RTRIM(SUBSTRING(
																   js.command
																 , CHARINDEX('/FILE', js.command, 1) + 9
																 , CHARINDEX('\""', js.command, CHARINDEX('/FILE', js.command, 1) + 1)
																   - CHARINDEX('/FILE', js.command, 1) - 9
															   )
													 )
											   )
								ELSE 'N/A'
							END											 AS PkgPath
						  , spr.name									 AS ProxyName
						  , spr.credential_id
		FROM				dbo.sysjobs			  j
			LEFT JOIN		jobhist				  oa
				ON oa.job_id = j.job_id
			LEFT OUTER JOIN agglastsixty		  als
				ON als.job_id = oa.job_id
				   AND als.run_status = oa.run_status
			LEFT OUTER JOIN aggtotal			  agt
				ON agt.job_id = oa.job_id
				   AND agt.run_status = oa.run_status
			INNER JOIN		sys.server_principals sp
				ON j.owner_sid = sp.sid
			INNER JOIN		dbo.syscategories	  sc
				ON j.category_id = sc.category_id
			INNER JOIN		dbo.sysjobsteps		  js
				ON js.job_id = j.job_id
			LEFT JOIN		dbo.sysproxies		  spr
				ON js.proxy_id = spr.proxy_id
		WHERE				oa.rownum = 1
		--AND oa.run_status = 0
		ORDER BY			RunDate DESC;

And here is a sample of the output.

With this script, I have the ability to quick show which step is failing, what the command is for that step, what kind of process is running on that step, any passwords (in the event of an SSIS password), and of course the failure frequency. This is golden information at the fingertips. There is no need to click through the GUI to gather this information. You can get it quickly and easily in one fell swoop.

The Wrap

An important part of any DBAs job is to ensure database related jobs are running prim and proper. Sometimes that just doesn’t happen. When jobs are being overlooked, it is useful to be able to gather data related to consistency of job success or failure. This script will help you in your investigation efforts. In addition, I also recommend this article in your agent job audit efforts.

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

Data professionals around the globe are frequently finding themselves occupied with figuring out why and when a file (data or log) for a database has changed in size. Whether that change is a growth or shrink, or if the change was expected to happen or not.

I have written previously about the need to monitor these changes and how to do it more efficiently (there is even a fail-safe that occasionally works. As SQL Server improves, so does our ability to capture these types of events.

Given that SQL Server has undergone many enhancements, let’s take a look at the enhancements related to capturing the database file size changes.

Database File Related Events

If you read the articles mentioned previously, you will have encountered a previous article that include an Extended Event session that can be used for tracking file changes. That session is really good enough in most cases. One major change that I would suggest off the top is the asynchronous file target. While that target will work on all editions of SQL Server since 2008, the name of the target was changed in SQL Server 2012 to event_file. No big deal there. The XEM file is also no longer necessary, so that piece can just be wiped out.

That said, what else has changed? Let’s cover some deprecated events that may get you frustrated if you encounter them. The following events should be avoided because they will do nothing (a couple of them have been covered in this previous article).

• sqlserver.databases_log_growth – Databases log growth
• sqlserver.databases_log_file_size_changed – Databases log file size changed
• sqlserver.databases_data_file_size_changed – Databases data file size change
• sqlserver.auto_shrink_log – Auto shrink log ended.

Each of the preceding events have been replaced by a single event called database_file_size_change.

And then there is this one that I can’t seem to get to generate any events but it might possibly still be valid. In short, don’t be surprised one way or the other if it does something.

• sqlserver.auto_shrink_log_caught_exception – Auto shrink log caught an exception.

Great, we have some events we can avoid. Now let’s look at some events we should consider.

• sqlserver.database_file_size_change – Occurs when any of the data or log files for a database changes size. Use this event to monitor changes to the size of database files.
• sqlserver.databases_shrink_data_movement – Databases shrink data movement
• sqlserver.databases_log_file_used_size_changed – Databases log file used size changed (this one gets noisy – use judiciously).
• sqlserver.databases_log_shrink – Databases log shrink

Now that we have some events that are viable as well as some events to avoid, let’s put a session together.

USE master;
GO
-- Create the Event Session
IF EXISTS ( SELECT *
				FROM sys.server_event_sessions
				WHERE name = 'AuditFileSizev2' )
	DROP EVENT SESSION AuditFileSizev2
    ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE';
GO

CREATE EVENT SESSION [AuditFileSizev2] ON SERVER
ADD EVENT sqlserver.database_file_size_change ( --good
	ACTION ( sqlserver.database_id, sqlserver.session_id,
	sqlserver.database_name, sqlserver.client_hostname, sqlserver.sql_text,
	sqlserver.nt_username, sqlserver.username, sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.context_info,
	sqlserver.client_connection_id )
	),
ADD EVENT sqlserver.databases_shrink_data_movement ( --good
	ACTION ( sqlserver.database_id, sqlserver.session_id,
	sqlserver.database_name, sqlserver.client_hostname, sqlserver.sql_text,
	sqlserver.nt_username, sqlserver.username, sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.context_info,
	sqlserver.client_connection_id )
	),
ADD EVENT sqlserver.databases_log_file_used_size_changed ( --good --this is a very chatty event and may be left off
	ACTION ( sqlserver.database_id, sqlserver.session_id,
	sqlserver.database_name, sqlserver.client_hostname, sqlserver.sql_text,
	sqlserver.nt_username, sqlserver.username, sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.context_info,
	sqlserver.client_connection_id )
	),
ADD EVENT sqlserver.databases_log_shrink ( --good
	ACTION ( sqlserver.database_id, sqlserver.session_id,
	sqlserver.database_name, sqlserver.client_hostname, sqlserver.sql_text,
	sqlserver.nt_username, sqlserver.username, sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.context_info,
	sqlserver.client_connection_id )
	)
ADD TARGET package0.event_file (  SET filename = 'C:\Database\XE\AuditFileSizev2.xel'
												, max_file_size = 5
												, max_rollover_files = 4 )
WITH ( MAX_MEMORY = 4 MB
		, EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
		, TRACK_CAUSALITY = ON
		, MAX_DISPATCH_LATENCY = 1 SECONDS
		, STARTUP_STATE = ON );
GO

All we need to do at this juncture is test the session.

Let’s drill into the database_file_size_change event and take a closer look.

There are a few things going on with this event. As previously mentioned, this event captures multiple different types of size related events. If you now look at the sections that I have circled in red, you will note that there is a flag that tracks if the event was an automatic size change. It doesn’t matter if it was a growth or shrink, both can be automatic. The way we can tell if it was an automatic shrink is due to the negative value in the size_change_kb field.

Next up, we have the green highlighted section. I have three distinct timestamps circled and separated by a dotted red line. Note the time difference between each of the groups. Each group of events is separated by 30 minutes. As it turns out, if you have Autoshrink enabled on your database, the timer is a 30 minute interval. Yes! Autoshrink just so happened to be enabled on this database – for testing purposes.

Wrapping it Up

Extended Events is a powerful tool with plenty of ease of use and flexibility. This flexibility allows the DBA to capably monitor the server for any issue be it small or large. This article demonstrated how to use Extended Events to monitor for file size changes and the same principles can be applied to any of the waits you may need to investigate. If you are in need of a little tune-up for your XE skiils, I recommend reading a bit on Extended Events to get up to date. For other basics related articles, feel free to read here.

In addition, through the power of XE, we were able to capture a quick glimpse into some internals related to the database engine. In this case, we were able to see autoshrink in action and discover the frequency that it runs on when enabled.

How necessary is that SQL Server Agent job that you have running on the server? I ask that question of my clients on a routine basis.

Sometimes, I will ask that question as a routine part of a health check for the server. Others, it might be due to a migration or upgrade that is required for the server. Generally, the answer to the question will be one of two things: a) “Yes it is necessary.” or b) “What is that job doing?”.

Believe it or not, both answers will typically spawn more questions. You see, I don’t usually ask that question unless the job is failing on a regular basis. You wouldn’t believe how many jobs exist out there that are scheduled and just fail every time as well.

When I encounter a situation like this, it means it is time to have a discussion. In order to have that discussion, one needs to have done a bit of homework in order to better understand the situation. For me, part of this homework involves running the numbers and figuring out the frequency of the job’s failure or success.

Data Gathering

For me, I like to understand how often a job has executed and what is the frequency of failure for that quantity of executions. If I see a job that has not succeeded successfully in 60 consecutive executions, it is probably a safe bet that the job is not needed. Why? Well, if nobody has noticed the job hasn’t been working for that long, the likelihood of the job providing any use to the business is slim to none. In this case, I would present a case to the business as to why it should be removed.

But, how do I get to that point? Well, you could go through the job history for each job one by one and run some manual analytics. Or, you could take advantage of a script. I prefer the script route because it is faster, more reliable and a lot less mundane.

WITH jobhist AS
	(SELECT
			   jh.job_id
			 , jh.run_date
			 , jh.run_status
			 , jh.step_id
			 , ROW_NUMBER() OVER (PARTITION BY jh.job_id, jh.run_status ORDER BY jh.run_date DESC) AS rownum
		 FROM  dbo.sysjobhistory jh
		 WHERE jh.step_id = 0)
   , agglastsixty AS
	(SELECT
						sjh.job_id
					  , sjh.run_status
					  , COUNT(*) AS RunCount
		 FROM			dbo.sysjobhistory sjh
			 INNER JOIN jobhist			  jh
				 ON jh.job_id = sjh.job_id
					AND jh.run_status = sjh.run_status
					AND jh.step_id = sjh.step_id
		 WHERE			CONVERT(DATE, CONVERT(VARCHAR, sjh.run_date)) > DATEADD(
																				   d
																				 , -60
																				 , CONVERT(DATE, CONVERT(VARCHAR, jh.run_date))
																			   )
						AND jh.rownum = 1
		 GROUP BY		sjh.job_id
					  , sjh.run_status)
   , aggtotal AS
	(SELECT
						sjh.job_id
					  , sjh.run_status
					  , COUNT(*) AS RunCount
		 FROM			dbo.sysjobhistory sjh
			 INNER JOIN jobhist			  jh
				 ON jh.job_id = sjh.job_id
					AND jh.run_status = sjh.run_status
					AND jh.step_id = sjh.step_id
		 WHERE			jh.rownum = 1
		 GROUP BY		sjh.job_id
					  , sjh.run_status)
	SELECT
							j.name										 AS JobName
						  , sc.name										 AS CategoryName
						  , sp.name										 AS OwnerName
						  , j.owner_sid
						  , j.date_created
						  , j.enabled
						  , CONVERT(DATE, CONVERT(VARCHAR, oa.run_date)) AS RunDate
						  , CASE oa.run_status
								WHEN 0
									 THEN 'Failed'
								WHEN 1
									 THEN 'Succeeded'
								WHEN 2
									 THEN 'Retry'
								WHEN 3
									 THEN 'Canceled'
								WHEN 4
									 THEN 'In Progress'
							END											 AS run_status
						  , als.RunCount								 AS Last60StatusCount
						  , agt.RunCount								 AS TotalStatusCount
		FROM				dbo.sysjobs			  j
			LEFT OUTER JOIN		jobhist				  oa
				ON oa.job_id = j.job_id
			LEFT OUTER JOIN agglastsixty		  als
				ON als.job_id = oa.job_id
				   AND als.run_status = oa.run_status
			LEFT OUTER JOIN aggtotal			  agt
				ON agt.job_id = oa.job_id
				   AND agt.run_status = oa.run_status
			INNER JOIN		sys.server_principals sp
				ON j.owner_sid = sp.sid
			INNER JOIN		dbo.syscategories	  sc
				ON j.category_id = sc.category_id
		WHERE				oa.rownum = 1
		ORDER BY			RunDate DESC;

Running that script against my sandbox, I may see something such as the following.

Here you will note that the “wtf” job has two entries. One entry for “Succeeded” (in green) and one entry for “Failed” (in red). Each row receiving counts for number of executions.

This is the type of information I can use in a meeting to discuss with the business. This is no longer a discussion of opinion, but rather one that is now based on facts and data. It now becomes very easy to demonstrate to the business that a job has failed 60/60 times and nobody noticed it or cared enough about the consistent failures to do anything about it. Imagine if the failing job happens to be the database backups. I wonder what the action items for that job failure might include.

The Wrap

An important part of any DBAs job is to ensure database related jobs are running prim and proper. Sometimes that just doesn’t happen. When jobs are being overlooked, it is useful to be able to gather data related to consistency of job success or failure. This script will help you in your investigation efforts. In addition, I also recommend this article in your agent job audit efforts.

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

Permissions in the database environment is such an important task. Setting permissions correctly is paramount to a successful audit and one of your best defenses against improper/unwanted access. Yet, in spite of the importance, security is often overlooked, neglected, improperly configured or just flat out ignored. Let’s not forget the times that security is intentionally misconfigured so certain individuals are permitted a backdoor access.

Security, just like performance tuning, is a perpetual (and sometimes iterative) task. There is no excuse for setting your security and forgetting it. It must be routinely reviewed.

While performing a script review for a client, I was reminded of the need to also review their security setup. The reminder was more of a slap in the face as I noticed that the developer had built in some permissions assignments for some upgrade scripts. Unfortunately, we were not permitted to alter any of the scripts due to them being from a third party vendor (and that vendor refused as well to fix the problems with the scripts but I digress).

What could be wrong with this?

GRANT REFERENCES,
      SELECT,
      INSERT,
      DELETE,
      UPDATE,
      ALTER,
      VIEW CHANGE TRACKING,
      VIEW DEFINITION,
      CONTROL
ON [dbo].[tblRptWeeklyCorporateBookingListing]
TO  PUBLIC;

GO

I want you to digest that for just a moment. This is an example of the permissions this particular vendor insists on setting for the public role. What could possibly be wrong with that? Let’s examine a couple of the permissions like “Control” and “View Change Tracking”.

View Change Tracking

This permission is an elevated permission that is required in order to use the change tracking functions. This permission is necessary for the following reasons:

1. Change tracking records contain the PK value for rows that have been deleted. If sensitive information was deleted that a user should not be able to access, the user would be able to reverse engineer the data from the change tracking data.
2. A user may be denied access to a column that contains sensitive data. If data in the column is changed, the data would be stored in the change tracking and a user can determine the values that were updated for the sensitive data.

Control

I am going to take this one direct from the Microsoft documentation.

Confers ownership-like capabilities on the grantee. The grantee effectively has all defined permissions on the securable. A principal that has been granted CONTROL can also grant permissions on the securable. Because the SQL Server security model is hierarchical, CONTROL at a particular scope implicitly includes CONTROL on all the securables under that scope. For example, CONTROL on a database implies all permissions on the database, all permissions on all assemblies in the database, all permissions on all schemas in the database, and all permissions on objects within all schemas within the database.

Now digest that a bit. Once digested, consider what the public role does to user access in a database. The public role permissions are inherited by all users of the database whether the users have been granted the permission or not. You should only grant permissions to the public role that you really honestly believe that ALL users should have. If you are being serious in your role, then the amount of times you grant permissions to the public role should either be a) never, b) when you want to have a data breach, or c) you are testing in a sandbox to improve your skills.

Check for Perms

When you are uncertain of which permissions have been assigned to the public role, or you just haven’t reviewed your permissions real-estate in some time, it is best to pull out a script and start the process. As luck would have it, I have a few scripts that can help with that (here or here) and I have a new one that I am sharing now.

Let’s start with a basic query that will display all of the permissions assigned to the public role in a specific database.

SELECT
	sp.state_desc
  , sp.permission_name
  , s.name AS SchemaName
  , o.name
	FROM sys.database_permissions	 sp
		LEFT JOIN sys.all_objects	 o
			ON sp.major_id = o.object_id
		LEFT OUTER JOIN sys.schemas	 s
			ON s.schema_id = o.schema_id
		JOIN sys.database_principals u
			ON sp.grantee_principal_id = u.principal_id
	WHERE u.name = 'public'
		  AND o.name IS NOT NULL
		  AND o.is_ms_shipped = 0
	ORDER BY o.name;

There is nothing super special about this query. Looking at it, it is querying the permissions for the public role specifically. I display where the permission is a “Deny” or “Grant”. Then we list the permission name and then the schema and the object.

Let’s take that script and evolve it now. I am going to plan for the worst and expect that some permissions have been applied that shouldn’t have by some vendor upgrade script (because – well, history). Since I am expecting the worst, I am going to add some script generating code that will revoke the unwanted permissions. And still expecting the worst would be that revoking the permissions will break something, I will also add some code that can generate the appropriate “Grant” statements.

SELECT
	sp.state_desc
  , sp.permission_name
  , s.name			   AS SchemaName
  , o.name
  , 'REVOKE ' + sp.permission_name COLLATE Latin1_General_CI_AS_KS_WS + ' ON [' + s.name + '].[' + o.name
	+ '] FROM PUBLIC;' AS RevokeStmnt
  , 'GRANT ' + sp.permission_name COLLATE Latin1_General_CI_AS_KS_WS + ' ON [' + s.name + '].[' + o.name
	+ '] TO PUBLIC;'   AS GrantStmnt
	FROM sys.database_permissions		sp
		LEFT OUTER JOIN sys.all_objects o
			ON sp.major_id = o.object_id
		LEFT OUTER JOIN sys.schemas		s
			ON s.schema_id = o.schema_id
		JOIN sys.database_principals	u
			ON sp.grantee_principal_id = u.principal_id
	WHERE u.name = 'public'
		  AND o.name IS NOT NULL
		  AND o.is_ms_shipped = 0
	ORDER BY o.name;

That looks better. I have a way of identifying the unwanted permissions as well as an easy script I can execute to remove the unwanted permissions. Note the use of the collate in the final two columns. As it turns out, permission_name from sys.database_permissions has a column collation of Latin1_General_CI_AS_KS_WS. Since I ran into some errors (shown below), it is easier to direct the DB engine to use the collation that matches the permission_name column.

Msg 451, Level 16, State 1, Line 11
Cannot resolve collation conflict between “SQL_Latin1_General_CP850_CS_AS” and “Latin1_General_CI_AS_KS_WS” in add operator occurring in SELECT statement column 5.
Msg 451, Level 16, State 1, Line 11
Cannot resolve collation conflict between “SQL_Latin1_General_CP850_CS_AS” and “Latin1_General_CI_AS_KS_WS” in add operator occurring in SELECT statement column 6.

Alas, this is still not quite as efficient of a script as I would like. I may have hundreds of databases on the instance and need to evaluate all of them. Time for the bigger guns.

/* now do it for all databases */
DECLARE
     @DBName  SYSNAME
	 ,@SQL VARCHAR(2048);

DECLARE @PermsTab AS TABLE (DBName VARCHAR(128),PermissionState VARCHAR(20), PermissionName VARCHAR(128), SchemaName VARCHAR(128), ObjName VARCHAR(128), RevokeStmnt VARCHAR(256), GrantStmnt VARCHAR(256))
 
DECLARE pubperms CURSOR
  LOCAL STATIC FORWARD_ONLY READ_ONLY
     FOR SELECT d.name
		FROM sys.databases d
		WHERE d.state_desc  NOT IN ('SUSPECT','OFFLINE')

		OPEN pubperms;
 
FETCH NEXT FROM pubperms INTO @DBName;
 
WHILE @@FETCH_STATUS = 0
BEGIN

SET @SQL = 'SELECT
	''[' + @DBName + ']''
	,sp.state_desc
  , sp.permission_name
  , s.name			   AS SchemaName
  , o.name
  , ''REVOKE '' + sp.permission_name COLLATE Latin1_General_CI_AS_KS_WS + '' ON ['' + s.name + ''].['' + o.name
	+ ''] FROM PUBLIC;'' AS RevokeStmnt
  , ''GRANT '' + sp.permission_name COLLATE Latin1_General_CI_AS_KS_WS + '' ON ['' + s.name + ''].['' + o.name
	+ ''] TO PUBLIC;''   AS GrantStmnt
	FROM [' + @DBName + '].sys.database_permissions		sp
		LEFT OUTER JOIN [' + @DBName + '].sys.all_objects o
			ON sp.major_id = o.object_id
		LEFT OUTER JOIN [' + @DBName + '].sys.schemas		s
			ON s.schema_id = o.schema_id
		JOIN [' + @DBName + '].sys.database_principals	u
			ON sp.grantee_principal_id = u.principal_id
	WHERE u.name = ''public''
		  AND o.name IS NOT NULL
		  AND o.is_ms_shipped = 0
	ORDER BY o.name;'

	--PRINT @SQL;

INSERT INTO @PermsTab
	(
		DBName
	  , PermissionState
	  , PermissionName
	  , SchemaName
	  , ObjName
	  , RevokeStmnt
	  , GrantStmnt
	)
EXECUTE (@SQL)

FETCH NEXT FROM pubperms INTO @DBName;

END
 
CLOSE pubperms;
DEALLOCATE pubperms;

SELECT DBName
	 , PermissionState
	 , PermissionName
	 , SchemaName
	 , ObjName
	 , RevokeStmnt
	 , GrantStmnt
	FROM @PermsTab;

That will take care of all of the permissions for the public role in all of the databases, with a slight caveat. I am only checking against that objects that are not flagged as is_ms_shipped. Now, isn’t there also a public role at the server scope? Indeed there is! Let’s also capture those permissions.

SELECT sp.state_desc, sp.permission_name, sp.class_desc, sp.major_id, sp.minor_id, e.name
FROM sys.server_permissions AS sp
  INNER JOIN sys.server_principals AS l
    ON sp.grantee_principal_id = l.principal_id
  LEFT OUTER JOIN sys.endpoints AS e
    ON sp.major_id = e.endpoint_id
WHERE l.name = 'public';

Now, I feel many times better about what could possibly be going wrong with the public role.

If you are in a tightly controlled environment or you are just sick of people doing this sort of thing to your servers, there are more extreme measures that can be taken. You can read about it here or here.

The Wrap

It is amazing what some people will do that just doesn’t make sense. Granting permissions to the public role is one of these cases. That behavior also explains why there are documents and procedures for hardening the public role (here and here).

If necessary, I recommend locking down your public role. It will make your job a little easier and give you better rest at night.

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

Every day, they could see the delta on this wait_type continue to climb higher and higher. The problem was not in their ability to capture the deltas for the biggest wait_types. The problem as it were, was in figuring out what exactly was the root cause of their biggest wait_type.

What was the wait_type? Well, the obscure wait_type was called PREEMPTIVE_OS_PIPEOPS. What causes this wait? As it turns out, this is a generic wait that is caused by SQL pipe related activities such as xp_cmdshell.

Knowing this much information however does not get us to the root cause of this particular problem for this client. How do we get there? This is another case for Extended Events (XEvents).

Tracking Pipe Ops

Knowing which wait_type needs attention is most of the battle in figuring out what precisely is causing the specific wait. The second piece of the puzzle is to understand whether the wait is an internal or external. This is essential because a different event is triggered depending on whether the wait is internal or external. There will be more on that in a future article.

Since I happen to know already that the PREEMPTIVE_OS_PIPEOPS wait is an external wait, then I also know that I will want to use the sqlos.wait_info_external. The downside to this method is that conventional wisdom teaches us that we must use the integer value for the wait in order to monitor for it. To find that integer value, one can query the sys.dm_xe_map_values dmv. Since the values of many of the waits change from CU to CU, then one should validate this value on every server instead of using the same value on all servers without regard.

SELECT *
	FROM sys.dm_xe_map_values ws
	WHERE ws.map_value = 'PREEMPTIVE_OS_PIPEOPS'
	AND ws.name = 'wait_types';

Depending on which server I run that code on, I may receive different results.

As you can see, I have a small sample showing five different values for this specific wait. Using the wrong value on the wrong server could result in a bit of hair tugging since nothing may be trapped.

Due to this potential for change, writing a script to monitor for the session gets a little bit trickier and a lot more dynamic – for now.

USE master;
GO

DECLARE @XelFilePath VARCHAR(256) = 'C:\Database\XE\'; --be sure to include the trailing slash as shown here


/* why?
 well as it turns out, many wait_types change from CU to CU of SQL Server
 */

-- Create the Event Session
EXECUTE xp_create_subdir @XelFilePath;
GO

DECLARE 	@mapkey VARCHAR(4)
	, @XESessionName VARCHAR(128) = 'PREEMPTIVE_OS_PIPEOPS'
	,@SQLString VARCHAR(4096)
	,@XelFilePath VARCHAR(256) = 'C:\Database\XE\';

SELECT @mapkey = ws.map_key
	FROM sys.dm_xe_map_values ws
	WHERE ws.map_value = 'PREEMPTIVE_OS_PIPEOPS'
	AND ws.name = 'wait_types';
	
SET @SQLString = 'IF EXISTS
	(
		SELECT
				  *
			FROM  sys.server_event_sessions
			WHERE name = ''' + @XESessionName + '''
	)
	DROP EVENT SESSION ' + @XESessionName + ' ON SERVER;
--GO
'

SET @SQLString = @SQLString + '

CREATE EVENT SESSION [' + @XESessionName + ']
	ON SERVER
	ADD EVENT sqlos.wait_info_external
		(ACTION
			 (
				 package0.callstack
			   , package0.collect_system_time
			   , package0.event_sequence
			   , sqlserver.client_app_name
			   , sqlserver.client_connection_id
			   , sqlserver.client_hostname
			   , sqlserver.context_info
			   , sqlserver.database_id
			   , sqlserver.database_name
			   , sqlserver.nt_username
			   , sqlserver.server_principal_name
			   , sqlserver.session_id
			   , sqlserver.session_nt_username
			   , sqlserver.sql_text
			   , sqlserver.username
			 )
		 WHERE ([wait_type] = ('+ @mapkey + ')) --PREEMPTIVE_OS_PIPEOPS
		)
	ADD TARGET package0.event_file
		(SET filename = N''' + @XelFilePath + 'PREEMPTIVE_OS_PIPEOPS.xel'')
	WITH
		(
			MAX_MEMORY = 4096KB
		  , EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
		  , MAX_DISPATCH_LATENCY = 5 SECONDS
		  , TRACK_CAUSALITY = ON
		  , STARTUP_STATE = ON
		);
--GO

ALTER EVENT SESSION ' + @XESessionName + ' ON SERVER
STATE = START;
--GO
'
PRINT @SQLString;
EXECUTE (@SQLString)

After executing that script, the session will be created with the appropriate wait_type as well as the session will be started. To verify it is working, then I can execute a query that will use xp_cmdshell.

With that information in hand, we now have a tool that will help us identify the root cause of this wait climbing to the top of the waits list. Once I am able to identify the root cause, then I can start applying an appropriate fix (or do nothing if the situation deems that necessary).

But what about this map_key that keeps on changing with updates to SQL Server? Surely there is a better method than figuring out the integer value. As luck would have it, conventional wisdom is wrong in this case. There is a much easier method of creating this event session. So much easier in fact that I don’t need to worry about the map_key value. All I need to know is the wait_type I wish to monitor. So let’s look at a better version of the script.

USE master;
GO
-- Create the Event Session
IF EXISTS
	(
		SELECT
				  *
			FROM  sys.server_event_sessions
			WHERE name = 'PREEMPTIVE_OS_PIPEOPS'
	)
	DROP EVENT SESSION PREEMPTIVE_OS_PIPEOPS ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE\';
GO

CREATE EVENT SESSION [PREEMPTIVE_OS_PIPEOPS]
	ON SERVER
	ADD EVENT sqlos.wait_info_external
		(ACTION
			 (
				 package0.callstack
			   , package0.collect_system_time
			   , package0.event_sequence
			   , sqlserver.client_app_name
			   , sqlserver.client_connection_id
			   , sqlserver.client_hostname
			   , sqlserver.context_info
			   , sqlserver.database_id
			   , sqlserver.database_name
			   , sqlserver.nt_username
			   , sqlserver.server_principal_name
			   , sqlserver.session_id
			   , sqlserver.session_nt_username
			   , sqlserver.sql_text
			   , sqlserver.username
			 )
		 WHERE ([wait_type] = 'PREEMPTIVE_OS_PIPEOPS')
		)
	ADD TARGET package0.event_file
		(SET filename = N'C:\Database\XE\PREEMPTIVE_OS_PIPEOPS.xel')
	WITH
		(
			MAX_MEMORY = 4096KB
		  , EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
		  , MAX_DISPATCH_LATENCY = 5 SECONDS
		  , TRACK_CAUSALITY = ON
		  , STARTUP_STATE = ON
		);
GO

This will yield the same results as the first script, but without the dynamic sql and with a more friendly script for the DBA to understand. This method is actually employed in the system_health system default session if you ever decide to peruse that session.

Wrapping it Up

Extended Events is a powerful tool with plenty of ease of use and flexibility. This flexibility allows the DBA to capably monitor the server for any issue be it small or large. This article demonstrated how to use Extended Events to monitor for a specific wait_type and the same principles can be applied to any of the waits you may need to investigate. If you are in need of a little tune-up for your XE skiils, I recommend reading a bit on Extended Events to get up to date. For other basics related articles, feel free to read here.

The second Tuesday of the month is upon us once again. That means it is time for another group blog party called TSQLTuesday. This party that was started by Adam Machanic has now been going for long enough that changes have happened (such as Steve Jones (b | t) managing it now). For a nice long read, you can find a nice roundup of all TSQLTuesdays over here.

Sands of Time

This month, MOHAMMAD DARAB (b | t) invites us to do the 40 yr. old self reflection. You know that age that is almost as dreadful to reach as the righteous age of 50? From there age only gets better – or at least I have been told that. I have done a couple of articles similar to this in past TSQLTuesdays. Similar yet still different just enough. Check out my article on passion,  or this on one influence.

The problem with this particular challenge is the request to write a note to your 20 yr. old self. Well, maybe not an extreme problem other than the time travel part. As luck would have it, I recently discussed the topic with my grandmother. There was a lot of wisdom in what she taught me as we discussed the topic and I must admit, I was in agreement prior to even broaching the topic with her.

Passing Notes

Here is basically what my note to myself would say: “Self, you are doing good. Keep it up!”

That is even if I chose to send myself a note. You see, I would not want to change how things have happened. Life experience is good – both the good and the bad. All of it is necessary to grow and develop. Just like Mater’s dents, each experience is valuable and shouldn’t be changed.

Now, what if I were to create a note of advice for other more junior level professionals that I might be mentoring? That is a different story. My note to them would be simple – “Listen to me.” Maybe not so literally “me” and maybe more precisely as advice to listen to anybody who is giving their time to present, mentor, blog, or write (articles, books, etc). They are imparting of their experience and time and you should respect that and take advantage of it.

Not only should you pay respect to those giving of their time, but you should spend a little time to also give back to the community. Why? Well, I covered that pretty well in this article here. I will say it again, the more you give of yourself to help others learn your trade, the more you will personally learn and grow in your professional and personal self. It is an investment well worth taking.

Wrapping it Up

I don’t believe in trying to alter who I have become with any sort of visit back to my past self. The changes could cause me to change my path and not end up where I have reached due to personal growth and experience. Trial and error is good for growth. Everybody needs to learn that for themselves and cannot be forced into it. Just learn to grow and be patient with anybody you may mentor.

There has long been a sticking point about allowing various people access to production servers. Part of the sticking point is the developer who believes that access to read and evaluate trace files is mandatory on prod (yes I have heard many times where this has been used effectively by developers to gain prod access). But is that prod access truly necessary? Without an adequate method to provide the developer access to the trace data, the DBA loses confidence (from management) and standing ground for their side of the argument. How can a middle ground be met?

XEvents to the Rescue

With the use of XEvent tracing, a DBA is given a new realm of possibilities. This particular realm of possibilities becomes available when the XEvent trace (session) is created using a file target. Through the use of a file target, and the flexibility of XEvents, I have shown the beginnings of how a developer might be able to access the trace data easily.

That is merely the beginning though. There are a few more layers to making life easier for both the developer and DBA in regards to fetching event trace data. This article will show a few methods that will help make life just a bit easier.

Let’s start with the basics. After an XEvent session is created, it is important to know where the event file is being stored. If you don’t happen to know where it is, that is not a problem. The file source is easy enough to find and I have detailed it in this article.

Method 1

As shown in the previous article, the first quick method to open an XEvent trace file is from within SSMS as shown here.

After clicking on “Merge Extended Event Files…”, a new window titled “Merge Extended Event Files” will open. From the new window, follow the following steps.

Method 2

Now, that is an extremely simple method to open and view a trace file for XEvents. That said, would you believe there are other equally easy methods? Let’s look at the next method that is very simple as well.

As was previously mentioned, you will need to know the file location first and then navigate to that location. So let’s do that. My common location is C:\Database\XE. Once navigated to the folder, I need to choose my file to be opened and follow the prompts as shown here.

A new window will pop up. By default, SSMS will be showing. If not, you can select SSMS and enable the option to always use SSMS for this file type by clicking the check box before clicking OK.

Method 3

If the prior two methods feel unreasonably easy, then there is this next method that will spice things up a bit. It does require a bit of coordination. Once you have identified the file to be opened, you can elect to perform a drag and drop operation of the file onto SSMS. Once done properly, the file will open in SSMS and you can start evaluating the data.

Method 4

This is as equally simplistic as the previous method but requires just a scoch less coordination. This method involves a double-click method. A double-click on the file will open the file up in SSMS.

Method 5

If you paid close attention to the screenshots in method 2, you will have noticed that there was a menu option called “Open” that I did not discuss. If you select open from the context menu at that point, then the xel file will open in SSMS just like any of the other methods mentioned thus far.

Wrapping it Up

Extended Events is a powerful tool with plenty of ease of use and flexibility. This flexibility allows the DBA to better service the needs of the developers when the developers need access to the trace data. This article demonstrated another use for Extended Events. If you are in need of a little tune-up for your XE skiils, I recommend reading a bit on Extended Events to get up to date. For other basics related articles, feel free to read here.

In a recent article, I covered a little bit about implicit conversions and an error that may crop up from time to time. It is one thing to understand what an error means, it is an entirely different problem to understand what is causing the error in the first place.

Seldom does a DBA have the opportunity to get out in front of infrequent or random errors such as implicit conversions. More often than not, it is the privilege of the DBA to find out about the problem after the fact from a developer or, worse yet, an end-user.

What if there was an easy method to track when these errors occurred? Even better, it would be ideal to understand what query caused the error as well as which types of queries are merely causing implicit conversion warnings. Even better, what if you had a way to understand which type of conversion problem is at hand.

Implicit Playground

To start this playground, let’s begin with a simple query based on the AdventureWorks database.

SELECT
		  emp.BusinessEntityID
		, NationalIDNumber
		, LoginID
	FROM  HumanResources.Employee emp
	WHERE NationalIDNumber = 112457891;
GO

This query will yield a plan similar to the following.

If I hover the select operator, I will see a little window like the following.

I have a few items highlighted in this window. In the larger red box, I am outlining much of the type conversion warning. Inside of that, there is a blue box highlighting the “convert_implicit” warning. If I look further into the error message, there is text about query plan choice and “CardinalityEstimate” or “SeekPlan” for the different convert_implicit warnings. Another way of looking at it is as follows.

This seems a lot more friendly than that little context window. Here we have an easy to follow layout of what is happening. As I drill down into the warnings, I can see there are three convert_implicit warnings and they all trigger a “planaffectingconvert” type of warning. As we look closer at this, I can see there are two distinct types of convert issues.

There is one for the conversion from nvarchar to int in the HumanResources.Employee table for the NationalIDNumber field (green).

Two warnings that state that an index seek has been disabled due to the conversions ( orange and purple).

Curious to see what all of the possible “ConvertIssue” values could be for these implicit conversions? That is easy enough with the following query.

SELECT mv.name, mv.map_value, mv.map_key
	FROM  sys.dm_xe_map_values mv
	WHERE mv.name = 'plan_affecting_convert_type';

Which shows us the following results.

That is a grand total of three. Having covered the cause of two of these already, all that is left is the type called “unknown”. I think we can safely presume that the cause of this one is something else that is undocumented and hence “unknown”.

So far, so good. We have seen how these warnings can crop up in execution plans. We have also seen how to query the for the “ConvertIssue” types or “plan_affecting_convert_type” from the Extended Events (XE) maps system DMV.

Right about now, the light bulb should be popping up for you. If we can query the XE metadata for the different convert issue types, then does that mean there is a way we can use XE to track these things and review them at our own convenience? Yes there is!

XE

Without further adieu, let’s look at an XE session that can be useful in discovering more information in regards to implicit conversions in the database environment.

USE master;
GO
-- Create the Event Session
IF EXISTS ( SELECT		*
			FROM	sys.server_event_sessions
			WHERE name = 'ImplicitConvertError' )
	DROP EVENT SESSION ImplicitConvertError ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE';
GO

CREATE EVENT SESSION [ImplicitConvertError]
	ON SERVER
	ADD EVENT sqlserver.error_reported
	( ACTION ( sqlserver.sql_text
			, sqlserver.nt_username
			, sqlserver.server_principal_name
			, sqlserver.client_hostname
			, package0.collect_system_time
			, package0.event_sequence
			, sqlserver.database_id
			, sqlserver.database_name
			, sqlserver.username
			, sqlserver.session_nt_username
			, sqlserver.client_app_name
			, sqlserver.session_id
			, sqlserver.context_info
			, sqlserver.client_connection_id )
	WHERE ( [error_number] = ( 257 )))
--fee free to uncomment these next few events if you wish to receive a ton more information
--, ADD EVENT sqlserver.module_end
--( ACTION ( ssqlserver.sql_text
--		, sqlserver.nt_username
--		, sqlserver.server_principal_name
--		, sqlserver.client_hostname
--		, package0.collect_system_time
--		, package0.event_sequence
--		, sqlserver.database_id
--		, sqlserver.database_name
--		, sqlserver.username
--		, sqlserver.session_nt_username
--		, sqlserver.client_app_name
--		, sqlserver.session_id
--		, sqlserver.context_info
--		, sqlserver.client_connection_id ))
--, ADD EVENT sqlserver.rpc_starting
--( ACTION ( sqlserver.sql_text
--		, sqlserver.nt_username
--		, sqlserver.server_principal_name
--		, sqlserver.client_hostname
--		, package0.collect_system_time
--		, package0.event_sequence
--		, sqlserver.database_id
--		, sqlserver.database_name
--		, sqlserver.username
--		, sqlserver.session_nt_username
--		, sqlserver.client_app_name
--		, sqlserver.session_id
--		, sqlserver.context_info
--		, sqlserver.client_connection_id ))
--, ADD EVENT sqlserver.sql_statement_completed
--( ACTION ( sqlserver.sql_text
--		, sqlserver.nt_username
--		, sqlserver.server_principal_name
--		, sqlserver.client_hostname
--		, package0.collect_system_time
--		, package0.event_sequence
--		, sqlserver.database_id
--		, sqlserver.database_name
--		, sqlserver.username
--		, sqlserver.session_nt_username
--		, sqlserver.client_app_name
--		, sqlserver.session_id
--		, sqlserver.context_info
--		, sqlserver.client_connection_id ))
--, ADD EVENT sqlserver.sql_statement_starting
--( ACTION ( sqlserver.sql_text
--		, sqlserver.nt_username
--		, sqlserver.server_principal_name
--		, sqlserver.client_hostname
--		, package0.collect_system_time
--		, package0.event_sequence
--		, sqlserver.database_id
--		, sqlserver.database_name
--		, sqlserver.username
--		, sqlserver.session_nt_username
--		, sqlserver.client_app_name
--		, sqlserver.session_id
--		, sqlserver.context_info
--		, sqlserver.client_connection_id ))
, ADD EVENT sqlserver.plan_cache_cache_hit
( ACTION ( sqlserver.sql_text
		, sqlserver.nt_username
		, sqlserver.server_principal_name
		, sqlserver.client_hostname
		, package0.collect_system_time
		, package0.event_sequence
		, sqlserver.database_id
		, sqlserver.database_name
		, sqlserver.username
		, sqlserver.session_nt_username
		, sqlserver.client_app_name
		, sqlserver.session_id
		, sqlserver.context_info
		, sqlserver.client_connection_id ))
, ADD EVENT sqlserver.plan_cache_cache_attempt
( ACTION ( sqlserver.sql_text
		, sqlserver.nt_username
		, sqlserver.server_principal_name
		, sqlserver.client_hostname
		, package0.collect_system_time
		, package0.event_sequence
		, sqlserver.database_id
		, sqlserver.database_name
		, sqlserver.username
		, sqlserver.session_nt_username
		, sqlserver.client_app_name
		, sqlserver.session_id
		, sqlserver.context_info
		, sqlserver.client_connection_id ))
, ADD EVENT sqlserver.plan_affecting_convert
( ACTION ( sqlserver.sql_text
		, sqlserver.nt_username
		, sqlserver.server_principal_name
		, sqlserver.client_hostname
		, package0.collect_system_time
		, package0.event_sequence
		, sqlserver.database_id
		, sqlserver.database_name
		, sqlserver.username
		, sqlserver.session_nt_username
		, sqlserver.client_app_name
		, sqlserver.session_id
		, sqlserver.context_info
		, sqlserver.client_connection_id )
WHERE ( [package0].[equal_boolean]([sqlserver].[is_system], ( 0 ))
		AND [sqlserver].[database_id] > ( 4 )))
, ADD EVENT sqlserver.sp_cache_remove
( ACTION ( sqlserver.client_app_name
		, sqlserver.sql_text )
WHERE ( [sqlserver].[is_system] = ( 0 )))
, ADD EVENT sqlserver.uncached_sql_batch_statistics
( ACTION ( sqlserver.sql_text
		, sqlserver.nt_username
		, sqlserver.server_principal_name
		, sqlserver.client_hostname
		, package0.collect_system_time
		, package0.event_sequence
		, sqlserver.database_id
		, sqlserver.database_name
		, sqlserver.username
		, sqlserver.session_nt_username
		, sqlserver.client_app_name
		, sqlserver.session_id
		, sqlserver.context_info
		, sqlserver.client_connection_id )
WHERE ( [sqlserver].[is_system] = ( 0 )))
	ADD TARGET package0.event_file
(	SET filename = N'C:\Database\XE\ImplicitConvertError.xel', max_rollover_files = ( 5 ))
	WITH ( MAX_MEMORY = 4096KB
		, EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
		, MAX_DISPATCH_LATENCY = 3 SECONDS
		, MAX_EVENT_SIZE = 0KB
		, MEMORY_PARTITION_MODE = NONE
		, TRACK_CAUSALITY = ON
		, STARTUP_STATE = OFF );
GO

This session is pretty comprehensive with the ability to be very much like a fire-hose session if the commented out events are uncommented prior to session creation. Running this on a busy server ( less than 1 minute) for a client with a third party vendor application produces results that look like the following.

As we can see here, the session is trapping when I have any of those pesky implicit conversion errors (red box outline) as mentioned in the previous article, as well as the convert_implicit warnings (purple box outline) that were demonstrated in the execution plans in this article.

How am I trapping the implicit conversion error? That is done via the following piece of the event session shown above.

ADD EVENT sqlserver.error_reported
	( ACTION ( sqlserver.sql_text
			, sqlserver.nt_username
			, sqlserver.server_principal_name
			, sqlserver.client_hostname
			, package0.collect_system_time
			, package0.event_sequence
			, sqlserver.database_id
			, sqlserver.database_name
			, sqlserver.username
			, sqlserver.session_nt_username
			, sqlserver.client_app_name
			, sqlserver.session_id
			, sqlserver.context_info
			, sqlserver.client_connection_id )
	WHERE ( [error_number] = ( 257 ))) --implicit conversion error

Error number 257 is the message id for the implicit conversion error. When filtering on that ID for this event, then I will receive only the implicit errors that match that number.

As for the implicit conversion warnings, we have the following that accomplishes that requirement.

, ADD EVENT sqlserver.plan_affecting_convert
( ACTION ( sqlserver.sql_text
		, sqlserver.nt_username
		, sqlserver.server_principal_name
		, sqlserver.client_hostname
		, package0.collect_system_time
		, package0.event_sequence
		, sqlserver.database_id
		, sqlserver.database_name
		, sqlserver.username
		, sqlserver.session_nt_username
		, sqlserver.client_app_name
		, sqlserver.session_id
		, sqlserver.context_info
		, sqlserver.client_connection_id )
WHERE ( [package0].[equal_boolean]([sqlserver].[is_system], ( 0 ))
		AND [sqlserver].[database_id] > ( 4 )))

The event, plan_affecting_convert should seem eerily familiar at this point. We have seen terminology like that in the execution plans as well as in the query for the map values.

The combination of these events will provide great insight into the issues revolving around any implicit conversions you may be having in your database environment.

Wrapping it Up

Implicit conversions are a fail-safe for when bad design or code (or both) crops up in your environment. When this fail-safe is being used in your environment, it is helpful to gain further insights via Extended Events. This article demonstrated another use for Extended Events. If you are in need of a little tune-up for your XE skiils, I recommend reading a bit on Extended Events to get up to date. For other basics related articles, feel free to read here.

Every now and again, you may run into an error about implicit conversions in your queries. Implicit conversions are not uncommon in the computing world and can be viewed as kind of a fail-safe for when we don’t quite follow decent practices when designing the database or when writing queries or both.

Despite this little fail-safe, there are times when a nasty little error will pop up and cause a bit of consternation.

Implicit conversion from data type %ls to %ls is not allowed. Use the CONVERT function to run this query.

What Went Wrong

Unlike many other errors in SQL Server, this error message makes some sense. The major components of what is wrong are present and you are given a decent idea of what the conversion attempt is that failed. When this particular error happens, you can bet that there are issues with some TSQL code somewhere for sure. In addition, you can bet there is likely a problem with the database design as well. Yay! More work for your back burner.

First, this error comes with an error id of 257 and we can see the message text via the following query. This id is important for when we want to monitor for this problem in the future.

SELECT *
	FROM sys.messages m
	WHERE m.message_id = 257
	AND m.language_id = 1033;

Let’s see how we can recreate this problem.

DECLARE @BeginDate DATE = '2019-01-01',
        @EndDate DATE = '2019-12-31';
/* quick and easy method to create a test table with millions of rows if needed */
SELECT TOP 100
       ImplicitID = IDENTITY(INT, 1, 1),
       ImplicitInt = ABS(CHECKSUM(NEWID())) % 50000 + 1,
       ImplicitDate = DATEADD(DAY, RAND(CHECKSUM(NEWID())) * (1 + DATEDIFF(DAY, @EndDate, @BeginDate)), @EndDate)
INTO #ImplicitTest
FROM master.dbo.syscolumns t1,
     master.dbo.syscolumns t2;

ALTER TABLE #ImplicitTest ADD PRIMARY KEY CLUSTERED (ImplicitID);

SELECT *
FROM #ImplicitTest dq
WHERE ISNULL(dq.ImplicitID, GETDATE()) = '5/17/2020';

DROP TABLE #ImplicitTest;

Which will produce the following.

This is a prime example of a bad query producing an error. Obviously, I am using the wrong data type to try and query the temp table. The ImplicitID column is an integer and I am trying to query it using a date. The quick fix, would be to query the appropriate date column if I must use a date in my query, or i can use an integer to query the ImplicitID column.

SELECT *
FROM #ImplicitTest dq
WHERE dq.ImplicitDate > '5/17/2019';

After this minor tweak, now the query will work and I start to see results. Given the random nature of the data in this query, the results will vary from batch to batch.

Wrapping it Up

Implicit conversions are a fail-safe for when bad design or code (or both) crops up in your environment. Sometimes, this fail-safe is inadequate and needs further assistance. Sometimes, that may be an explicit conversion and sometimes that means an appropriate rewrite of the query to use the appropriate columns and data types in the queries. This was an introductory article into the world of implicit conversions. There will be a follow-up or two about implicit conversions and monitoring for them. In preparation for the upcoming articles, I recommend reading a bit on Extended Events to get up to date. For other basics related articles, feel free to read here.

Well, I have previously written about this topic, you are welcome to read it here. The article discusses quite a bit about scripting out your XE Sessions. One thing lacking in that article is the ability to script out every session on a server.

If you are still not using Extended Events, I recommend checking out this library of articles that will cover just about all of the basics concerning Extended Events.

New and Improved

What about scripting out all of the sessions in SSMS? Surely there is an easy way to do that, right? Well, you might think that. Let me step through the problem that I have seen in SSMS (and unfortunately it is not consistent).

First, from Object Explorer Details (or F5), let’s try to script a single session.

When scripting a single session from the “Object Explorer Details”, I have several sub-menus that allow me to script the session to a “New Query Editor Window”. Now, let’s see what happens when trying to script multiple sessions.

With several sessions selected, I try yet again to script the sessions and I get an unwanted result. Notice that the “Script Session as” option is grayed out and unusable. However, if I try it again (several times or maybe just once, your mileage may vary and it seems to not be relevant to version of SSMS), I may see something like this.

Tada! Luck was with me and it finally worked that time. So, what should I do to be able to consistently script all of sessions? Well, that comes with an enhancement to the script I presented in the prior article here.

Lets just dive straight into the new script.

SET NOCOUNT ON;

DECLARE
	@statement	   VARCHAR(MAX)
  , @statement1	   VARCHAR(MAX)
  , @statement2	   VARCHAR(MAX)
  , @sessionname   VARCHAR(128)				 --	= 'AlwaysOn_health'						 -- 'system_health'
  , @csessionname  VARCHAR(128)
  , @XeFilePath	   VARCHAR(256)
  , @IsDefaultPath TINYINT
											 /* order of events scripted is different 
default order of creation is different than if scripted from the gui
the gui scripts by alpha order of package then event
while original creation has order by event_id only
*/
  , @eventorder	   VARCHAR(64) = 'original'; --'original' for order based on original script, --'ssms' for order based on ssms guy scripter 

DECLARE xessions CURSOR LOCAL STATIC FORWARD_ONLY READ_ONLY FOR
	SELECT
				 ses.name
		FROM	 sys.server_event_sessions ses
		WHERE	 ses.name = ISNULL(@sessionname, ses.name)
		ORDER BY ses.name;

OPEN xessions;

FETCH NEXT FROM xessions
INTO
	@csessionname;

DECLARE @ErrorLog VARCHAR(256) = CONVERT(VARCHAR(256), SERVERPROPERTY('ErrorLogFileName'));
DECLARE @ErrorPath VARCHAR(256) = LEFT(@ErrorLog, LEN(@ErrorLog) - CHARINDEX('\', REVERSE(@ErrorLog)) + 1);

WHILE @@FETCH_STATUS = 0
	BEGIN
		IF EXISTS
			(
				SELECT
								   1
					FROM		   sys.server_event_sessions	   ses
						INNER JOIN sys.server_event_session_fields sesf
							ON ses.event_session_id = sesf.event_session_id
					WHERE		   sesf.name = 'filename'
								   AND ses.name = @csessionname
			)
			BEGIN
				IF
					(
						SELECT
										   CHARINDEX('\', CONVERT(VARCHAR(256), sesf.value))
							FROM		   sys.server_event_sessions	   ses
								INNER JOIN sys.server_event_session_fields sesf
									ON ses.event_session_id = sesf.event_session_id
							WHERE		   sesf.name = 'filename'
										   AND ses.name = @csessionname
					) > 0
					BEGIN
						SELECT
										   @XeFilePath
							= LEFT(CONVERT(VARCHAR(256), sesf.value), LEN(CONVERT(VARCHAR(256), sesf.value))
																	  - CHARINDEX(
																					 '\'
																				   , REVERSE(CONVERT(
																										VARCHAR(256)
																									  , sesf.value
																									)
																							)
																				 ) + 1)
							FROM		   sys.server_event_sessions	   ses
								INNER JOIN sys.server_event_session_fields sesf
									ON ses.event_session_id = sesf.event_session_id
							WHERE		   sesf.name = 'filename'
										   AND ses.name = @csessionname;
					END;
				ELSE
					BEGIN
						IF
							(
								SELECT				TOP 1
													LEFT(t2.file_name, LEN(t2.file_name) - CHARINDEX('\', REVERSE(t2.file_name)) + 1) AS XEFilePath
									FROM
													(
														SELECT
																		   REPLACE(CONVERT(NVARCHAR(128), sesf.value), '.xel', '*.xel') AS targetvalue
																		 , ses.name														AS SessionName
															FROM		   sys.server_event_sessions	   ses
																INNER JOIN sys.server_event_session_fields sesf
																	ON ses.event_session_id = sesf.event_session_id
															WHERE		   sesf.name = 'filename'
																		   AND ses.name = @csessionname
													)																	cte1
										CROSS APPLY sys.fn_xe_file_target_read_file(cte1.targetvalue, NULL, NULL, NULL) t2
							) IS NOT NULL
							BEGIN
								SELECT
									@XeFilePath
									=
									(
										SELECT				TOP 1
															LEFT(t2.file_name, LEN(t2.file_name)
																			   - CHARINDEX('\', REVERSE(t2.file_name)) + 1) AS XEFilePath
											FROM
															(
																SELECT
																				   REPLACE(CONVERT(NVARCHAR(128), sesf.value), '.xel', '*.xel') AS targetvalue
																				 , ses.name														AS SessionName
																	FROM		   sys.server_event_sessions	   ses
																		INNER JOIN sys.server_event_session_fields sesf
																			ON ses.event_session_id = sesf.event_session_id
																	WHERE		   sesf.name = 'filename'
																				   AND ses.name = @csessionname
															)								 cte1
												CROSS APPLY sys.fn_xe_file_target_read_file(
																							   cte1.targetvalue
																							 , NULL
																							 , NULL
																							 , NULL
																						   ) t2
									);
							END;
						ELSE
							BEGIN
								SELECT
									@XeFilePath = @ErrorPath;
								SELECT
									@XeFilePath
								  , @ErrorPath;
							END;

					END;
			END;

		IF @ErrorPath = @XeFilePath
			BEGIN
				SET @IsDefaultPath = 1;
			END;
		ELSE
			BEGIN
				SET @IsDefaultPath = 0;
			END;
		SELECT
				  @statement
			= 'USE master;
GO
-- Create the Event Session
IF EXISTS ( SELECT *
				FROM sys.server_event_sessions
				WHERE name = ' + ses.name + ' )
	DROP EVENT SESSION ' + ses.name + ' 
    ON SERVER;
GO
'
			FROM  sys.server_event_sessions ses
			WHERE ses.name = @csessionname;

		IF @IsDefaultPath = 1
			BEGIN
				SET @statement = @statement + '';
			END;
		ELSE
			BEGIN
				SET @statement = @statement + '
EXECUTE xp_create_subdir '''	 + @XeFilePath + ''';
GO
'				;
			END;
		SELECT
				  @statement = @statement + '
CREATE EVENT SESSION ['	 + ses.name + '] ON SERVER
	'
			FROM  sys.server_event_sessions ses
			WHERE ses.name = @csessionname;

		WITH presel AS
			(SELECT
								ses.name											 AS SessionName
							  , sese.event_id
							  , sese.package
							  , 'ADD EVENT ' + sese.package + '.' + sese.name + ' (' AS EventName
							  , CASE
									WHEN ISNULL(sese.predicate, '') = ''
										 THEN ')'
									ELSE 'WHERE ' + sese.predicate + ' )'
								END													 AS SQLPredicate
							  , CASE
									WHEN ISNULL(sesf.EventSetOps, '') = ''
										 THEN ''
									ELSE 'SET ' + sesf.EventSetOps + ''
								END													 AS SETOperation
				 FROM			sys.server_event_sessions		ses
					 INNER JOIN sys.server_event_session_events sese
						 ON ses.event_session_id = sese.event_session_id
					 LEFT OUTER JOIN
								(
									SELECT	 DISTINCT
											 ST2.event_session_id
										   , ST2.object_id
										   , SUBSTRING(
												 (
													 SELECT
																		', ' + ST1.name + '=' + '(' + CONVERT(VARCHAR(60), ST1.value) + ')' AS [text()]
														 FROM			sys.server_event_session_fields ST1
															 INNER JOIN sys.server_event_session_events sese
																 ON sese.event_session_id = ST1.event_session_id
																	AND sese.event_id = ST1.object_id
														 WHERE			ST1.event_session_id = ST2.event_session_id
																		AND ST1.object_id = ST2.object_id
														 ORDER BY		ST1.object_id
													 FOR XML PATH('')
												 )
											   , 3
											   , 8000
													  ) [EventSetOps]
										FROM sys.server_event_session_fields ST2
								)								sesf
						 ON sese.event_session_id = sesf.event_session_id
							AND sese.event_id = sesf.object_id
				 WHERE			ses.name = @csessionname)
		   , actpresel AS
			(SELECT
								ses.name					   AS SessionName
							  , sesa.event_session_id
							  , sesa.event_id
							  , sesa.package + '.' + sesa.name AS ActPack
				 FROM			sys.server_event_session_events	 sese
					 INNER JOIN sys.server_event_session_actions sesa
						 ON sese.event_session_id = sesa.event_session_id
							AND sese.event_id = sesa.event_id
					 INNER JOIN sys.server_event_sessions		 ses
						 ON sesa.event_session_id = ses.event_session_id
				 WHERE			ses.name = @csessionname)
		   , pstuff AS
			(SELECT
						  p.SessionName
						, p.event_id
						, 'ACTION ( 
				' +		STUFF((
								  SELECT
											', ' + pin.ActPack + '
				'
									  FROM	actpresel pin
									  WHERE pin.SessionName = p.SessionName
											AND pin.event_id = p.event_id
								  FOR XML PATH(''), TYPE
							  ).value('.', 'varchar(max)')
							, 1
							, 2
							, ''
							 ) + '
			)'			AS ActionPack
				 FROM	  actpresel p
				 GROUP BY p.SessionName
						, p.event_id)
		SELECT
			@statement1 = @statement + STUFF((
												 SELECT
																		 ', ' + CAST(p.EventName + '
		'														 + p.SETOperation + '
		'														 + CASE
																	   WHEN ISNULL(ps.ActionPack, '') = ''
																			THEN ''
																	   ELSE ps.ActionPack
																   END + '
		'														 + p.SQLPredicate + '
'													 AS VARCHAR(MAX)) AS EventStatement
													 FROM				 presel p
														 LEFT OUTER JOIN pstuff ps
															 ON p.SessionName = ps.SessionName
																AND p.event_id = ps.event_id
													 ORDER BY			 p.event_id
												 --ORDER BY p.package, p.EventName 
												 FOR XML PATH(''), TYPE
											 ).value('.', 'varchar(max)')
										   , 1
										   , 2
										   , ''
											)
		  , @statement2 = @statement + STUFF((
												 SELECT
																		 ', ' + CAST(p.EventName + '
		'														 + p.SETOperation + '
		'														 + CASE
																	   WHEN ISNULL(ps.ActionPack, '') = ''
																			THEN ''
																	   ELSE ps.ActionPack
																   END + '
		'														 + p.SQLPredicate + '
'													 AS VARCHAR(MAX)) AS EventStatement
													 FROM				 presel p
														 LEFT OUTER JOIN pstuff ps
															 ON p.SessionName = ps.SessionName
																AND p.event_id = ps.event_id
													 ORDER BY			 p.package
																	   , p.EventName
												 FOR XML PATH(''), TYPE
											 ).value('.', 'varchar(max)')
										   , 1
										   , 2
										   , ''
											);

		IF @eventorder = 'original'
			BEGIN
				SET @statement = @statement1;
			END;
		IF @eventorder = 'ssms'
			BEGIN
				SET @statement = @statement2;
			END;

		SELECT
			@statement
			= @statement
			  + STUFF(
			(
				SELECT
								   ', ' + '
ADD TARGET '										+ sest.package + '.' + sest.name
								   + CASE
										 WHEN ISNULL(ca.setop, '') = ''
											  THEN ' 
'
										 ELSE '
			(SET ' +								ca.setop + ' )
'
									 END
					FROM		   sys.server_event_sessions		ses
						INNER JOIN sys.server_event_session_targets sest
							ON ses.event_session_id = sest.event_session_id
						CROSS APPLY
								   (
									   SELECT
										   STUFF(
			(
				SELECT
								   ', ' + CAST(sesf.name AS VARCHAR(128)) + ' = '
								   + CASE
										 WHEN ISNUMERIC(CAST(sesf.value AS VARCHAR(128))) = 1
											  THEN '( ' + CAST(sesf.value AS VARCHAR(128)) + ' )
							'
										 ELSE 'N''' + CAST(sesf.value AS VARCHAR(128)) + '''
						'
									 END AS setop
					FROM		   sys.server_event_session_fields	sesf
						INNER JOIN sys.server_event_sessions		sesi
							ON sesi.event_session_id = sesf.event_session_id
						INNER JOIN sys.server_event_session_targets sesti
							ON sesf.object_id = sesti.target_id
							   AND sesi.event_session_id = sesti.event_session_id
					WHERE		   sest.target_id = sesti.target_id
								   AND sest.event_session_id = sesti.event_session_id
				FOR XML PATH(''), TYPE
			).value('.', 'varchar(max)')
		  , 1
		  , 2
		  , ''
												)
								   )								AS ca(setop)
					WHERE		   ses.name = @csessionname
				FOR XML PATH(''), TYPE
			).value('.', 'varchar(max)')
		  , 1
		  , 2
		  , ''
					 );

		SELECT
				  @statement
			= @statement
			  + CAST('WITH ( MAX_MEMORY = ' + CAST(ses.max_memory AS VARCHAR(20)) + ' KB
		,EVENT_RETENTION_MODE = '		   + ses.event_retention_mode_desc + '
		, MAX_DISPATCH_LATENCY = '		   + CAST(ses.max_dispatch_latency / 1000 AS VARCHAR(20))
					 + ' SECONDS
		, MAX_EVENT_SIZE = '			   + CAST(ses.max_event_size AS VARCHAR(20))
					 + ' KB
		, MEMORY_PARTITION_MODE = '		   + ses.memory_partition_mode_desc + '
		, TRACK_CAUSALITY = '			   + CASE ses.track_causality
												 WHEN 1
													  THEN 'ON'
												 ELSE 'OFF'
											 END + '
		, STARTUP_STATE = '				   + CASE ses.startup_state
												 WHEN 1
													  THEN 'ON'
												 ELSE 'OFF'
											 END + '
	);'		AS VARCHAR(1024))
			FROM  sys.server_event_sessions ses
			WHERE ses.name = @csessionname;

		PRINT @statement;

		FETCH NEXT FROM xessions
		INTO
			@csessionname;
	END;

CLOSE xessions;
DEALLOCATE xessions;

This is a rather lengthy script, so I won’t explain the entire thing. That said, this script will produce the exact XE Session as it was written when you deployed it to the server. In addition, the script will ensure the destination directory for the event_file target is created as a part of the script.

I can definitely hear the gears of thought churning as you ponder about this whole scenario. Surely, you have all of your XE Sessions stored in source control so there is no need whatsoever for this little script. Then again, that would be in an ideal environment. Sadly, source control is seldom considered for XE Sessions. Thus, it is always good to have a backup plan.

Why

Sadly, I had the very need of migrating a ton of sessions from one server to another recently and the methods in SSMS just wouldn’t work. There was no source control in the environment. Building out this little script saved me tons of time in migrating all of the sessions for this server and also provided me with a good script to place in source control.

Conclusion

In the article today, I have provided an excellent tool for backing up all of your XE sessions on the server. This script will help create the necessary scripts for all of your XE Sessions (or even just a single session if you like) in order to migrate the sessions to a new server or place them in source control.

To read more about Extended Events, I recommend this series of articles.

This is a topic that I ventured into some time ago. The previous article can be read here. In that article I covered some of the various trouble spots with capturing the file path for various XE log files. One of the main problems being that there is frequently an inconsistency in where XE logs may actually be stored.

Using what was shown in that previous article, I have some improvements and minor tweaks to fill some gaps I hadn’t completed in the previous script.

If you are still not using Extended Events, I recommend checking out this library of articles that will cover just about all of the basics concerning Extended Events.

New and Improved

First, lets just dive straight into the new script.

DECLARE @ErrorPath VARCHAR(256) = CONVERT(VARCHAR(256), SERVERPROPERTY('ErrorLogFileName'));
DECLARE
	@SessionName  VARCHAR(128) = NULL --'AlwaysOn_health'
  , @csessionname VARCHAR(128);

CREATE TABLE #SessionPaths
	(
		SessionName VARCHAR(128)
	  , XEFilePath	VARCHAR(256)
	  , SelectPhase VARCHAR(24)
	);

DECLARE xessions CURSOR LOCAL STATIC FORWARD_ONLY READ_ONLY FOR
	SELECT
				 ses.name
		FROM	 sys.server_event_sessions ses
		WHERE	 ses.name = ISNULL(@SessionName, ses.name)
		ORDER BY ses.name;

OPEN xessions;

FETCH NEXT FROM xessions
INTO
	@csessionname;

WHILE @@FETCH_STATUS = 0
	BEGIN

		IF EXISTS
			(
				SELECT
								   1
					FROM		   sys.server_event_sessions	   ses
						INNER JOIN sys.server_event_session_fields sesf
							ON ses.event_session_id = sesf.event_session_id
					WHERE		   sesf.name = 'filename'
								   AND ses.name = @csessionname
			)
			BEGIN
				IF
					(
						SELECT
										   CHARINDEX('\', CONVERT(VARCHAR(256), sesf.value))
							FROM		   sys.server_event_sessions	   ses
								INNER JOIN sys.server_event_session_fields sesf
									ON ses.event_session_id = sesf.event_session_id
							WHERE		   sesf.name = 'filename'
										   AND ses.name = @csessionname
					) > 0
					BEGIN
						INSERT INTO #SessionPaths
							(
								SessionName
							  , XEFilePath
							  , SelectPhase
							)
							SELECT
											   ses.name													 AS SessionName
											 , LEFT(CONVERT(VARCHAR(256), sesf.value), LEN(CONVERT(VARCHAR(256), sesf.value))
																					   - CHARINDEX(
																									  '\'
																									, REVERSE(CONVERT(
																														 VARCHAR(256)
																													   , sesf.value
																													 )
																											 )
																								  ) + 1) AS XEFilePath
											 , 'Phase1'
								FROM		   sys.server_event_sessions	   ses
									INNER JOIN sys.server_event_session_fields sesf
										ON ses.event_session_id = sesf.event_session_id
								WHERE		   sesf.name = 'filename'
											   AND ses.name = @csessionname;
					END;
				ELSE
					BEGIN
						INSERT INTO #SessionPaths
							(
								SessionName
							  , XEFilePath
							  , SelectPhase
							)
							SELECT				DISTINCT
												cte1.SessionName
											  , CASE
													WHEN RIGHT(LEFT(cte1.targetvalue, LEN(cte1.targetvalue)
																					  - CHARINDEX('\', REVERSE(cte1.targetvalue)) + 1), 1) = '\'
														 THEN LEFT(cte1.targetvalue, LEN(cte1.targetvalue)
																					 - CHARINDEX('\', REVERSE(cte1.targetvalue)) + 1)
													ELSE LEFT(@ErrorPath, LEN(@ErrorPath) - CHARINDEX('\', REVERSE(@ErrorPath)) + 1)
												END AS XEFilePath
											  , CASE
													WHEN RIGHT(LEFT(t2.file_name, LEN(t2.file_name)
																				  - CHARINDEX('\', REVERSE(t2.file_name)) + 1), 1) = '\'
														 THEN 'Phase2'
													ELSE 'FailSafe'
												END
								FROM
												(
													SELECT
																	   REPLACE(CONVERT(NVARCHAR(128), sesf.value), '.xel', '*.xel') AS targetvalue
																	 , ses.name														AS SessionName
														FROM		   sys.server_event_sessions	   ses
															INNER JOIN sys.server_event_session_fields sesf
																ON ses.event_session_id = sesf.event_session_id
														WHERE		   sesf.name = 'filename'
																	   AND ses.name = @csessionname
												)																	cte1
									OUTER APPLY sys.fn_xe_file_target_read_file(cte1.targetvalue, NULL, NULL, NULL) t2;

					END;


			END;

		FETCH NEXT FROM xessions
		INTO
			@csessionname;
	END;
CLOSE xessions;
DEALLOCATE xessions;

SELECT
		 *
	FROM #SessionPaths sp;

DROP TABLE #SessionPaths;

One of the things I wanted to accomplish with this update was to find the correct path for all of the sessions on the server. As mentioned in the previous article, sometimes there are complications with that. Due to the way log files can be specified for an XE session, behaviors can be a bit funky sometimes when trying to parse the correct paths. Due to those problems, I couldn’t quite short-cut the logic in the previous script and had to do the less desirable thing and create a cursor.

In addition to the cursor, I threw in a fix for when a full path is not declared for the session (at the time of creation) and the session was subsequently never started. In these odd cases, the script had been returning an empty result set and thus was not working properly. Now, it is fixed and here is an example of the output.

The third column in this result set is purely for informational purposes so I could determine at which point the file path was being derived. For the 30+ sessions running on my test instance, most paths are resolved via the first select. In the image, that is denoted by the label “Phase1” and circled in red. The system_health session happened to be running, but did not have a full path declared so it fell into the “Phase2” resolution group and is circled in blue. The last group includes those cases where a path could not be resolved for any number of reasons so they fall to the “FailSafe” grouping and an example is circled in green in the image.

Why

Truth be told, there is a method to short cut this script and get the results faster but I felt it would be less accurate. I could obviously just default to the “FailSafe” group automatically if a full path is not defined in the session creation. Would that be accurate though? Most of the time it would be accurate, but then there are the edge cases where occasionally we forget that something has changed. One such case of this is if after the session is created, you decide the SQL Server log files needs to be moved from the default path (this is where the XEL files default to if no path is defined)?

I have run across multiple scenarios where the logs were required (both technical as well as political) to be moved from the default location. Ideally, this move occurs prior to server startup. When the log file path is changed, the logs are not moved automatically to the new location. This, for me, is a case where it is best to be thorough rather than snake bit. I also like to document these things so I can compare them later if necessary.

Alternatively, here is the just good enough to pass muster version of that script.

/* or the "good enough" version */
DECLARE @ErrorPath VARCHAR(256) = CONVERT(VARCHAR(256), SERVERPROPERTY('ErrorLogFileName'));

SELECT	DISTINCT
		cte1.SessionName
	  , CASE
			WHEN RIGHT(LEFT(cte1.targetvalue, LEN(cte1.targetvalue) - CHARINDEX('\', REVERSE(cte1.targetvalue)) + 1), 1) = '\'
				 THEN LEFT(cte1.targetvalue, LEN(cte1.targetvalue) - CHARINDEX('\', REVERSE(cte1.targetvalue)) + 1)
			ELSE LEFT(@ErrorPath, LEN(@ErrorPath) - CHARINDEX('\', REVERSE(@ErrorPath)) + 1)
		END AS XEFilePath
	  , CASE
			WHEN RIGHT(LEFT(cte1.targetvalue, LEN(cte1.targetvalue) - CHARINDEX('\', REVERSE(cte1.targetvalue)) + 1), 1) = '\'
				 THEN 'Quick'
			ELSE 'FailSafe'
		END
	FROM
		(
			SELECT
							   REPLACE(CONVERT(NVARCHAR(128), sesf.value), '.xel', '*.xel') AS targetvalue
							 , ses.name														AS SessionName
				FROM		   sys.server_event_sessions	   ses
					INNER JOIN sys.server_event_session_fields sesf
						ON ses.event_session_id = sesf.event_session_id
				WHERE		   sesf.name = 'filename'
		) cte1;

Conclusion

In the article today, I have shown some of the internals to retrieving file paths for Extended Event Sessions. I dove into metadata to pull out the path for the session and discussed some concerns for some of these methods. In the end, you have a few viable options to help retrieve the file path in a more consistent fashion.

To read more about Extended Events, I recommend this series of articles.

The second Tuesday of the month comes to us a little early this month. That means it is time again for another group blog party called TSQLTuesday. This party that was started by Adam Machanic has now been going for long enough that changes have happened (such as Steve Jones (b | t) managing it now). For a nice long read, you can find a nice roundup of all TSQLTuesdays over here.

The Why?

This month, Matthew Mcgiffen (b | t) invites us to come to a little puzzle party for our TSQL Tuesday party. Bring your favorite brain teaser, puzzle, questions, or interesting and complex TSQL problems/solutions.

This ties pretty nicely into the topic from last month (well at least for me). If you recall, last month Todd asked all of us to share some of our uses for databases in our personal lives. I submitted my article here, but forgot about one of my favorite uses for a database in my personal life – a daily trivia set about SQL Server.

So, why not elaborate on that database a bit today. Almost like a two for one. However, there is one little quick departure I want to make. My first puzzle solved with TSQL was written about many moons ago and can be found here.

TSQL Challenges

A long time ago, Jacob Sebastian ran regular challenges involving TSQL to get you to think about ways to solve problems using TSQL. The site is no longer available, but challenge #97 was about solving Sudoku puzzles. Here is my solution to that particular challenge. A little TSQL and a bit of the black arts and there is a nifty little solution. Even though I have this little trick in the bag, I still solve the Sudoku puzzles the hard way.

What a nice little stroll down memory lane there. That solution alone could satisfy the request for this TSQL Tuesday. Alas, we won’t stop there!

QOTD

Several years ago, I put together a little database to help with daily trivia questions. The database is a simple design and had a primary function to help teach SQL Server facts and internals to those with varying levels of DBA experience as well as helpdesk members. That said, the topic of questions doesn’t have to be SQL specific – it was my primary use.

With just a few tables and a few procs, I have a database that I can use to create questions, track responses from participants, and email questions and answers to participants on a daily basis – automatically. If I run out of questions, I just add more to the table. Nothing super complex there.

What this offers me is a mechanism to mentor multiple people without the burnout and while also gauging their true interest level in improving their SQL skill set.

Wrapping it Up

One of my favorite database automations is to send trivia style questions daily. This helps me to mentor and assess other dba talent within an organization without being too aggressive. Not only can a database be used for automation but it can also be used for numerous other automations. Beyond being highly useful for automation and training, there is also the possibility of using TSQL to solve puzzles like Sudoku puzzles.

I have written a couple articles showing how to audit database offline events via the default trace as well as via Extended Events. These are great for a more proactive approach. Sometimes the proactive approach is just not plausible for the given circumstances. A good example of this is when you inherit servers (e.g. new client, new project, new job) and there are databases on the server that were taken offline prior to your inheritance.

In a situation like this, you need to find out when the databases were taken offline, but you are somewhat limited by the data available to you for your research purposes. The default trace has rolled beyond the retention point for your server for events that were trapped. Setting up an Extended Events session would be less than helpful given the databases are already offline. So we must find an alternative to finding approximately when the databases were taken offline.

Alternatives

If your predecessor leaves you with a database that has been taken offline and no documentation about it, there is an alternative to find the approximate time it was taken offline – using TSQL. Granted, you could explore the file system and make the same sort of guess based on the file modified date. Data people generally prefer some sort of script and a script that is native to their language (tsql).

So, let’s take a look at a viable option for figuring out the mystery behind your database that has been set offline with no good info surrounding it.

/* requires that the DB have had at least one full backup*/
DECLARE @Numdays INT = 1; 
--minute, d for days

SELECT
				   d.name					 AS DBName
				 , mf.name					 AS filelogicalname
				 , d.create_date
				 , mf.differential_base_time AS DBOfflineDate --utc time
	FROM		   sys.master_files mf
		INNER JOIN sys.databases	d
			ON mf.database_id = d.database_id
	WHERE		   mf.differential_base_time < DATEADD(DAY,@Numdays * -1,GETUTCDATE())
		AND d.state_desc = 'offline'
	ORDER BY	   mf.differential_base_time;

Running this query, I see the following on my test server:

Note that this query uses GETUTCDATE. The differential_base_time column in sys.master_files is in UTC time. Thus, in order to compare properly, we need to ensure we use a UTC datestamp for comparison. Also, of note, this query doesn’t work if there is no full database backup for the database in question. Who doesn’t backup their databases though, right? No seriously, you really should.

Since this is my test server and I am well prepared, I just so happen to have an XEvent session running that tracks the offline events that I can use to compare the results.

Yes, there is a bit of variance in the actual database offline event and the differential_base_time column. This variance is due to timing of the backup and the actual offline event. Again, this is about finding an approximate time of when the database was taken offline as a viable alternative when other data is not really available.

Conclusion

We all aspire to having a perfect database environment where nothing surprising or unexpected happens. Unfortunately, that is the desire of dreams and fairy tales. The unexpected will happen. A database can be offline at the time that you take over responsibility of the database. You will be asked when it was taken offline. This script will help you get a reasonable deduction for that offline event in the absence of better data.

Extended Events is a powerful tool to help in troubleshooting and tuning your environment. I recommend investing a little time in reading the 60 day series about Extended Events. This is not a short series but is designed to provide an array of topics to help learn the tool over time. Don’t forget to go back and read the companion article showing how to audit these events via the default trace.

The second Tuesday of the month comes to us a little early this month. That means it is time again for another group blog party called TSQLTuesday. This party that was started by Adam Machanic has now been going for long enough that changes have happened (such as Steve Jones (b | t) managing it now). For a nice long read, you can find a nice roundup of all TSQLTuesdays over here.

The Why?

This month, Todd Kleinhans (b | t) invites us to share something personal about ourselves. Well sort of. The invitation to each of us is to share a little about databases that we use in our personal lives or that we have created in our personal lives to help us in some fashion or another.

I think this is a really cool idea. Data and databases are very integrated into my life. The integration is enough that it would be difficult to un-blur the line where personal life and professional life begins when talking about data and databases.

Take a moment and reflect on the importance of data and databases within your life. At a minimum, you have taken the approach that data is your profession of choice and hopefully it is a career for you and not just a job.

If you have taken it down the career path, then you probably study your craft in your personal time to try and make yourself better at your trade-craft. Often times, if you are like me, these studies lead to trial and error, experiments, and possibly rabbit holes. If data is just a J-O-B, it is quite likely none of this will apply to you. Those are the types that fall into a category Steve Jones wrote about recently, and I would urge you to change that sooner rather than later. (Side note, I read that article as if it applied to me directly because I really do need to learn and practice more about my craft too!)

Personal Databases

Over the years, I have had sooo many personal databases that I have created for one reason or another. Some still need to be finished. Here are some examples of what I use a database for in my personal life.

Finances: A simple little database to track (sort of like a checkbook yes) credits and debits. More importantly, it tracked bills and due dates with dollar amounts for each bill. This is useful both for reminders for bills but also to see if there are any noteworthy trends with spending or increased costs. Now, many banking (and credit card) sites offer the same types of services – so long as the spending is done with their card.

Exercise: Back in the day, I tracked all of my activities in a database. I would track the duration, activity type, effort level and various biometric measures after each activity. Once again, this is something that is now tracked by so many different apps and databases that it seems counter productive to maintain my own system in favor of something that works more easily on a wearable device.

I have also previously done databases to inventory books, movies, track personal studies and link thoughts about studies from day to day, or even track routine chores (calendar reminders don’t offer much for tracking details). The short of it is, a database can be created to manage so many things as a part of everyday personal life.

Wrapping it Up

If you can use a database to create automation routines to help sustain your work life, you can certainly do the same to sustain your personal life. Databases are an essential piece of both my personal and professional life. Not only do I use them to create automation (such as this or this) for things, but I stand databases up to test numerous things from corruption to recovery to performance tuning to help improve my personal abilities and craft in the realm of data.

An indexed view is a view where the result set from the query (the view definition) becomes materialized in lieu of the virtual table result set of a standard (non-indexed) view. Many times we see that an indexed view would be created to help improve performance. Far too often, an indexed view is created without consideration for the costs of the indexed view. In this article, I hope to cover some of the more important costs that are frequently overlooked when considering an indexed view as a performance panacea.

The Setup

To demonstrate the overlooked costs of indexed views, I have decided to use the AdventureWorks2014 database (if you don’t have this sample database, you can get your copy here). I will create a view in that database and then I will add a few indexes to that view. Prior to sharing the entire setup, and like all good DBAs, I need to first grab a baseline. For this baseline, I am doing nothing terribly complex. I will just grab table sizes and storage information for the entire database.

Here are my results for the size of the objects within the AdventureWorks2014 database:

These results show that the largest object in the database happens to be the Person.Person table at just about 30MB. Not terribly large, but the database is not terribly large as a whole. Let’s see what happens when I add a materialized view, based on the Person.Person table, to the mix. Here is the definition of that view along with the indexes that I will add to help the performance of some essential production related queries.

USE [AdventureWorks2014];
GO

CREATE VIEW [Person].[vPerson]
WITH SCHEMABINDING
AS
	SELECT pp.BusinessEntityID
			, pp.FirstName
			, pp.LastName
			, sp.[StateProvinceID]
			, sp.[StateProvinceCode]
			, sp.[IsOnlyStateProvinceFlag]
			, sp.[Name] AS [StateProvinceName]
			, sp.[TerritoryID]
			, cr.[CountryRegionCode]
			, cr.[Name] AS [CountryRegionName]
			, pbe.AddressTypeID
			, pp.AdditionalContactInfo
			, pp.Demographics
		FROM [Person].[StateProvince] sp
			INNER JOIN [Person].[CountryRegion] cr
				ON sp.[CountryRegionCode] = cr.[CountryRegionCode]
			INNER JOIN Person.Address pa
				ON sp.StateProvinceID = pa.StateProvinceID
			INNER JOIN Person.BusinessEntityAddress pbe
				ON pbe.AddressID = pa.AddressID
			INNER JOIN Person.Person pp
				ON pp.BusinessEntityID = pbe.BusinessEntityID
		WHERE pbe.AddressTypeID = 2;

GO


CREATE UNIQUE CLUSTERED INDEX CI_vPersonID ON Person.vPerson(BusinessEntityID);

CREATE INDEX IX_vPersonName ON Person.vPerson(FirstName,LastName);
CREATE INDEX IX_vPersonState ON Person.vPerson(FirstName,LastName,StateProvinceID,StateProvinceCode,StateProvinceName);

After executing all of that code to create this new view with its indexes, I have the following size results:

The creation of this view has chewed up a bunch of storage. It has jumped right up to the number two spot on the biggest objects list within this database. You can see that differences by comparing the highlighted rows to the previous image. The vPerson view is highlighted in red in this second image to help point it out quickly.

Surely this must be a contrived example and people don’t really do this in the real world, right? The answer to that is simply: NO! It DOES happen. I see situations like this all too often. Far too often, large text fields are added to an indexed view to make retrieval faster. I have mimicked that by adding in two XML columns from the Person.Person table. This is definitely overkill because a simple join back to the table based on the BusinessEntityID would get me those two columns. All I have effectively done is duplicated data being stored and I have achieved that at the low low cost of increased storage of 25% for this small database. If you are curious, the column count between the Person.Person table and this new view is 13 columns each.

I call the 25% increase storage cost a significant increase. An increase of 25% in storage for a single materialized view is not an uncommon occurrence. I have seen multi-terabyte databases with 25% of the total data storage being caused by a single view. If you are tight on storage, any unexpected data volume increase can cause a non-linear growth in data requirements to occur just because of the presence of indexed views. Take that into consideration when presented with an option to either performance tune the code or to take a short-cut and create an indexed view.

Band-aid Performance

I alluded to the use of indexed views as a cog to help improve performance. This is done by taking the query that sucks and turning it into a view. The thought is that it will run faster because an index is created specifically for that data set. Generally the performance gains from using an indexed view, to camouflage the bad query, are seen when the query is complex. Unfortunately, the query I provided for this article is not terribly complex. Due to the simplicity of the first example, let’s look first at the execution plan for another query that runs terribly slow but is also not terribly complex:

There are a few table spools and index scans. The most costly operator of the bunch appears to be a merge join with a many-to-many join operation. The indexed view in this case does give me a very significant improvement on this query (look at that estimated cost and then the estimated number of rows). What if I could tune the query a little and avoid the all of storage cost? Let’s see!

Consider the base query for this new view:

SELECT pp.BusinessEntityID
			, pp.FirstName
			, pp.LastName
			, sp.[StateProvinceID]
			, sp.[StateProvinceCode]
			, sp.[IsOnlyStateProvinceFlag]
			, sp.[Name] AS [StateProvinceName]
			, sp.[TerritoryID]
			, cr.[CountryRegionCode]
			, cr.[Name] AS [CountryRegionName]
			, pbe.AddressTypeID
			, pp.AdditionalContactInfo
			, pp.Demographics
		FROM [Person].[StateProvince] sp 
			INNER JOIN [Person].[CountryRegion] cr
				ON RTRIM(LTRIM(sp.[CountryRegionCode])) = RTRIM(LTRIM(cr.[CountryRegionCode]))
			INNER JOIN Person.Address pa
				ON RTRIM(LTRIM(sp.StateProvinceID)) = RTRIM(LTRIM(pa.StateProvinceID))
			INNER JOIN Person.BusinessEntityAddress pbe
				ON RTRIM(LTRIM(pbe.AddressID)) = RTRIM(LTRIM(pa.AddressID))
			INNER JOIN Person.Person pp
				ON RTRIM(LTRIM(pp.BusinessEntityID)) = RTRIM(LTRIM(pbe.BusinessEntityID))
		WHERE RTRIM(LTRIM(pbe.AddressTypeID)) = 2;

Looking at the query, you may be asking yourself why use so many trim functions on each of the joins? This is an extreme that can be easily fixed. None of the fields in the joins should need to be trimmed since they are all numeric in this case. Starting there and removing all of the function use in the joins should provide significant improvements in query speed and without the use of a view. Testing it out would yield similar performance results to the first view in this article. This small change would cause the query to complete in 1 second compared to about 90 seconds. That is a huge gain, and without the added cost of increased storage use.

Do I really see stuff like this in the field? Yes! Spending a little time to fix the root of the problem rather than looking for a quick fix and performance can be spectacular. There are times, though, that an indexed view may be an absolute requirement. That would be perfectly fine. There are times when it is warranted and there is no way around it. If an indexed view is an absolute must, then there are a couple more considerations to be taken into account.

Disappearing Act

I showed previously that the indexed view requires storage space and showed that the new view created quickly claimed its place as the second largest object within the database. Now, to stage the next segment, I will show once again that the indexes are present and consuming disk space. This can be shown via the following query:

SELECT o.name AS ObjName
		, i.name AS IdxName
		, SUM(ps.page_count) AS page_count
		, SUM(ps.record_count) AS record_count
		, SUM(ps.page_count) / 128.0 AS SizeMB
	FROM sys.indexes i
		INNER JOIN sys.objects o
			ON i.object_id = o.object_id
		INNER JOIN sys.dm_db_index_physical_stats(DB_ID(), NULL, NULL, NULL,
													'detailed') ps
			ON ps.object_id = o.object_id
	WHERE o.name LIKE 'vperson%'
	GROUP BY o.name
		, i.name;

And the results of that query should look something like the following (quick reminder that I created a new indexed view for the second demo and these results will be based on that second view):

Now that we know the index is present and consuming space, time to make a change to the view and see what happens. I am proceeding under the premise that I have determined that including the two XML columns in the view are completely unnecessary and are the cause of too much space consumption. I can reap the benefits of the view at a fraction of the cost if I ALTER the view and remove those columns. So I will proceed by issuing the following ALTER statement:

ALTER VIEW [Person].[vPerson2]
WITH SCHEMABINDING
AS
	SELECT pp.BusinessEntityID
			, pp.FirstName
			, pp.LastName
			, sp.[StateProvinceID]
			, sp.[StateProvinceCode]
			, sp.[IsOnlyStateProvinceFlag]
			, sp.[Name] AS [StateProvinceName]
			, sp.[TerritoryID]
			, cr.[CountryRegionCode]
			, cr.[Name] AS [CountryRegionName]
			, pbe.AddressTypeID
		FROM [Person].[StateProvince] sp 
			INNER JOIN [Person].[CountryRegion] cr
				ON RTRIM(LTRIM(sp.[CountryRegionCode])) = RTRIM(LTRIM(cr.[CountryRegionCode]))
			INNER JOIN Person.Address pa
				ON RTRIM(LTRIM(sp.StateProvinceID)) = RTRIM(LTRIM(pa.StateProvinceID))
			INNER JOIN Person.BusinessEntityAddress pbe
				ON RTRIM(LTRIM(pbe.AddressID)) = RTRIM(LTRIM(pa.AddressID))
			INNER JOIN Person.Person pp
				ON RTRIM(LTRIM(pp.BusinessEntityID)) = RTRIM(LTRIM(pbe.BusinessEntityID))
		WHERE RTRIM(LTRIM(pbe.AddressTypeID)) = 2
;

GO

Take note of how long this ALTER command takes to complete – nearly instantaneous.  That rocks! Right? Let’s look at the indexes to make sure they are still in working order and don’t require a defrag. Based on that, I will re-run that query to check the indexes and sizes – for giggles.

SELECT o.name AS ObjName
		, i.name AS IdxName
		, SUM(ps.page_count) AS page_count
		, SUM(ps.record_count) AS record_count
		, SUM(ps.page_count) / 128.0 AS SizeMB
	FROM sys.indexes i
		INNER JOIN sys.objects o
			ON i.object_id = o.object_id
		INNER JOIN sys.dm_db_index_physical_stats(DB_ID(), NULL, NULL, NULL,
													'detailed') ps
			ON ps.object_id = o.object_id
	WHERE o.name LIKE 'vperson%'
	GROUP BY o.name
		, i.name;

This time when the query completes, I get a nice little surprise: I no longer have any indexes on that view. Changing the view definition instantly drops any indexes that were built on that view. If this is done in a production system, imagine the implications and problems that could arise.

Conclusion

In this article, I have covered indexed views and three different considerations that are often overlooked when looking to implement a materialized view. The three considerations for indexed views, as discussed, are: storage, changes to the view, and the band-aid effect. Overlooking the cost of storage can have serious repercussions on the production environment. Implementing an indexed view without attempting to tune the code first could lead to overlooking the storage cost and could end up just being a waste of resources. The last tidbit is that any change to an indexed view will drop the indexes. That is an easily overlooked feature of indexed views. If you forget to recreate the indexes on the view after making changes, you could be looking at a production outage.

There are a couple more critical considerations for indexed views that are often overlooked as well. These additional considerations are blocking, deadlocking, and maintenance. These considerations will be discussed further in a future article.

For more reading enjoyment try the Back to Basics or Xevents series.

The first critical task any data professional should ever learn how to do is how to connect to SQL Server. Without a connection to SQL Server, there is barely anything one could do to be productive in the data professional world (for SQL Server).

Yet, despite the commonality of this requirement and ease of the task, I find that there is frequent need to retrain professionals on how to connect to SQL Server. This connection could be attempted from any of the current options but for some reason it still perplexes many.

Let’s look at just how easy it is (should be) to connect to SQL Server (using both SQL Auth and Windows Auth).

Simplicity

First let’s take a look at the dialog box we would see when we try to connect to a server in SQL Server Management Studio (SSMS).

Circled in red we can see the two types of authentication I would like to focus on: “Windows Authentication” and “SQL Server Authentication”. These are both available from the dropdown box called Authentication. The default value here is “Windows Authentication”.

If I choose the default value for authentication or “Windows Authentication”, I only need to click on the connect button at the bottom of that same window. Upon closer inspection, it becomes apparent that the fields “User name:” and “Password:” are disabled and cannot be edited when this default value is selected. This is illustrated in the next image.

Notice that the fields circled in red are greyed out along with their corresponding text boxes. This is normal and is a GOOD thing. Simply click the connect button circled in green and then you will be connected if you have permissions to the specific server in the connection dialog.

Complicating things a touch is the “SQL Server Authentication”. This is where many people get confused. I see many people trying to enter windows credentials in this authentication type. Sure, we are authenticating to SQL Server, but the login used in this method is not a windows account. The account to be used for this type of authentication is strictly the type that is created explicitly inside of SQL Server. This is a type of login where the principal name and the password are both managed by SQL Server.

Let’s take a look at an example here.

Notice here that the text boxes are no longer greyed out and I can type a login and password into the respective boxes. Once done, I then click the “Connect” button and I will be connected (again, presuming I have been granted access and I typed the user name and password correctly).

What if I attempt to type a windows username and password into these boxes?

If I click connect on the preceding image, I will see the following error.

This is by design and to be expected. The authentication methods are different. We should never be able to authenticate to SQL Server when selecting the “SQL Server Authentication” and then providing windows credentials. Windows credentials must be used with the “Windows Authentication” option. If you must run SSMS as a different windows user, then I recommend reading this article.

The Wrap

Sometimes what may be ridiculously easy for some of us may be mind-blowing to others. Sometimes we may use what we think are common terms only to see eyes start to glaze over and roll to the backs of peoples heads. This just so happens to be one of those cases where launching an app as a different principal may be entirely new to the intended audience. In that vein, it is worthwhile to take a step back and “document” how the task can be accomplished.

Connecting to SQL Server is ridiculously easy. Despite the ease, I find myself helping “Senior” level development and/or database staff.

If you feel the need to read more connection related articles, here is an article and another on the topic.

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

Well, for whatever reason, there seems to be a myth out there that when deadlocks come a-knocking, then just throw this little directive at it and all will be well. Today, we get to test that and show what will continue to happen.

First, lets look at what the ROWLOCK actually means:

Specifies that row locks are taken when page or table locks are ordinarily taken.

This seems like a fabulous idea if the deadlocks are occurring against a table involving a massive update. Let’s take a look at a small update scenario involving just a handful of records. For the setup, we will use the same setup used in a previous article by Wayne.

USE tempdb;
GO
IF OBJECT_ID('dbo.Test1') IS NOT NULL
	DROP TABLE dbo.Test1;
IF OBJECT_ID('dbo.Test2') IS NOT NULL
	DROP TABLE dbo.Test2;
CREATE TABLE dbo.Test1 ( col1 INT );
CREATE TABLE dbo.Test2 ( col2 INT );
INSERT INTO dbo.Test1
	VALUES	( 1 ),
			( 2 ),
			( 3 ),
			( 4 ),
			( 5 );
INSERT INTO dbo.Test2
	VALUES	( 1 ),
			( 2 ),
			( 3 ),
			( 4 ),
			( 5 );
GO
BEGIN TRANSACTION;
UPDATE dbo.Test1
	SET	col1 = col1 * 10
	WHERE col1 = 3;

/* query window 2*/

USE tempdb;
BEGIN TRANSACTION;
UPDATE dbo.Test2
	SET	col2 = col2 * 20
	WHERE col2 = 4;
UPDATE dbo.Test1
	SET	col1 = col1 * 20
	WHERE col1 = 3;
COMMIT TRANSACTION;

/* back to the first query window*/
UPDATE dbo.Test2
	SET	col2 = col2 * 10
	WHERE col2 = 4;
COMMIT TRANSACTION;

Looking at the code, we can see there are only five rows in each of the tables. In addition, an update will be performed to both col1 and col2 in each table for one specific row. So we are keeping this to a singleton type of update, and we are able to force a deadlock by using this setup. Not only do we see that a deadlock will occur consistently, we would see the following in the sys.dm_tran_locks DMV as well as in the deadlock graphs.

In my environment, I used session 51 and 54 consistently for the deadlock repro. In each of the tests, each spid did obtain page locks as well as RID locks (row locks) that were waiting on each other in order to perform an Update. This is what we saw when I ran the setup without the ROWLOCK directive. What if I used the ROWLOCK directive (are you guessing already that there would be no change because the deadlock occurs on the lock held for the update that is waiting on the other update to complete?)? Let’s take a look at that too!

/* what about using rowlock hint? let's try it */
USE tempdb;
GO
IF OBJECT_ID('dbo.Test1') IS NOT NULL
	DROP TABLE dbo.Test1;
IF OBJECT_ID('dbo.Test2') IS NOT NULL
	DROP TABLE dbo.Test2;
CREATE TABLE dbo.Test1 ( col1 INT );
CREATE TABLE dbo.Test2 ( col2 INT );
INSERT INTO dbo.Test1
	VALUES	( 1 ),
			( 2 ),
			( 3 ),
			( 4 ),
			( 5 );
INSERT INTO dbo.Test2
	VALUES	( 1 ),
			( 2 ),
			( 3 ),
			( 4 ),
			( 5 );
GO
BEGIN TRANSACTION;
UPDATE dbo.Test1 WITH (ROWLOCK)
	SET	col1 = col1 * 10
	WHERE col1 = 3;

/* query window 2*/

USE tempdb;
BEGIN TRANSACTION;
UPDATE dbo.Test2 WITH (ROWLOCK)
	SET	col2 = col2 * 20
	WHERE col2 = 4;
UPDATE dbo.Test1 WITH (ROWLOCK)
	SET	col1 = col1 * 20
	WHERE col1 = 3;
COMMIT TRANSACTION;

/* back to the first query window*/
UPDATE dbo.Test2 WITH (ROWLOCK)
	SET	col2 = col2 * 10
	WHERE col2 = 4;
COMMIT TRANSACTION;

The only change to this setup is that the ROWLOCK directive has been added to the update statements. Examining the sys.dm_tran_locks DMV reveals the same locks being held as was seen without the directive. This shouldn’t be too big of a surprise since the updates are against a single row.

In addition to the same locks being held, we continue to experience the same deadlock problem. Using an Extended Events session to trap deadlock information (similar to the article previously discussed), we can pull out some pretty good info. Let’s examine some of the deadlock data trapped by an XE session.

WITH	cte	AS ( SELECT t2.event_data.value('(event/@name)[1]', 'varchar(50)') AS event_name
							, t2.event_data.value('(event/@timestamp)[1]',
													'datetime2') AS StartTime
							, t2.event_data.value('(event/data[@name="duration"]/value)[1]',
													'bigint') AS duration
							, t2.event_data.value('(event/data[@name="database_name"]/value)[1]',
													'sysname') AS DBName
							, t2.event_data.value('(event/action[@name="sql_text"]/value)[1]',
													'varchar(max)') AS sql_text
							, t2.event_data.value('(event/action[@name="nt_username"]/value)[1]',
													'varchar(500)') AS nt_username
							, t2.event_data.value('(event/data[@name="mode"]/value)[1]',
													'varchar(15)') + ' ('
							+ t2.event_data.value('(event/data[@name="mode"]/text)[1]',
													'varchar(50)') + ')' AS mode
							, t2.event_data.value('(event/data[@name="object_id"]/value)[1]',
													'integer') AS object_id
							, t2.event_data.value('(event/data[@name="resource_description"]/value)[1]',
													'varchar(max)') AS resource_description
							, t2.event_data.value('(event/data[@name="resource_owner_type"]/text)[1]',
													'varchar(max)') AS resource_owner_type
							, t2.event_data.value('(event/data[@name="resource_type"]/text)[1]',
													'varchar(max)') + ' ('
							+ t2.event_data.value('(event/data[@name="resource_type"]/value)[1]',
													'varchar(max)') + ')' AS resource_type
							, t2.event_data.value('(event/action[@name="server_principal_name"]/value)[1]',
													'varchar(max)') AS server_principal_name
							, t2.event_data.value('(event/action[@name="session_id"]/value)[1]',
													'varchar(max)') AS session_id
							, t2.event_data.value('(event/action[@name="client_pid"]/value)[1]',
													'integer') AS client_pid
							, t2.event_data.value('(event/action[@name="client_app_name"]/value)[1]',
													'varchar(max)') AS client_app_name
							, t2.event_data
							,ROW_NUMBER() OVER (ORDER BY t2.event_data.value('(event/@timestamp)[1]',
													'datetime2')
													,t2.event_data.value('(event/@name)[1]', 'varchar(50)')) AS Event_Seq
						FROM sys.fn_xe_file_target_read_file('C:\Database\XE\Deadlock*.xel',
																NULL, NULL, NULL) t1
							CROSS APPLY ( SELECT CONVERT(XML, t1.event_data)
										) t2 ( event_data )
				)
	SELECT c1.event_name AS LockEvent, c2.event_name AS GraphEvent
			, c1.StartTime
			, DATEADD(MICROSECOND, c1.duration, CONVERT(DATETIME2, [c1].StartTime)) AS EndDate
			, c1.duration
			, c1.DBName
			, c1.sql_text
			, c1.nt_username
			, c1.server_principal_name
			, c1.mode
			, c1.object_id
			, c1.resource_description
			, c1.resource_owner_type
			, c1.resource_type
			, c1.session_id
			, c1.client_pid
			, c1.client_app_name
			, c1.event_data
			, graph.query('.') AS EventDeadlockGraph
		FROM cte c1
			INNER JOIN cte c2
				ON c1.Event_Seq = c2.Event_Seq - 1
			CROSS APPLY c2.event_data.nodes('(event/data[@name="xml_report"]/value)[last()]/*')
				AS deadlock ( graph )
		WHERE c1.event_name = 'lock_deadlock'
			AND c2.event_name = 'xml_deadlock_report'
		ORDER BY c1.StartTime;

The results from this query will show us the deadlock graph, the event data, as well as several other pieces of data already parsed from the session data for you. And looking at the session data, one can see that the sql_text from each of the queries will demonstrate both the ROWLOCK directive and the directive-free versions of the query. In this query you can also see that I did a little black magic to match up the two event types from the event session (lock_deadlock and xml_deadlock_report). Then I was able to join the two together to produce one row per deadlock event and to see the sql_text with the deadlock graph on one row. Otherwise, the sql_text does not produce with the deadlock_report event. I leave the rest of the query to the reader to discover and question.

From the EventDeadlockGraph column, we could click the cell and take a close look at the XML generated for the deadlock event. Further, if I choose to save the xml as an XDL file and then reopen it in SSMS, I can see the deadlock graphical report as shown in the following.

We see that row locks are still in effect for the update coming from both sides. This further supports that the directive really is just a waste of time in trying to combat this type of deadlock. This is one of those cases where the best option would be to optimize the code and work things out without trying to take a shortcut.

Wrapping Up

Look to optimize the code instead of trying to take a shortcut. In addition, take a look at the deadlocks, the locks held, and the code to get a better understanding of what is truly happening.

This article demonstrates briefly the power of Extended Events while diving into deadlocks. For more on using Extended Events, start reading here! This article may also be of interest.

A powerful little expression in SQL Server is the CASE statement. The CASE statement has the ability to “transform” data per business rules to help better understand the data in a given query. In the simplest form, the CASE statement is a statement that evaluates a list of conditions (in order) and returns the first matching result expressions.

Most uses of the CASE statement are benign enough that one wouldn’t even bat an eye at them. Every now and again, along comes a nifty little query that contains over 300 conditions in the CASE statement. Something that extreme certainly deserves a second (third or even fourth) look. In this article, I am going to explore one such CASE statement in an effort to squeeze more efficiency out of it.

300!

300+ conditions may be a bit hard to imagine. I know I had to double take the list more than once. Immediately upon seeing such an awful piece of code (not pulling punches there), I really wanted to dump the conditions/evaluations into a table and join to it. Eventually, I did put it into a table but not before catching some metrics. Let’s take a look at what this thing looked like from the execution plan properties perspective.

In the preceding image, note the query map in the scrollbar. I have the case statement circled in green. Yes – it is really that large. Next, I have “cached plan size”, “estimated number of rows”, and “estimated subtree cost” circled in red. Each of these shows something that is a little alarming to me. First, the plan size is 48MB! Next, the cost of the query is a little elevated at 51 and change. Last is what seems to be a high value for the estimated number of rows to be returned (only ~6k actually get returned).

On the plus side, the query only takes 381ms, or does it? The query actually takes 3 seconds to complete despite the timing in the properties window. Sometimes, we have to take some of these values in the properties window with a grain of salt.

What if I try to run this query on SQL Server 2017 instead of SQL Server 2014? Let’s take a look at what that might look like.

Firstly, we have a few properties that seem to indicate that we have moved in a slightly better direction with this particular query. The cached plan size has dropped all the way down to just 37MB. The estimated cost dropped to 36 and change and the elapsed time dropped to 353ms. Nothing terribly earth shattering there – but just changing to SQL Server 2017, in this case, we see a bit of an improvement.

That said, take a look at the compile memory and compile time. Memory is 30,544kb and the time is 1,324ms. Aha – we see why this query takes 3 seconds on SQL 2014 and roughly 2 seconds on SQL 2017. High compile times. That memory for the compile is too high for my liking as well so let’s go ahead and take a look at why that might be a little high. It’s time to look at the plan XML and the converted tree for the query (because it is fun).

This is where we get a little geeky with the CASE statement. In the execution plan XML (and the converted tree we will see in a bit), every evaluation criteria for the case statement must have some sort of instruction on how and what should be done. In the case of the XML, we have multiple scalar operators to denote the CASE statement and then each of the evaluations for the CASE statement.

Once we enter the CASE statement, the first thing we see is that there is an “OR” operation (circled in red). Then for each of the values to be compared, we will see an “Equality” operation (circled in green) followed by the value to be evaluated. For just the three values I have in the image, notice how that immediately bloats the execution plan XML for this query.

In the preceding image, we see the converted tree for this query. We see the same pattern evolve here. There is a scalar operator for each value to be evaluated. In this image, I just have the equality comparisons. However, there is also an OR operator that is defined further up in this converted tree. Could this possibly be the reason for the bloated compile time and compile memory?

Let’s evolve that query just a bit and use a table and a join for that particular case statement to see what happens.

Immediately, we can see that the cached plan size improves dramatically! The cached plan size is down to 1.1MB. The compile memory is down to ~4MB and the compile time is down to 174ms. These are all significant improvements but now we see an interesting problem with the memory grant. This query is requesting over 400MB for the memory grant. That should be sounding sirens everywhere. Also of note is the estimated number of rows has dropped to ~28k (a significant improvement in estimate). This query is completing in under 1 second at this point compared to the 3 seconds otherwise but that memory grant is worrying me (and hopefully you too).

For giggles, I went ahead and added a columnstore index to this table (for good reasons beyond the scope of this article). Let’s see what that did.

Notice how the memory grant has dropped to just over 28MB? This is an acceptable improvement for this query. Also note the query execution time improved by a decent amount. This query timing for this query is similar with both a warm cache or cold cache at this point.

Remember that note about the XML and converted tree bloat? With the case statement, the query plan was in excess of 37MB and the revised version is sitting at about 1.1MB. Looking at the quantity of text in each, I see 5819 lines of XML in the version with the case statement and only 3755 lines of XML in the table version of the query (and more room for optimization). The converted tree is more dramatic in that the case statement version of the converted tree requires 3157 lines of instruction and the version that uses a table instead only has 1353 lines of instruction.

What does any of that matter? It helps to illustrate that this 300+ condition filled CASE statement is causing bloat and wasting space along with execution resources.

Wrap

If you ever find yourself writing a query that requires an excessive amount of conditions to be evaluated in a CASE statement, you should reconsider the query design. Too many conditions can lead to plan bloat and excessive memory grants. Keep an eye out for stuff like that!

Thanks for reading! This has been a pseudo deep dive into a really ugly CASE statement. If you would like to read other articles that are less of a deep dive, You can read check out the Back to Basics articles here, or the Extended Events articles here.

One of the frequently required job functions of the database administrator is to track disk space consumption. Whether this requirement comes from management or from a learning opportunity after a production outage, the need exists.

As a hard working DBA, you want to make sure you hit all of the notes to make management sing your praises. Knowing just when the database may fill the drives and prevent a production outage just happens to be one of those sharp notes that could result in a raise and management singing hallelujah. The problem is, how do you do it from within SQL Server? You are just a DBA after all and the disk system is not your domain, right?

Trying to figure it out, you come across a pretty cool function within SQL Server. The name of the function is sys.dm_os_volume_stats. Bonus! This is an excellent discovery, right? Let’s see just how it might work. First a sample query:

SELECT DISTINCT  ovs.logical_volume_name AS VolumeName
		, ovs.volume_mount_point AS DiskDrive
		, ovs.available_bytes AS FreeSpace
		, ovs.total_bytes AS CurrentSize
	FROM sys.master_files mf
		CROSS APPLY sys.dm_os_volume_stats(mf.database_id, mf.file_id) ovs;

If I run that on my local system, I might end up with something that looks like the following:

Looking at the image you may be wondering to yourself right now why I have highlighted a couple of things. You may also be wondering why I used the word “might” in the previous paragraph as well. The reasoning will become more evident as we progress. For now, you have resolved to continue testing the script so execute it again and end up with something that may look like the following (for the same server):

Whoa! What just happened there? Why are there two listings for the C: drive? Why does each register a different value for the FreeSpace column? In addition without any additional usage on the C drive (as verified through other tools) the FreeSpace is changing between executions as well as within the same execution. This is problematic, so you continue testing:

And yet again!

This can’t be correct, can it? Just for giggles let’s modify it just a bit to see if there are any additional clues. Using the following changed script, hopefully a clue will help shed some light on this:

SELECT ovs.logical_volume_name AS VolumeName
		, ovs.volume_mount_point AS DiskDrive
		, ovs.available_bytes AS FreeSpace
		, ovs.total_bytes AS CurrentSize
	FROM sys.master_files mf
		CROSS APPLY sys.dm_os_volume_stats(mf.database_id, mf.file_id) ovs;

This script yields the following potential results:

Look at the different highlighted areas! There are three different values for FreeSpace for the C: drive in this particular execution. The case of the C: drive plays no role in whether the value is recorded differently or not. This seems to be more of a bug within the dynamic management function. From execution to execution, using this particular method, one could end up with duplicate entries but distinct values. The sort of the execution could be returned differently (though we could fix that).

All of these tests were run on my local machine and I really do only have one C: drive. I should never receive multiple entries back for any drive. If using this particular DMF to track space usage, it could be somewhat problematic if the duplicate drive data pops up. How do we get around it, you ask? Here is another example that I have that has not yet produced this duplication:

WITH presel AS (SELECT database_id, file_id,LEFT(mf1.physical_name,3) AS Volume, ROW_NUMBER() OVER (PARTITION BY LEFT(mf1.physical_name,3) ORDER BY mf1.database_id) AS RowNum
					FROM sys.master_files mf1)
,roundtwo AS (SELECT DISTINCT pr.database_id, pr.file_id
				FROM presel pr
				WHERE pr.RowNum = 1)

SELECT ovs.logical_volume_name AS VolumeName
		, ovs.volume_mount_point AS DiskDrive
		, ovs.available_bytes AS FreeSpace
		, ovs.total_bytes AS CurrentSize
	FROM roundtwo mf
		CROSS APPLY sys.dm_os_volume_stats(mf.database_id, mf.file_id) ovs;

Using this version of the script is not terribly more complex, but it will prove to be more reliable. You can see I used some CTEs to provide a little trickery and ensure that I limit my results. What if it is a mount point or a non-standard drive letter? I have not tested that. Let me know how that goes. As you can see, I am restricting the drive selection by using the row_number function against the drive letter.

For alternative reliable methods to find your disk space consumption, I would recommend something different because it is tried and tested. I would recommend using a wmi call to fetch the data. Samples are provided as follows:

/* for mount points, something like this */
EXECUTE sys.xp_cmdshell 'wmic volume get name, freespace, capacity, label'
/* the base wmi query that does not support mount points */
EXECUTE xp_cmdshell 'wmic logicaldisk get name,freespace,size,volumename,blocksize'

Easy peasy, right? Have at it and try tracking your disk space.

Thanks for reading! This has been another article in the Back to Basics series. You can read many more here. For other interesting articles about database space – check here or here.

The second Tuesday of the month comes to us a little early this month. That means it is time again for another group blog party called TSQLTuesday. This party that was started by Adam Machanic has now been going for long enough that changes have happened (such as Steve Jones (b | t) managing it now). For a nice long read, you can find a nice roundup of all TSQLTuesdays over here.

The Why?

Long time friend Andy Leonard (b | t) invites us this month to do a little checkup on ourselves and talk about the “why” around what we do. This could be a very easy topic for some. Equally, this could be a very difficult topic for those same people at different times in their lives. Thus the problem, the topic is simple in nature but sure requires a firm reflection on self and what you have been doing.

The problem for me is less about the “why” behind what I do, and more about how to stretch it out into something more than a few sentences. Think! Think! Think!

Challenges

One of my biggest reasons why I do what I do, boils down to the challenges that I frequently get to encounter. There is a wild satisfaction to working on a very difficult and challenging task, product, tool, profession, skill, etc. This satisfaction often involves reward and a sense of accomplishment.

The challenge can be anything from how to effectively communicate with a difficult person, a tough to find internals problem in SQL Server that could be causing a performance issue, or taking over a project and turning it back from the edge of failure and onto a track of success. Sometimes, the challenge may be as simple as converting a pathetic cursor into a set based approach and gaining an improvement of 100x in performance.

I really do enjoy some of the puzzles (challenges) that I get to work on routinely. This gives me an opportunity to improve my skillset as well as continue to learn. Being able to continually improve is a great motivation for me. The frequent challenges and continual opportunity to learn presents a great opportunity to evolve ones self and career. In a constantly changing world, being able to naturally and easily evolve your personal career is a bonus!

Passion

“Do what you love and you will never work a day in your life.” This is a common saying in the United States. Agree or disagree – there is some truth to it. Being able to do something one loves makes the really hard days a lot easier. Knowing, I may be able to solve a complex problem makes it easier to face the day.

I really enjoy the opportunity to face difficult challenges and resolve those challenges. The passion to solve these puzzles with data doesn’t end there. I also really do enjoy the opportunity to learn which brings up two other challenges that help me learn: speaking and writing.

By putting myself out there regularly to speak and write, I am becoming a better technical person. I am becoming better equipped to solve many of the puzzles I face. Those are great benefits. That said, I don’t feel I could get out there and talk about something about which I wasn’t passionate. I have learned to become passionate about writing and speaking – though I still have plenty of room for improvement (just as I do in my quest to become a good DBA).

Wrapping it Up

I really do enjoy the challenges I get to face on a frequent basis in the world of data. This is the big “WHY” for me to continue my progress in this career.

Find something you are passionate about and strive to envelop your career with as many opportunities to do that thing. If that means accepting some less wanted tasks in order to do more of the thing you love, it could very well be worth it!

Today, I am diverging from the more technical posts that I routinely share. Instead, as the title suggests, I want to dive into a something a little more fun.

Anybody that has interviewed for a job has most likely run into the trick question. Some interviewers like to throw out multiple trick questions all in an effort to trip up the candidate and get the candidate to doubt him/her self. Sure, there can be some benefit to throwing out a trick question or four. One such benefit would be to see how the candidate performs under pressure (see them squirm).

The downside to throwing out trick questions, in my opinion, would be that you can turn a serious candidate into an uninterested candidate. So, when throwing out the tricks, tread carefully.

Let’s take a look at an interview trick question candidate. This is a more technical question and is designed to make you think a little bit. Before reading on to see the answer, I implore that you try to answer the question for yourself legitimately.

How can you insert data into two tables using a single statement without the use of triggers, service broker or some other behind-the-scenes feature?

Are you thinking about it?

Do you have your answer yet?

Now that you have your answer, go ahead and continue reading.

Is your answer to this question something along the lines of “You can’t do that and this is just a trick question”?

Well, honestly, it is a bit of a trick question. But I assure you, you can certainly perform an insert into multiple tables from a single statement. Here is one such setup that demonstrates how you can do this:

CREATE TABLE tin1
	(
		t1id INT IDENTITY(1, 1)
					PRIMARY KEY CLUSTERED
		, someval VARCHAR(100)
	);
CREATE TABLE tin2
	(
		t2id INT IDENTITY(1, 1)
					PRIMARY KEY CLUSTERED
		, someval VARCHAR(100)
	);
GO


INSERT INTO tin1
		( someval )
	OUTPUT INSERTED.someval
		INTO dbo.tin2 ( someval )
	VALUES ( 'double my insert pleasure' );


SELECT *
	FROM dbo.tin1;
SELECT *
	FROM dbo.tin2;

Do you see how I was able to perform that insert into multiple tables? The trick is in using the OUTPUT clause. This little feature in SQL Server can be of great use for things such as building multiple staging tables during an ETL process.

Here is that little trick again just to highlight it.

OUTPUT INSERTED.someval
INTO

Conclusion

There are cases when an interview trick question is suitable. It is when the purported question is truly more technical than trick and is really trying to evaluate your depth and knowledge of SQL Server. The puzzle during the interview boils down to figuring out when it is a trick and when it might not be. Then from there, work your way through possible solutions. But don’t be afraid to admit when you haven’t got a clue. That will be far more impressive than to try and flim-flam the interviewer.

I invite you to share your trick questions in the comments.  Also, how did you solve this particular trick question?

Thanks for reading! This has been another article in the Back to Basics series. You can read many more here.

Today is a day to discuss defaults. It started with the day being TSQL Tuesday and having a topic of “Say No to Defaults.” You can read more about that from the invite – here. I already participated in the party but did also want to discuss defaults a little bit more. That said, this article is not participating in the blog party. That would seem a bit silly.

While, this post is not a part of the party, the defaults to be discussed are fairly important. I have seen severe consequences due to these defaults being ignored and not changed. So today, in addition to my earlier article (you can read it here), I implore you to make some fundamental changes to your production servers with regards to various defaults.

A Trio of msdb Defaults

There aren’t really that many defaults within the msdb database that must be changed, are there? I mean, seriously, beyond the defaults that are available to every database, what could possibly be unique to this database that could have a severe consequence?

I am so glad you asked!

The defaults in the msdb database are more about what is missing than what is actually there. By default, this database is missing quite a few things that could be deemed critical to your environment.

Let’s start with an easy one – Indexes

There are a few out there that may disagree, but the proof really is in the effect on performance for backup jobs and such. I have three indexes I like to put on every instance. I have seen the implementation of these indexes aid in improved job times as well as aid in reduced time to “clean” up the database.

/* Number 1 */
/* msdb maintenance activity related indexes */

CREATE INDEX IX_maintplan_taskdetailid ON [msdb].[dbo].[sysmaintplan_logdetail]
([task_detail_id])
WITH (FILLFACTOR = 90);

CREATE INDEX IX_backupset_type ON msdb.dbo.backupset
([type],[backup_finish_date])
INCLUDE ([media_set_id], [database_name])
WITH (FILLFACTOR = 90);

CREATE INDEX IX_jobactivity_jobid ON [msdb].[dbo].[sysjobactivity]
([job_id])
INCLUDE ([session_id])
WITH (FILLFACTOR = 90);

Easy enough. These indexes are very straight forward and pretty small in the grand scheme of things. But if the index can help improve performance by a factor of 10, then I am in favor of them (and I have seen that performance gain).

Now that we have some supporting indexes to help a bit with performance, we should take a look at the next item. This one can help with job performance as well as help with keeping the msdb database nice and trim.

Data Pruning

I have walked into client instances that had backup history dating all the way back to 2005 and included two-three full backups a day per database with quarter-hourly log backups. Oh and this was for an instance containing well north of 200 databases. Can you say sluggish backups and sluggish msdb overall?

The fix is very easy! Not only do I recommend pruning the backup history, but also the job history, mail history and maintenance plan history (eew – if you use those things). Think about it – do you really need to know that Job XYZ ran successfully in 2006 and only took 15 seconds? This is 2015 and that kind of data is probably not pertinent at this point.

The pruning of this data is not enabled by default! You have to configure this for each of the servers under your purview. Luckily, this is easy to do!

/* NUMBER 2 */
/****** Object:  Job [DBA - Clean msdb Historical data]    Script Date: 03/06/2013 17:40:02 ******/
BEGIN TRANSACTION;
DECLARE @ReturnCode INT;
SELECT @ReturnCode = 0;
/****** Object:  JobCategory [DBA Maintenance]]    Script Date: 03/06/2013 17:40:03 ******/
IF NOT EXISTS ( SELECT name
					FROM msdb.dbo.syscategories
					WHERE name = N'DBA Maintenance'
						AND category_class = 1 )
	BEGIN
		EXEC @ReturnCode = msdb.dbo.sp_add_category @class = N'JOB',
			@type = N'LOCAL', @name = N'DBA Maintenance';
		IF ( @@ERROR <> 0
				OR @ReturnCode <> 0
			)
			GOTO QuitWithRollback;

	END;

DECLARE @jobId BINARY(16);
EXEC @ReturnCode = msdb.dbo.sp_add_job @job_name = N'DBA - Clean msdb Historical data',
	@enabled = 1, @notify_level_eventlog = 0, @notify_level_email = 0,
	@notify_level_netsend = 0, @notify_level_page = 0, @delete_level = 0,
	@description = N'No description available.',
	@category_name = N'DBA Maintenance', @owner_login_name = N'sa',
	@job_id = @jobId OUTPUT;
IF ( @@ERROR <> 0
		OR @ReturnCode <> 0
	)
	GOTO QuitWithRollback;
/****** Object:  Step [Purge Backup History]    Script Date: 03/06/2013 17:40:03 ******/
EXEC @ReturnCode = msdb.dbo.sp_add_jobstep @job_id = @jobId,
	@step_name = N'Purge Backup History', @step_id = 1,
	@cmdexec_success_code = 0, @on_success_action = 3, @on_success_step_id = 0,
	@on_fail_action = 2, @on_fail_step_id = 0, @retry_attempts = 0,
	@retry_interval = 0, @os_run_priority = 0, @subsystem = N'TSQL',
	@command = N'/* Cleanup old backup history */
DECLARE @BackupHistoryOlderThan DATETIME
Set @BackupHistoryOlderThan = DATEADD(d, -90, GETDATE())
EXECUTE msdb.dbo.sp_delete_backuphistory @BackupHistoryOlderThan

', @database_name = N'msdb', @flags = 4;
IF ( @@ERROR <> 0
		OR @ReturnCode <> 0
	)
	GOTO QuitWithRollback;
/****** Object:  Step [Purge MaintPlan Log Data]    Script Date: 03/06/2013 17:40:04 ******/
EXEC @ReturnCode = msdb.dbo.sp_add_jobstep @job_id = @jobId,
	@step_name = N'Purge MaintPlan Log Data', @step_id = 2,
	@cmdexec_success_code = 0, @on_success_action = 3, @on_success_step_id = 0,
	@on_fail_action = 2, @on_fail_step_id = 0, @retry_attempts = 0,
	@retry_interval = 0, @os_run_priority = 0, @subsystem = N'TSQL',
	@command = N'Declare @OldDate DateTime
Set @OldDate = GETDATE() - 14;

Execute msdb.dbo.sp_maintplan_delete_log @oldest_time = @OldDate',
	@database_name = N'msdb', @flags = 4;
IF ( @@ERROR <> 0
		OR @ReturnCode <> 0
	)
	GOTO QuitWithRollback;
/****** Object:  Step [Purge Mail History]    Script Date: 03/06/2013 17:40:04 ******/
EXEC @ReturnCode = msdb.dbo.sp_add_jobstep @job_id = @jobId,
	@step_name = N'Purge Mail History', @step_id = 3, @cmdexec_success_code = 0,
	@on_success_action = 3, @on_success_step_id = 0, @on_fail_action = 2,
	@on_fail_step_id = 0, @retry_attempts = 0, @retry_interval = 0,
	@os_run_priority = 0, @subsystem = N'TSQL',
	@command = N'/* Cleanup old mail items */
DECLARE @MailItemHistoryOlderThan DATETIME
Set @MailItemHistoryOlderThan = DATEADD(d, -14, GETDATE())
EXECUTE msdb.dbo.sysmail_delete_mailitems_sp @sent_before = @MailItemHistoryOlderThan

', @database_name = N'master', @flags = 0;
IF ( @@ERROR <> 0
		OR @ReturnCode <> 0
	)
	GOTO QuitWithRollback;
/****** Object:  Step [Purge Job History]    Script Date: 03/06/2013 17:40:04 ******/
EXEC @ReturnCode = msdb.dbo.sp_add_jobstep @job_id = @jobId,
	@step_name = N'Purge Job History', @step_id = 4, @cmdexec_success_code = 0,
	@on_success_action = 1, @on_success_step_id = 0, @on_fail_action = 2,
	@on_fail_step_id = 0, @retry_attempts = 0, @retry_interval = 0,
	@os_run_priority = 0, @subsystem = N'TSQL',
	@command = N'/*Cleanup of Old Job History*/
Declare @JobHistoryOlderThan DATETIME
Set @JobHistoryOlderThan = DATEADD(d, -14, GETDATE())
EXECUTE msdb.dbo.sp_purge_jobhistory @oldest_date = @JobHistoryOlderThan',
	@database_name = N'master', @flags = 0;
IF ( @@ERROR <> 0
		OR @ReturnCode <> 0
	)
	GOTO QuitWithRollback;
EXEC @ReturnCode = msdb.dbo.sp_update_job @job_id = @jobId, @start_step_id = 1;
IF ( @@ERROR <> 0
		OR @ReturnCode <> 0
	)
	GOTO QuitWithRollback;
EXEC @ReturnCode = msdb.dbo.sp_add_jobschedule @job_id = @jobId,
	@name = N'Daily', @enabled = 1, @freq_type = 4, @freq_interval = 1,
	@freq_subday_type = 1, @freq_subday_interval = 0,
	@freq_relative_interval = 0, @freq_recurrence_factor = 0,
	@active_start_date = 20120831, @active_end_date = 99991231,
	@active_start_time = 0, @active_end_time = 235959;
IF ( @@ERROR <> 0
		OR @ReturnCode <> 0
	)
	GOTO QuitWithRollback;
EXEC @ReturnCode = msdb.dbo.sp_add_jobserver @job_id = @jobId,
	@server_name = N'(local)';
IF ( @@ERROR <> 0
		OR @ReturnCode <> 0
	)
	GOTO QuitWithRollback;
COMMIT TRANSACTION;
GOTO EndSave;
QuitWithRollback:
IF ( @@TRANCOUNT > 0 )
	ROLLBACK TRANSACTION;
EndSave:

GO

If you use this code sample, be sure to adjust the number of days shown in the retention to match your specific needs.

Now we have addressed a couple of defaults in msdb that can impact your performance. We are tidying up the database and in a much happier state these days. There is one more default, though, that is really critical to your data’s well being. This one is set within the msdb database but it really is for all of your databases!

Configuring Alerts!

I’m not talking about just any alerts. There are some very specific alerts that really should be configured. These are the alerts that can help you intervene to minimize corruption.

If you haven’t faced a problem with corruption – you will. It is only a matter of time. Corruption happens. When it happens, the earlier one can intervene, usually the better the outcome. Every minute counts, so why not try to reduce that time as much as possible?

This one is not terribly difficult to implement. I happen to have a query ready to go for that as well. All that needs to be done is a minor adjustment to the alert email address:

/* Number 3 */
USE [msdb];
GO

IF EXISTS ( SELECT name
				FROM msdb.dbo.sysalerts
				WHERE name = N'DBA Alert: Message Id: 823' )
	EXEC msdb.dbo.sp_delete_alert @name = N'DBA Alert: Message Id: 823';
GO

IF NOT EXISTS ( SELECT name
					FROM msdb.dbo.sysalerts
					WHERE name = N'DBA Alert: Message Id: 823' )
	EXEC msdb.dbo.sp_add_alert @name = N'DBA Alert: Message Id: 823',
		@message_id = 823, @severity = 0, @enabled = 1,
		@delay_between_responses = 600, @include_event_description_in = 1,
		@notification_message = N'I/O error detected. This could be a sign of impending I/O failure. Forward this ticket to the primary DBA for this account for followup.',
		@category_name = N'[Uncategorized]',
		@job_id = N'00000000-0000-0000-0000-000000000000';
GO

IF EXISTS ( SELECT name
				FROM msdb.dbo.sysalerts
				WHERE name = N'DBA Alert: Message Id: 824' )
	EXEC msdb.dbo.sp_delete_alert @name = N'DBA Alert: Message Id: 824';
GO

IF NOT EXISTS ( SELECT name
					FROM msdb.dbo.sysalerts
					WHERE name = N'DBA Alert: Message Id: 824' )
	EXEC msdb.dbo.sp_add_alert @name = N'DBA Alert: Message Id: 824',
		@message_id = 824, @severity = 0, @enabled = 1,
		@delay_between_responses = 600, @include_event_description_in = 1,
		@notification_message = N'I/O error detected. This could be a sign of impending I/O failure. Forward this ticket to the primary DBA for this account for followup.',
		@category_name = N'[Uncategorized]',
		@job_id = N'00000000-0000-0000-0000-000000000000';
GO

IF EXISTS ( SELECT name
				FROM msdb.dbo.sysalerts
				WHERE name = N'DBA Alert: Message Id: 825' )
	EXEC msdb.dbo.sp_delete_alert @name = N'DBA Alert: Message Id: 825';
GO

IF NOT EXISTS ( SELECT name
					FROM msdb.dbo.sysalerts
					WHERE name = N'DBA Alert: Message Id: 825' )
	EXEC msdb.dbo.sp_add_alert @name = N'DBA Alert: Message Id: 825',
		@message_id = 825, @severity = 0, @enabled = 1,
		@delay_between_responses = 600, @include_event_description_in = 1,
		@notification_message = N'I/O error detected. This could be a sign of impending I/O failure. Forward this ticket to the primary DBA for this account for followup.',
		@category_name = N'[Uncategorized]',
		@job_id = N'00000000-0000-0000-0000-000000000000';
GO

IF EXISTS ( SELECT name
				FROM msdb.dbo.sysalerts
				WHERE name = N'DBA Alert: Message Id: 829' )
	EXEC msdb.dbo.sp_delete_alert @name = N'DBA Alert: Message Id: 829';
GO

IF NOT EXISTS ( SELECT name
					FROM msdb.dbo.sysalerts
					WHERE name = N'DBA Alert: Message Id: 829' )
	EXEC msdb.dbo.sp_add_alert @name = N'DBA Alert: Message Id: 829',
		@message_id = 829, @severity = 0, @enabled = 1,
		@delay_between_responses = 600, @include_event_description_in = 1,
		@notification_message = N'I/O error detected. This could be a sign of impending I/O failure. Forward this ticket to the primary DBA for this account for followup.',
		@category_name = N'[Uncategorized]',
		@job_id = N'00000000-0000-0000-0000-000000000000';
GO

IF EXISTS ( SELECT name
				FROM msdb.dbo.sysalerts
				WHERE name = N'DBA Alert: Message Id: 832' )
	EXEC msdb.dbo.sp_delete_alert @name = N'DBA Alert: Message Id: 832';
GO

IF NOT EXISTS ( SELECT name
					FROM msdb.dbo.sysalerts
					WHERE name = N'DBA Alert: Message Id: 832' )
	EXEC msdb.dbo.sp_add_alert @name = N'DBA Alert: Message Id: 832',
		@message_id = 832, @severity = 0, @enabled = 1,
		@delay_between_responses = 600, @include_event_description_in = 1,
		@notification_message = N'Page memory error detected. This could be a sign of impending memory failure. Forward this ticket to the primary DBA for this account for followup.',
		@category_name = N'[Uncategorized]',
		@job_id = N'00000000-0000-0000-0000-000000000000';
GO

IF NOT EXISTS ( SELECT name
					FROM msdb.dbo.sysoperators
					WHERE name = N'DBA Support' )
	EXEC msdb.dbo.sp_add_operator @name = N'DBA Support', @enabled = 1,
		@weekday_pager_start_time = 90000, @weekday_pager_end_time = 180000,
		@saturday_pager_start_time = 90000, @saturday_pager_end_time = 180000,
		@sunday_pager_start_time = 90000, @sunday_pager_end_time = 180000,
		@pager_days = 0, @email_address = N'support@DBA.com',
		@category_name = N'[Uncategorized]';
GO
EXEC msdb.dbo.sp_add_notification @alert_name = N'DBA Alert: Message Id: 823',
	@operator_name = N'DBA Support', @notification_method = 1;
EXEC msdb.dbo.sp_add_notification @alert_name = N'DBA Alert: Message Id: 824',
	@operator_name = N'DBA Support', @notification_method = 1;
EXEC msdb.dbo.sp_add_notification @alert_name = N'DBA Alert: Message Id: 825',
	@operator_name = N'DBA Support', @notification_method = 1;
EXEC msdb.dbo.sp_add_notification @alert_name = N'DBA Alert: Message Id: 829',
	@operator_name = N'DBA Support', @notification_method = 1;
EXEC msdb.dbo.sp_add_notification @alert_name = N'DBA Alert: Message Id: 832',
	@operator_name = N'DBA Support', @notification_method = 1;
GO

Wrap

Wow! Now there are three quick defaults that must be changed on every server. These defaults will help improve performance as well as help you stay on top of things when they start to go south (corruption). With timely notifications, and better performance, your servers will be happier, healthier, and longer lasting.

Thanks for reading! This has been another article in the Back to Basics series. You can read many more here.

One probably seldom thinks of the SQL Agent jobs scheduled on the SQL Server instance – unless they fail. What if the job failed because something was changed in the job? Maybe you knew about the change, maybe you didn’t.

Once upon a time, I was in the position of trying to figure out why a job failed. After a bunch of digging and troubleshooting, it was discovered that the job had changed but nobody knew when or why. Because of that, I was asked to provide a low cost audit solution to try and at least provide answers to the when and who of the change.

Tracking who made a change to an agent job should be a task added to each database professionals checklist / toolbox. Being caught off guard from a change to a system under your purview isn’t necessarily a fun conversation – nor is it pleasant to be the one to find that somebody changed your jobs without notice – two weeks after the fact! Usually, that means that there is little to no information about the change and you find yourself getting frustrated.

To the Rescue

When trying to come up with a low to no-cost solution to provide an audit, Extended Events (XE) is quite often very handy. XE is not the answer to everything, but it does come in handy very often. This is one of those cases where an out of the box solution from XE is pretty handy. Let’s take a look at how a session might be constructed to help track agent job changes.

USE master;
GO
-- Create the Event Session
IF EXISTS ( SELECT *
				FROM sys.server_event_sessions
				WHERE name = 'AuditAgentJobs' )
	DROP EVENT SESSION AuditAgentJobs 
    ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE';
GO

CREATE EVENT SESSION [AuditAgentJobs] ON SERVER

ADD EVENT sqlserver.sp_statement_completed ( SET collect_object_name = ( 1 )
	ACTION ( package0.callstack, sqlserver.client_app_name,
	sqlserver.database_name, sqlserver.nt_username, sqlserver.query_hash,
	sqlserver.query_plan_hash, sqlserver.session_id, sqlserver.sql_text,
	sqlserver.tsql_stack,package0.event_sequence )
	WHERE (  [sqlserver].[client_app_name] LIKE 'SQLAgent%'
			AND database_name = 'msdb'
	/* if desire just agent jobsteps try something like 'SQLAgent%JobStep%' */
			) ),
ADD EVENT sqlserver.degree_of_parallelism ( 
	ACTION ( package0.callstack, sqlserver.client_app_name,
	sqlserver.database_name, sqlserver.nt_username, sqlserver.query_hash,
	sqlserver.query_plan_hash, sqlserver.session_id, sqlserver.sql_text,
	sqlserver.tsql_stack,package0.event_sequence )
	WHERE (  database_name = 'msdb'
			and statement_type <> 'Select'
	/* if desire just agent jobsteps try something like 'SQLAgent%JobStep%' */
			) ),
ADD EVENT sqlserver.sql_statement_completed (
	ACTION ( package0.callstack, sqlserver.client_app_name,
	sqlserver.database_name, sqlserver.nt_username, sqlserver.query_hash,
	sqlserver.query_plan_hash, sqlserver.session_id, sqlserver.sql_text,
	sqlserver.tsql_stack,package0.event_sequence )
	WHERE (   [sqlserver].[client_app_name] LIKE 'SQLAgent%'
			AND database_name = 'msdb'
	/* if desire just agent jobsteps try something like 'SQLAgent%JobStep%' */
			) )
ADD TARGET package0.event_file ( SET filename = N'C:\Database\XE\AuditAgentJobs.xel' )
WITH ( MAX_MEMORY = 512000 KB
		, EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
		, MAX_DISPATCH_LATENCY = 5 SECONDS
		, MAX_EVENT_SIZE = 0 KB
		, MEMORY_PARTITION_MODE = NONE
		, TRACK_CAUSALITY = ON
		, STARTUP_STATE = OFF );
GO
ALTER EVENT SESSION AuditAgentJobs ON SERVER
STATE = START;
GO

With this session, I am using degree_of_parallelism as a sort of catch-all in the event that queries that cause a change are not trapped by the other two events (sql_statement_completed and sp_statement_completed). With the degree_of_parallelism event, notice I have a filter to exclude all “Select” statement types. This will trim some of the noise and help track the changes faster.

Looking at data captured by this session, I can expect to see results like the following.

And the degree_of_parallelism event will catch data such as this.

In this example, the deletion of a job was captured by the degree_of_parallelism event. In addition to catching all of the various events that fire as Jobs are being changed and accessed, one will also be able to get a closer look at how SQL Agent runs about its routine.

The Wrap

Extended Events can prove helpful for many additional tasks that may not be thought of on an every day basis. With a little more thought, we can often find a cool solution via Extended Events to help us be better data professionals. In this article, we see one example of that put to use by using XE to audit Agent Job changes.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

For another interesting article about SQL Agent, check this one out!

The second Tuesday of the month comes to us a little early this month. That means it is time again for another group blog party called TSQLTuesday. This party that was started by Adam Machanic has now been going for long enough that changes have happened (such as Steve Jones (b | t) managing it now). For a nice long read, you can find a nice roundup of all TSQLTuesdays over here.

Automation

The theme as chosen by Garry Bargsley (b | t) is about automation. Specifically, Garry has provided two requirements about automation for this month. As is always, there is leeway in a post that participates in TSQL Tuesday.

One of the things that should seem very commonplace to a data professional is the effort to become a lazy DBA. A lazy DBA is not a bad thing. It just means the DBA works hard to automate the repetitive mundane tasks that may be tedious and/or time consuming. Time can always be better spent somewhere else, right?

If you are lacking in any ideas for what can be automated, here are a few TSQL Tuesday roundups from when we have talked about automation previously (yes it is a hot topic – ALWAYS!).

1. August 2010 – Beach Time – what do you automate to earn beach time?
2. February 2011 – Automation in SQL Server – Give your best tricks for making your life easier through automation.
3. January 2014 – Automation – How much of it is the same?
4. September 2015 – The Enterprise – How does one manage an enterprise of databases?
5. September 2017 – PowerShell Automation – Find something and automate it.

In the past, I have written about automation a few times. Some of my favorites are automated restores, automation in the cloud, and my poor mans automated audit.

I automate many processes and have automated loads of tasks over the years. You see, automation means I can spend more time doing other tasks that require more time, more thought/concentration, more effort, and frankly more interest. So what have I automated recently that may be different from what I have previously written? This time, I have something that may seem utterly trivial but in the end it is rather tedious and time consuming to manually check over and over and over.

PowerShell

When I automate a task, I generally will try to use the tool that seems the most appropriate for the task: windows scheduler, SQL Agent, TSQL, SSIS, VB, C#  and now I am trying to add PoSh to that list. I don’t believe there is a one size fits all automation tool. Sometimes, one has to be flexible enough to adapt other technologies into the tool-belt.

I have been working with a client to check their servers for SQL Server version, SSMS version, PoSH version and so on. All of this to try and get the appropriate updates installed on the server. Believe it or not, many of their servers were still running PoSH v2 and didn’t have any Service Packs installed for their database servers. OUCH!

Touching every single server (even if it is only 10 servers) is far too tedious and error prone. So, I spent a little time klooging together with my neanderthal level PoSH skills and found a way to retrieve various pieces of information from the servers and then store those data points in a database so I could report on the entire environment easily with TSQL. In addition, I could show change history and find approximately (at worst) when an update was installed.

Of all of the things I scripted to start tracking, the one I want to share this time can also be used to audit security on each of the database servers. I use the following script to audit the localadmins on each of the database servers in the enterprise. In order to trap each local admin on the server, I also recurse through domain groups to find all users of a group to find everybody that may have access. Here is a version of the script that is similar to what I use now.

Param (
[string]$TypeName = $null #"myserver_type"
,[string]$svrname = "" #"myserver"
)

import-module sqlserver
import-module activedirectory

$c = Get-Credential -Credential mydomain\mylogin

$dtime = Get-Date -Format "yyyy-MM-dd HH:mm:ss.fff"
$ScriptProcess = "Get-AdminsV2"

$Result = [PSCustomObject] @()

IF($svrname){
$SQLServerList = [ordered]@{
    ServerName = $svrname}
}
else {
if($TypeName){
$SQLServerList = Invoke-Sqlcmd -serverinstance myserver -database "DBA" -query "select distinct ServerName From Mgmt.ServerList sl inner join [Mgmt].[ServerType] st on sl.ServerTypeID = st.TypeID where st.TypeName = '$TypeName'"
}
else{
$SQLServerList = Invoke-Sqlcmd -serverinstance myserver -database "DBA" -query "select distinct ServerName From Mgmt.ServerList sl inner join [Mgmt].[ServerType] st on sl.ServerTypeID = st.TypeID where st.TypeID <> 7;"
}
}

foreach ($server in $SQLServerList)
{
$svrname = $server.ServerName

If  (Test-Connection -ComputerName  $svrname -Count  1 -Quiet ) {
try {
$dtime = get-date
$skipchecks = New-PSSessionOption -SkipCACheck -SkipCNCheck -SkipRevocationCheck

 Try  { 
$admins = Invoke-Command -Computername $svrname -ErrorAction Stop -SessionOption $skipchecks -Credential $c -ScriptBlock {  Get-LocalGroupMember -Group "Administrators" }
<#$admins = $admins |? {$_.groupcomponent –like '*"Administrators"'}  
  
$admins |% {  
$_.partcomponent –match “.+Domain\=(.+)\,Name\=(.+)$” > $nul  
$admin = $matches[1].trim('"') + “\” + $matches[2].trim('"') 
#>
}
Catch {
  $EObject = [ordered]@{
  ScriptErrorId = ''
  Computername = $svrname
  ScriptError = $_.ToString()
  ScriptProcess = $ScriptProcess
  PollDate = $dtime
  }
  $EResult = (New-Object -TypeName PSCustomObject -Property $EObject)

  # $EResult
  Write-SqlTableData -ServerInstance "myserver" -DatabaseName "DBA" -SchemaName "Monitor" -TableName "ScriptErrors" -Force -InputData $EResult

   # Continue 
  }
  #$admins
 Foreach ($admin in $admins) {
  
    $Adm = $admin.Name.ToString()
    $ObjClass = $admin.objectclass.ToString()
<#
    $admin
    $Adm
    $Admin.Name
    $Admin.ObjectClass
    $ObjClass
    #>

  If ($ObjClass -match 'Group') {
  #"Group Matched"
  $domain,$group = $adm.split('\')
  #$domain
  #$group
    $members = Get-ADGroupMember -Identity "$group" -Recursive
    $grouphierarchy = $Adm

    function Get-GroupHierarchy ($group)
        {
        import-module activedirectory
        $groupMember = get-adgroupmember $group | sort-object objectClass -descending
           foreach ($member in $groupMember)
            {
            if ($member.ObjectClass -match "group")
                {
                $grouphierarchy += "\^\$($domain)\$($member.name)"
                $grouphierarchy
                Get-GroupHierarchy $member.name}}
        }
 
    $grouphierarchy += Get-GroupHierarchy $group
    #$grouphiho.ToString()
    #$grouphierarchy.tostring()

  $Object = [ordered]@{
  AdminId = ''
  Computername = $svrname
  Admin = $Adm
  ScriptProcess = $ScriptProcess
  PollDate = $dtime
}

  $Result = (New-Object -TypeName PSCustomObject -Property $Object)
  #"Result before write to sql attempt"

  Write-SqlTableData -ServerInstance "myserver" -DatabaseName "DBA" -SchemaName "Monitor" -TableName "LocalAdmins" -Force -InputData $Result
  $Result.Clear


    foreach ($member in $members) {
 
 #$member.name

    $Object = [ordered]@{
    AdminId = ''
    Computername = $svrname
    # Admin = "$($domain)\$($group)\^\$($domain)\$($member.name)"
    Admin = "$($grouphierarchy)\^\$($domain)\$($member.name)"
    ScriptProcess = $ScriptProcess
    PollDate = $dtime
    }
  
  $Result = (New-Object -TypeName PSCustomObject -Property $Object)

  Write-SqlTableData -ServerInstance "myserver" -DatabaseName "DBA" -SchemaName "Monitor" -TableName "LocalAdmins" -Force -InputData $Result
  $Result.Clear
  }

 }
  ELSE {
  #"else reached"
    $Object = [ordered]@{
    AdminId = ''
    Computername = $svrname
    Admin = $Adm
    ScriptProcess = $ScriptProcess
    PollDate = $dtime
    }
  
  $Result = (New-Object -TypeName PSCustomObject -Property $Object)

  Write-SqlTableData -ServerInstance "myserver" -DatabaseName "DBA" -SchemaName "Monitor" -TableName "LocalAdmins" -Force -InputData $Result
  $Result.Clear
 
 }
}
 #$Result |FT -AutoSize
}
catch{}
}
}

Could I improve on the efficiency of this script? Most definitely I believe there is room for improvement. Remember, I am very novice at my PoSH skills. Scripting issues aside, it works and basically fetches a list of servers from a database, then iterates through each of those servers to fetch the complete list of local admins on each of the servers. Then the script writes out the complete list of admins for each server back to my database so I can generate a history of changes to the admins or report on who has admin access on the server.

For anybody that has admin access to a database server, the permission path (nested group path) is recorded in hierarchical form separated by the carrot character (^). Using this script, I have been able to provide a report to domain admins to clean out various unwanted individuals from access that was not intended or necessary.

Wrapping it Up

Automation is an essential tool for every data professional. Wait, no, that’s not accurate. Automation is an essential tool in all facets of IT. Automation is a definitive method to work more efficiently and offload some of the mundane repetitive tasks that consume too much time.

Even if the task is not trivial but needs to be repeated and done so without error, the best tool is automation. Performing tasks over and over naturally leads to higher risk of error. The way to minimize that risk is to perform the task via some automation script or routine.

It isn’t very often that one would consider a short circuit to be a desired outcome. In SQL Server we have a cool exception to that rule – Extended Events (XE).

What exactly is a short circuit and why would it be undesirable in most cases? I like to think of a short circuit as a “short cut” in a sense.

I remember an experience that happened while running a marathon many years ago. A person I had pulled up next to and started chatting with needed to use the restroom. I continued along on the course and a mile later I saw the same person suddenly reappear on the course ahead of me. This person had found a short cut on the course and decided to use it. If caught, he would have been disqualified. He may have saved himself a mile of running and gotten a better time, but the act was to take a course that was not the intended official course for that race.

In electricity, a short circuit does a similar thing. The electricity will follow the path of least resistance. Sometimes, this means the unofficial desired path for the current to flow. The end result can be very bad in electrical terms as an overload may occur which can cause overheating and sparking.

Why would we want an overload?

In electricity and mechanical parts, we really don’t want anything to cause short cuts in the system. On the other hand, when we are dealing with tracing and anything that can put a load on the system, we want that load to be as small as possible.

Trying to trace for problems in the SQL Server engine comes with a cost. That cost comes in the form of additional resource requirements which could mean fewer resources available for the engine to process user requests. None of us wants for the end-user to be stuck waiting in a queue for resources to free due to our tracing activities (i.e. Profiler). So a lightweight method (to trace) is needed.

XE is that lightweight method. A big part of the reason for that is the ability of XE to short-circuit (short-cut) to the end result. How can an XE session short-circuit? Think logic constraints and predicates. I previously demonstrated how to short-cut the system by using a counter in the predicate, but the short circuit isn’t constrained to just a counter in the predicate. The short-circuit is super critical to performance and success, but it is often misunderstood and poorly explained. So, I am trying to explain it again – better.

If we follow the principle that a short-circuit is the path of least resistance, we have a construct for how to build the predicate for each event in a session. Think of it as path of least work. Just like with children, XE and electricity will evaluate each junction with a bit of logic. Do I have to do more work if I go down this path or less work? Less work? Great, I am going in this direction.

As an event is fired off and is picked up by the XE session, the session compares that event payload to the conditions in the predicate. Everything in the predicate is processed in precise order – until a predicate condition fails the comparison (or result is false). Immediately when a condition results to negative (false) then the XE session jumps right to the end and closes. Nothing more is processed.

This is why predicate order matters. If a predicate evaluates to false, the short-circuit is invoked and the evaluation ends. With that in mind, what is the most desirable condition in the predicate to be first?

I have heard multiple people state that the “most likely to succeed” predicate should be first. Well, if the “most likely success” is first what does that mean for your session? It will have to do more work! That is exactly the model that Profiler used (uses) and we all know what happens with Profiler and performance!

No! We don’t want the most likely to succeed to be the first predicate to be evaluated. We want the least likely to succeed to be first. This means less work – just as illustrated in the previous image where the short-circuit is represented by the red line. If you would like, we can also call each of the three light-bulbs “predicates” and the switch would be the event (nothing is traced in the session if the event doesn’t even match).

Which Comes First?

This brings us to the hard part. How should one order the predicates for each event? The answer to that is not as cut and dry as you probably want. There are many variables in the equation. For instance, the first variable would be the environment. Each SQL environment is different and that makes a difference in evaluating events and predicates. However, lets use a common-ish set of criteria and say we need to decided between database name and query duration.

ADD EVENT sqlserver.sp_statement_completed ( 
	SET collect_object_name = ( 1 )
	WHERE ( [sqlserver].[database_name] = 'AdventureWorks2014'
			AND [duration] > ( 5000000 ) -- 5 seconds
)

The questions in this case now comes down to 1) how many databases are on the server? and 2) what are the chances of a query lasting more than 5 seconds? If you have 100 databases on the server and 99 of them frequently see queries over 5 seconds, then this predicate order would make sense. What if you have only 4 databases and a query over 5 seconds occurs roughly 1 in 10,000 times? Then the predicate order should be switched to the following.

ADD EVENT sqlserver.sp_statement_completed ( 
	SET collect_object_name = ( 1 )
	WHERE ( [duration] > ( 5000000 ) -- 5 seconds
			AND [sqlserver].[database_name] = 'AdventureWorks2014'
)

If you don’t have a database by the name of “AdventureWorks2014” then the database name predicate would remain first but really it should be changed to an appropriate database name that exists.

The Wrap

Predicate order in an XE session is very important. A well designed predicate can lead to a highly tuned and well performing trace that will ease your life as a data professional. Just remember, contrary to various people out there, the most desirable predicate order is to have the “least likely to succeed” first and the “most likely to succeed” should be last.

And yes, we truly do want our XE sessions to short-circuit! As we aspire to do less busy work, an XE session should be configured to do as little work as is necessary.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This has been the eleventh article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

Cool new toys/tools have been made available to the data professional. Among these tools are query data store and automatic tuning. These two tools actually go hand in hand and work pretty nicely together.

With most new tools, there is usually some sort of instruction manual along with a section on how to troubleshoot the tool. In addition to the manual, you usually have some sort of guide as to whether or not the tool is working within desired specifications or not.

Thanks to Extended Events (XE), we have access to a guide of sorts that will help us better understand if our shiny new tool is operating as desired.

Operationally Sound

XE provides a handful of events to help us in evaluating the usage of Automatic Tuning in SQL Server. To find these events, we can simply issue a query such as the following.

select xo.name, xo.object_type, xo.description
	From sys.dm_xe_objects xo
	where xo.name like '%autom%tun%'
	or xo.description like '%autom%tun%'
	order by xo.object_type;

When executed, this query will provide a result set similar to the following.

I have grouped the results from this query into three sets. In the red set, I have four events that are useful in the diagnostics and monitoring of automatic tuning. These events show errors, diagnostic (and performance) data, configuration changes and state changes.

For instance, the state change event will fire when automatic tuning is enabled and will also fire when the database is started (assuming the session is running). The automatic_tuning_diagnostics event fires roughly every 30 minutes on my server to gather performance and diagnostic data that can help me understand how well the feature is performing for my workload in each database.

Highlighted in the green section is a couple of maps that show the various values for the current phase or state of the automatic tuning for each database. One can view these different values with the following query.

select mv.name,mv.map_key,mv.map_value
	From sys.dm_xe_map_values mv
	where mv.name in ('aprc_state','aprc_task')
	order by mv.name, mv.map_key;

This query yields these results.

We will see those values in use in the events in a session shortly.

We have seen some of the events and some of the maps at a very quick glance. That said, it is a good time to pull it all together and create a session.

USE master;
GO
-- Create the Event Session
IF EXISTS ( SELECT *
				FROM sys.server_event_sessions
				WHERE name = 'AutoTuneMonitor' )
	DROP EVENT SESSION AutoTuneMonitor
    ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE';
GO

CREATE EVENT SESSION [AutoTuneMonitor] ON SERVER

ADD EVENT qds.automatic_tuning_config_change(
ACTION ( sqlserver.database_id, 
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname)
	  )
,ADD EVENT qds.automatic_tuning_diagnostics(
ACTION ( sqlserver.database_id, 
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname)
  )
  ,ADD EVENT qds.automatic_tuning_state_change(
  ACTION ( sqlserver.database_id, 
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname)
	)
  ,ADD EVENT qds.automatic_tuning_error(
  ACTION ( sqlserver.database_id, 
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname)
	)
ADD TARGET package0.event_file(SET filename=N'C:\Database\XE\AutoTuneMonitor.xel')
,ADD TARGET package0.ring_buffer
 WITH (MAX_DISPATCH_LATENCY=5 SECONDS,TRACK_CAUSALITY=ON,STARTUP_STATE=OFF);
GO

ALTER EVENT SESSION [AutoTuneMonitor] ON SERVER STATE = START;
GO

Seeing as this session won’t produce any results without Query data store being enabled and automatic tuning being configured for a database, I have set all of that up in a demo database and have some fresh results to display.

Here I show an example of the output filtered for just the diagnostics event. Note the phase_code shows some of those map values previously discussed. I can also see that roughly every 30 minutes each database undergoes a diagnostics check.

Now, looking at another event in that same session, I can see the following.

The state_code in this event payload demonstrates more values from the maps previously discussed (CorrectionEnabled and DetectionEnabled). In this case, the automatic_tuning_state_change fired a few times for database 6 because that database was intentionally taken offline and set back online to test the event.

The use of these particular events in this session is very lightweight. I don’t have a predicate configured for any of the events because I wanted to trap everything. Of course, the number of events can increase with an increased load and usage scenarios on different servers.

The Wrap

Automatic tuning can be a pretty sharp tool in your tool-belt on your way to becoming that rock-star DBA. As you start to sharpen your skills with this tool, you will need to have some usage and diagnostic information at your fingertips to ensure everything is running steady. This event session is able to provide that diagnostic information and keep you on top of the automatic tuning engine.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This has been the eleventh article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

With all of that in mind, I feel this is a really cool feature and it is something that can be useful in bringing the groups together that most commonly butt heads in IT (Ops, DBA, Devs) by creating a commonality in trace data and facts. There may be more on that later!

Target Rich

The ETW target is a trace file that can be merged with other ETW logs from Windows or applications (if they have enabled this kind of logging). You can easily see many default ETW traces that are running or can be run in Windows via Perfmon or from the command line with the following command.

logman -ets

And from the gui…

Finding the traces is not really the difficult part with this type of trace. The difficult parts (I believe) come down to learning something new and different, and that Microsoft warns that you should have a working knowledge of it first (almost like a big flashing warning that says “Do NOT Enter!”). Let’s try to establish a small knowledgebase about this target to ease some of the discomfort you may now have.

One can query the DMVs to get a first look at what some of the configurations may be for this target (optional and most come with defaults already set).

SELECT	xo.name AS TargetName
	, xoc.name AS Attribute
	, xoc.type_name
	, xoc.column_type
	, xoc.column_value AS DefaultValue
	, xoc.description
FROM	sys.dm_xe_objects xo
		INNER JOIN sys.dm_xe_object_columns xoc
			ON xo.name = xoc.object_name
			AND xo.package_guid = xoc.object_package_guid
WHERE xo.name = 'etw_classic_sync_target';

Six configurations in total are available for the ETW target. In the query results (just above) you will see that the default value for each configuration option is displayed. For instance, the default_xe_session_name has a default value of XE_DEFAULT_ETW_SESSION. I like to change default names and file paths, so when I see a name such as that, rest assured I will change it. (Contrary to popular belief, the path and session name default values can certainly be changed.)

As I go forward into creating an XE session using the ETW target, it is important to understand that only 1 ETW session can exist. This isn’t a limitation of SQL Server per se, rather a combination of the use of the ETW Classic target (for backwards compatibility) and Windows OS constraints. If the ETW target is used in more than one XE session on the server (even in a different SQL Server instance), then all of them will use the same trace target in windows (consumer). This can cause a bit of confusion if several sessions are running concurrently.

My recommendation here is to use a very precise and targeted approach when dealing with the ETW target. Only run it for a single XE session at a time. This will make your job of correlating and translating the trace much easier.

The ETW target is a synchronous target and does NOT support asynchronous publication. With the synchronous consumption of events by the target, and if you have multiple sessions with the same event defined, the event will be consumed just a single time by the ETW target. This is a good thing!

Two more tidbits about the ETW target before creating an event session and looking at more metadata. The default path for the target is %TEMP%\<filename>.etl. This is not defined in the configuration properties but is hardwired. Any ideas why one might want to specify a different path? I don’t like to use the temp directory for anything other than transient files that are disposable at any time!

Whether you change the directory from the default or leave it be, understand that it cannot be changed after the session starts – even if other sessions use the same target and are started later. However, if you flush the session and stop it, then you can change it. I do recommend that it be changed!

Second tidbit is that other than the classic target, ETW does have a manifest based provider. Should Extended Events (XE) be updated to use the manifest based provider then some of the nuances will disappear with translating some of the trace data (future article to include ntrace and xperf – stay tuned). For now, understand that viewing the ETW trace data is not done via SQL Server methods. Rather, you need to view it with another tool. This is due to the fact that the ETW is an OS level trace and not a SQL Server trace.

Session Building

If it is not clear at this point, when creating an XE session that utilizes the ETW target, two traces are, in essence, created. One trace is a SQL server (XE) trace that can be evaluated within SQL Server. The second trace is an ETW trace that is outside the realm of SQL Server and thus requires new skills in order to review it. Again, this second trace can be of extreme help because it is more easily merged with other ETW traces (think merging perfmon with sql trace).

When I create a session with an ETW target, it would not be surprising to see that I have two targets defined. One target will be the ETW target and a second may be a file target or any of the others if it makes sense. The creation of two targets is not requisite for the XE session to be created. The XE data will still be present in the livestream target even without a SQL related target.

Before creating a session, I need to cover a couple of possible errors that won’t be easy to find on google.

Msg 25641, Level 16, State 0, Line 101 For target, “5B2DA06D-898A-43C8-9309-39BBBE93EBBD.package0.etw_classic_sync_target”, the parameter “default_etw_session_logfile_path” passed is invalid.
The operating system returned error 5 (ACCESS_DENIED) while creating an ETW tracing session.
ErrorFormat: Ensure that the SQL Server startup account is a member of the ‘Performance Log Users’ group and then retry your command.

I received this error message even with my service account being a member of the “Performance Log Users” windows group. I found that I needed to grant explicit permissions to the service account to the logging directory that I had specified.

Msg 25641, Level 16, State 0, Line 105 For target, “5B2DA06D-898A-43C8-9309-39BBBE93EBBD.package0.etw_classic_sync_target”, the parameter “default_xe_session_name” passed is invalid.
The default ETW session has already been started with the name ‘unknown‘.
Either stop the existing ETW session or specify the same name for the default ETW session and try your command again.

This error was more difficult than the first and probably should have been easier. I could not find the session called ‘unknown’ hard as I might have tried. Then it occurred to me (sheepishly) that the path probably wanted a file name too. If you provide a path and not a filename for the trace file, then this error will nag you.

I found both error cases to be slightly misleading but resolvable quickly enough.

USE master;
GO
-- Create the Event Session
IF EXISTS ( SELECT *
				FROM sys.server_event_sessions
				WHERE name = 'SVRLoginAudit_etw' )
	DROP EVENT SESSION SVRLoginAudit_etw
    ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE';
GO

CREATE EVENT SESSION [SVRLoginAudit_etw]
	ON SERVER
	ADD EVENT sqlserver.login
	( SET collect_database_name = ( 1 )
		, collect_options_text = ( 1 )
	ACTION ( package0.collect_system_time
		, package0.event_sequence
		, sqlserver.client_app_name
		, sqlserver.client_connection_id
		, sqlserver.client_hostname
		, sqlserver.context_info
		, sqlserver.database_id
		, sqlserver.database_name
		, sqlserver.nt_username
		, sqlserver.server_principal_name
		, sqlserver.session_id
		, sqlserver.session_nt_username
		, sqlserver.sql_text
		, sqlserver.username ))
	ADD TARGET package0.event_file
	( SET filename = N'C:\Database\XE\SVRLoginAudit_etw.xel'
		, max_file_size = ( 5120 ), max_rollover_files = ( 6 )),
 ADD TARGET package0.etw_classic_sync_target ( --keeping mostly default values
	SET default_etw_session_logfile_path = 'C:\Database\XE\SVRLoginAudit_etw.etl' 
	, default_xe_session_name = 'NotYourETWXESession' -- changed from XE_DEFAULT_ETW_SESSION
	, default_etw_session_buffer_size_kb = 128
	, default_etw_session_logfile_size_mb = 20
	, retries = 0
	, deferred_etw_creation = 0
	)
	WITH ( MAX_MEMORY = 4096KB
		, EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
		, MAX_DISPATCH_LATENCY = 30 SECONDS
		, MAX_EVENT_SIZE = 0KB
		, MEMORY_PARTITION_MODE = NONE
		, TRACK_CAUSALITY = ON
		, STARTUP_STATE = OFF );
GO

The session is pretty straight forward here. I am just auditing logins that occur on the server and sending them to both the ETW and event_file targets. To validate the session is created and that indeed the ETW session is not present in SQL Server, I have the following script.

SELECT ISNULL(ses.name,xse.name) AS SessionName
		, CASE
			WHEN ISNULL(ses.name,'') = ''
			THEN 'Private'
			ELSE 'Public'
			END AS SessionVisibility
		, CASE
			WHEN ISNULL(xse.name,'') = ''
			THEN 'NO'
			ELSE 'YES'
			END AS SessionRunning
		, CASE
			WHEN ISNULL(xse.name,'') = ''
				AND ISNULL(ses.name,'') = ''
			THEN 'NO'
			ELSE 'YES'
			END AS IsDeployed
	FROM sys.server_event_sessions ses
	FULL OUTER JOIN sys.dm_xe_sessions xse
		ON xse.name =ses.name
	WHERE ses.name IN ('NotYourETWXESession','XE_DEFAULT_ETW_SESSION','SVRLoginAudit_etw')
		OR xse.name IN ('NotYourETWXESession','XE_DEFAULT_ETW_SESSION','SVRLoginAudit_etw')
	ORDER BY ses.event_session_id;
GO
ALTER EVENT SESSION SVRLoginAudit_etw ON SERVER
STATE = START;

SELECT ISNULL(ses.name,xse.name) AS SessionName
		, CASE
			WHEN ISNULL(ses.name,'') = ''
			THEN 'Private'
			ELSE 'Public'
			END AS SessionVisibility
		, CASE
			WHEN ISNULL(xse.name,'') = ''
			THEN 'NO'
			ELSE 'YES'
			END AS SessionRunning
		, CASE
			WHEN ISNULL(xse.name,'') = ''
				AND ISNULL(ses.name,'') = ''
			THEN 'NO'
			ELSE 'YES'
			END AS IsDeployed
	FROM sys.server_event_sessions ses
	FULL OUTER JOIN sys.dm_xe_sessions xse
		ON xse.name =ses.name
	WHERE ses.name IN ('NotYourETWXESession','XE_DEFAULT_ETW_SESSION','SVRLoginAudit_etw')
		OR xse.name IN ('NotYourETWXESession','XE_DEFAULT_ETW_SESSION','SVRLoginAudit_etw')
	ORDER BY ses.event_session_id;
GO

Despite the absence of the ETW session from SQL Server, I can still easily find it (again either shell or from the perfmon gui). Here is what I see when checking for it from a shell.

Even though the session (or session data) is not visible from SQL Server, I can still find out a tad more about the target from the XE related DMVs and catalog views.

SELECT xse.name AS SessionName, xse.buffer_policy_desc, xse.flag_desc,est.name AS Targetname
		,CASE WHEN est.name = 'event_file'
			THEN sesfo.filename
			ELSE NULL END AS filename
		,CASE WHEN est.name = 'event_file'
			THEN sesfo.max_file_size
			ELSE NULL END AS max_file_size
		,CASE WHEN est.name = 'event_file'
			THEN sesfo.max_rollover_files
			ELSE NULL END AS max_rollover_files
		,CASE WHEN est.name = 'event_file'
			THEN NULL
			ELSE sesfo.default_etw_session_logfile_path END AS default_etw_session_logfile_path
		,CASE WHEN est.name = 'event_file'
			THEN NULL 
			ELSE sesfo.default_xe_session_name END AS default_xe_session_name
		,CASE WHEN est.name = 'event_file'
			THEN NULL 
			ELSE sesfo.default_etw_session_buffer_size_kb END AS default_etw_session_buffer_size_kb
		,CASE WHEN est.name = 'event_file'
			THEN NULL 
			ELSE sesfo.default_etw_session_logfile_size_mb END AS default_etw_session_logfile_size_mb
	FROM sys.server_event_sessions ses
	INNER JOIN sys.dm_xe_sessions xse
		ON xse.name =ses.name
	INNER JOIN sys.server_event_session_targets est
		ON ses.event_session_id = est.event_session_id
	INNER JOIN (
			SELECT *
				FROM    (
					SELECT sesf.event_session_id, sesf.name AS FieldName, sesf.value AS FieldValue
						FROM sys.server_event_session_fields sesf
					) fields
					PIVOT   (
						max(FieldValue)
						FOR FieldName IN (begin_event
								,begin_matching_actions
								,begin_matching_columns
								,collect_cached_text
								,collect_call_stack
								,collect_column_list
								,collect_data
								,collect_database_name
								,collect_feature_data
								,collect_object_name
								,collect_options_text
								,collect_plan_handle
								,collect_resource_description
								,collect_statement
								,end_event
								,end_matching_actions
								,end_matching_columns
								,filename
								,filtering_event_name
								,max_events_limit
								,max_file_size
								,max_memory
								,max_rollover_files
								,metadatafile
								,occurrence_number
								,source
								,source_type
								,default_etw_session_logfile_path
								,default_etw_session_buffer_size_kb
								,default_etw_session_logfile_size_mb
								,retries
								,default_xe_session_name
								,deferred_etw_creation)
					) PVT
				) sesfo
		ON est.event_session_id = sesfo.event_session_id
	WHERE ses.name IN ('NotYourETWXESession','XE_DEFAULT_ETW_SESSION','SVRLoginAudit_etw');

Running that query will result in something similar to this:

The Wrap

I have just begun to scratch the surface of the ETW target. This target can provide plenty of power for troubleshooting when used in the right way. The difficulty may seem to be getting to that point of knowing what the right way is. This target may not be suitable for most troubleshooting issues – unless you really need to correlate real windows metrics to SQL metrics and demonstrate to Joe Sysadmin that what you are seeing in SQL truly does correlate to certain conditions inside of windows. Try it out and try to learn from it and figure out the right niche for you. In the interim, stay tuned for a follow-up article dealing with other tools and ETW.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This has been the tenth article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

When contention occurs, the users will probably start screaming because performance has tanked and deadlines are about to be missed. There are many different ways (e.g. here or here) to try and observe the memory conditions and even troubleshoot memory contention. Extended Events (XE) gives one more avenue to try and troubleshoot problems with memory.

Using XE to observe memory conditions is a method that is both geeky/fun and an advanced technique at the same time. If nothing else, it will certainly serve as a divergence from the mundane and give you an opportunity to dive down a rabbit hole while exploring some SQL Server internals.

Diving Straight In

I have a handful of events that I have picked for an event session to track when I might be running into some memory problems. Or I can run the session when I suspect there are memory problems to try and provide me with a “second opinion.” Here are the pre-picked events.

SELECT xo.name AS EventName, xo.capabilities_desc, xo.description
	FROM sys.dm_xe_objects xo
	WHERE xo.name IN ('large_cache_memory_pressure','buffer_manager_page_life_expectancy'
		,'buffer_node_page_life_expectancy'
		,'buffer_pool_eviction_thresholds_recalculated','server_memory_change');

Investigating those specific events a little further, I can determine if the payload is close to what I need.

SELECT oc.OBJECT_NAME AS EventName
		,oc.name AS column_name, oc.type_name
		,oc.column_type AS column_type
		,oc.column_value AS column_value
		,oc.description AS column_description
		,ca.map_value AS SearchKeyword
		,ch.Channel --added because large_cache_memory_pressure has no category/search keyword
	FROM sys.dm_xe_object_columns oc
		OUTER APPLY (SELECT TOP 1 mv.map_value
						FROM sys.dm_xe_object_columns occ
						INNER JOIN sys.dm_xe_map_values mv
							ON occ.type_name = mv.name
							AND occ.column_value = mv.map_key
							AND oc.object_package_guid = mv.object_package_guid
						WHERE occ.name = 'KEYWORD'
							AND occ.object_name = oc.object_name) ca
		INNER JOIN (SELECT c.object_name AS EventName,c.object_package_guid AS PkgGuid, v.map_value AS Channel
					FROM sys.dm_xe_object_columns c
						INNER JOIN sys.dm_xe_map_values v
							ON c.type_name = v.name
								AND c.column_value = CAST(v.map_key AS NVARCHAR)
					WHERE c.name = 'channel') ch
			ON ch.EventName = oc.object_name
			AND ch.PkgGuid = oc.object_package_guid
	WHERE oc.object_name IN ('large_cache_memory_pressure','buffer_manager_page_life_expectancy'
			,'buffer_node_page_life_expectancy','buffer_pool_eviction_thresholds_recalculated'
			,'server_memory_change')
		AND oc.column_type <> 'readonly' 
	ORDER BY EventName,oc.column_id;

That is a small snippet of the payload for all of the pre-picked events. Notice that the large_cache_memory_pressure event has no “SearchKeyword” / category defined for it. There are a few other events that also do not have a category assigned which makes it a little harder to figure out related events. That said, from the results, I know that I have some “server” and some “memory” tagged events, so I can at least look at those categories for related events.

/* Keyword search */
DECLARE @Keyword VARCHAR(64) = 'memory' --memory --server
/* note there are a few more interesting events
buffer_node_database_pages
buffer_manager_database_pages
buffer_manager_target_pages
memory category
query_memory_grant_usage
memory_manager_stolen_server_memory
allocation_failure
bad_memory_detected
*/
SELECT oc.OBJECT_NAME AS EventName
		,oc.name AS column_name, oc.type_name
		,oc.column_type AS column_type
		,oc.column_value AS column_value
		,oc.description AS column_description
		,REVERSE(LEFT(REVERSE(olm.name),CHARINDEX('\',REVERSE(olm.name))-1)) AS DLLName
		,olm.file_version
		,xp.name AS PackageName
		,mv.map_value AS SearchKeyword
		,ch.Channel
	FROM sys.dm_xe_object_columns oc
		INNER JOIN sys.dm_xe_map_values mv
			ON oc.type_name = mv.name
			AND oc.column_value = mv.map_key
			AND oc.object_package_guid = mv.object_package_guid
			AND oc.name = 'KEYWORD'
		INNER JOIN sys.dm_xe_packages xp
			ON oc.object_package_guid = xp.guid
		INNER JOIN sys.dm_os_loaded_modules olm
			ON xp.module_address = olm.base_address
		INNER JOIN (SELECT c.object_name AS EventName,c.object_package_guid AS PkgGuid, v.map_value AS Channel
					FROM sys.dm_xe_object_columns c
						INNER JOIN sys.dm_xe_map_values v
							ON c.type_name = v.name
								AND c.column_value = CAST(v.map_key AS NVARCHAR)
					WHERE c.name = 'channel') ch
			ON ch.EventName = oc.object_name
			AND ch.PkgGuid = oc.object_package_guid
	WHERE mv.map_value = @Keyword
	ORDER BY oc.object_name;

This query will yield results similar to the following.

If you look closely at the script, I included a note about some additional interesting events that are related to both categories “server” and “memory.”

After all of the digging and researching, now it’s time to pull it together and create a session that may possibly help to identify various memory issues as they arise or to at least help confirm your sneaking suspicion that a memory issue is already present.

USE master;
GO
-- Create the Event Session
IF EXISTS ( SELECT *
				FROM sys.server_event_sessions
				WHERE name = 'BuffMemPressure' )
	DROP EVENT SESSION BuffMemPressure
    ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE';
GO


CREATE EVENT SESSION [BuffMemPressure] ON SERVER
ADD EVENT sqlos.large_cache_memory_pressure (
	ACTION ( sqlserver.database_id, 
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname)
		),
ADD EVENT sqlserver.buffer_manager_page_life_expectancy (
	ACTION ( sqlserver.database_id, 
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname)
		),
ADD EVENT sqlserver.buffer_node_page_life_expectancy (
	ACTION ( sqlserver.database_id, 
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname)
		),
ADD EVENT sqlserver.buffer_pool_eviction_thresholds_recalculated (
	ACTION ( sqlserver.database_id, 
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname)
		),
ADD EVENT sqlserver.server_memory_change (
	ACTION ( sqlserver.database_id, 
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname)
		)
ADD TARGET package0.event_file (  SET filename = N'C:\Database\XE\BuffMemPressure.xel'
									, max_rollover_files = ( 5 ) )
WITH (  MAX_MEMORY = 4096 KB
		, EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
		, MAX_DISPATCH_LATENCY = 5 SECONDS
		, MAX_EVENT_SIZE = 0 KB
		, MEMORY_PARTITION_MODE = NONE
		, TRACK_CAUSALITY = ON
		, STARTUP_STATE = ON );
GO

ALTER EVENT SESSION BuffMemPressure ON SERVER
STATE = START;

When running this session for a while, you will receive a flood of events as they continue to trigger and record data to your trace file. You will want to keep a steady eye on the trace files and possibly only run the session for short periods.

Here is an example of my session with events grouped by event name. Notice anything of interest between the groups?

If the data in the session does not seem to be helpful enough, I recommend looking at adding the additional events I noted previously.

Here is another view on a system that has been monitoring these events for a while longer and does experience memory pressure.

Here we can see some of the direct results of index operations on memory as well as the effects on memory for some really bad code. Really cool is that we can easily find what query(ies) may be causing the memory pressure issues and then directly tune the offending query(ies).

The Wrap

Diving in to the internals of SQL Server can be useful in troubleshooting memory issues. Extended Events provides a means to look at many memory related events that can be integral to solving or understanding some of your memory issues. Using Extended Events to dive into the memory related events is a powerful tool to add to the memory troubleshooting toolbelt.

Try it out on one or more of your servers and let me know how it goes.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This has been the ninth article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

This is only one method, there are other methods. My second option is usually to drop into the DMVs – but others exist beyond that. Tara Kizer jumps into some of those other methods here.

Easy Stuff First

Before diving into XE, first it makes sense to get some more data on what the possible connection settings include. We can query SQL Server for most of the applicable information. For the extended details we have to look it up online.

SELECT sv.name AS ConfigName, sv.number AS ConfigValue
	FROM master.dbo.spt_values sv
	WHERE sv.type = 'sop';

Inside SQL Server, we have been given the information for what the values are and what the setting name happens to be. Querying the spt_values table for the group of values of type “sop” (think set options) we get the results we need. That will yield results similar to this.

If I take that a little further, I can modify the query to figure out what configurations are enabled for my current session (in SSMS).

DECLARE @Options BIGINT = @@OPTIONS;

SELECT @Options AS OptionsValue

SELECT sv.name AS ConfigName, sv.number AS ConfigValue
		,CASE WHEN sv.number & @Options > 0
			THEN 1
			ELSE 0
			END AS EnabledForCurrentSession
	FROM master.dbo.spt_values sv
	WHERE sv.type = 'sop';

For me, currently, this yields the following.

Everything marked with a “1” is enabled and the rest are disabled. Ok, easy enough. Now that we can figure out SSMS values and we have an idea of what they mean, it is time to trap the settings from the application. We will be doing that via XE.

App Settings

In order to find the application settings, we need to capture a specific data point called “collect_options_text”. To find which events have this type of data, we can query the XE infrastructure.

SELECT xo.name, xo.object_type, xo.description--, xoc.name AS ColumnName
		, xoc.column_type, xoc.capabilities_desc, xoc.description AS ColumnDescription
	FROM sys.dm_xe_objects xo
		INNER JOIN sys.dm_xe_object_columns xoc
			ON xo.name = xoc.object_name
			AND xo.package_guid = xoc.object_package_guid
	WHERE xoc.name = 'collect_options_text';

Running the preceding query finds two events – login and existing_connection. Both indicate that the “collect_options_text” is a flag that is disabled by default. When enabled it will collect the options_text for each session (new or existing depending on your connections).

If I delve further into the “login” event, I can see some nice data points for troubleshooting and learn more about what the event does.

DECLARE @EventName VARCHAR(64) = 'login' --'existing_connection
	,@ReadFlag VARCHAR(64) = 'readonly' --readonly' --ALL if all columntypes are desired
 
SELECT oc.OBJECT_NAME AS EventName
		,oc.name AS column_name, oc.type_name
		,oc.column_type AS column_type
		,oc.column_value AS column_value
		,oc.description AS column_description
		,ca.map_value AS SearchKeyword
	FROM sys.dm_xe_object_columns oc
		OUTER APPLY (SELECT TOP 1 mv.map_value
						FROM sys.dm_xe_object_columns occ
						INNER JOIN sys.dm_xe_map_values mv
							ON occ.type_name = mv.name
							AND occ.column_value = mv.map_key
						WHERE occ.name = 'KEYWORD'
							AND occ.object_name = oc.object_name) ca
	WHERE oc.object_name = @EventName
		AND oc.column_type <> @ReadFlag;

Which yields this…

Everything in the orange circles is useful in various troubleshooting scenarios. Just a little side tidbit to keep in your reserves. The blue box is highlighting the options and options_text data points. The options_text becomes enabled when we flip the “collect_options_text” flag to on.

Another interesting note is the “SearchKeyword”. This is a category of sorts (it is a category when looking at it in the GUI). This can tell me all of the events that also might be related to the login event. Looking deeper at that, I can see the following.

/* Keyword search */
DECLARE @Keyword VARCHAR(64) = 'session'
 
SELECT oc.OBJECT_NAME AS EventName
		,oc.name AS column_name, oc.type_name
		,oc.column_type AS column_type
		,oc.column_value AS column_value
		,oc.description AS column_description
		,REVERSE(LEFT(REVERSE(olm.name),CHARINDEX('\',REVERSE(olm.name))-1)) AS DLLName
		,olm.file_version
		,xp.name AS PackageName
		,mv.map_value AS SearchKeyword
		,ch.Channel
	FROM sys.dm_xe_object_columns oc
		INNER JOIN sys.dm_xe_map_values mv
			ON oc.type_name = mv.name
			AND oc.column_value = mv.map_key
			AND oc.object_package_guid = mv.object_package_guid
			AND oc.name = 'KEYWORD'
		INNER JOIN sys.dm_xe_packages xp
			ON oc.object_package_guid = xp.guid
		INNER JOIN sys.dm_os_loaded_modules olm
			ON xp.module_address = olm.base_address
		INNER JOIN (SELECT c.object_name AS EventName,c.object_package_guid AS PkgGuid, v.map_value AS Channel
					FROM sys.dm_xe_object_columns c
						INNER JOIN sys.dm_xe_map_values v
							ON c.type_name = v.name
								AND c.column_value = CAST(v.map_key AS NVARCHAR)
					WHERE c.name = 'channel') ch
			ON ch.EventName = oc.object_name
			AND ch.PkgGuid = oc.object_package_guid
	WHERE mv.map_value = @Keyword
	ORDER BY oc.object_name;

That is another juicy tidbit to keep in your back pocket as an extra tool for future use! Seventeen events are in the “session” category and could be related, but we will not use them for this particular event session.

The Juicy Center

Having covered some of the path to getting to the events that matter and what data is available in the events, we are now ready to put a session together.

USE master;
GO
-- Create the Event Session
IF EXISTS ( SELECT *
				FROM sys.server_event_sessions
				WHERE name = 'AppSessionOptions' )
	DROP EVENT SESSION AppSessionOptions 
    ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE';
GO

CREATE EVENT SESSION AppSessionOptions ON SERVER
ADD EVENT sqlserver.login ( SET collect_database_name = ( 1 ), collect_options_text = (1)
	ACTION ( sqlserver.sql_text,sqlserver.nt_username,sqlserver.server_principal_name
	,sqlserver.client_hostname,
	package0.collect_system_time,package0.event_sequence,
	sqlserver.database_id, sqlserver.database_name,
	sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id,
	sqlserver.context_info, sqlserver.client_connection_id ) 
	),
ADD EVENT sqlserver.existing_connection ( SET collect_database_name = ( 1 ), collect_options_text = (1)
	ACTION ( sqlserver.sql_text,sqlserver.nt_username,sqlserver.server_principal_name
	,sqlserver.client_hostname,
	package0.collect_system_time,package0.event_sequence,
	sqlserver.database_id, sqlserver.database_name,
	sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id,
	sqlserver.context_info, sqlserver.client_connection_id ) 
	)
ADD TARGET package0.event_file ( SET filename = N'C:\Database\XE\AppSessionOptions.xel' 
									, max_file_size = ( 5120 )
									, max_rollover_files = ( 4 ) )
WITH (STARTUP_STATE = OFF
	,TRACK_CAUSALITY = ON);

/* start the session */
ALTER EVENT SESSION AppSessionOptions 
ON SERVER 
STATE = START;
GO

After creating and starting the XE session, all that is needed is to wait for a login event to occur from the application. Once it does, then check the trace file and evaluate the data. As I look at the data from the application and look specifically at the options_text data, I will see something like the following.

I circled an interesting difference that pops up between the XE session and the @@Options server variable. A login captured by XE will also show the language and date settings for the connection.

The Wrap

Creating a session to capture the settings being used by an application is particularly easy. Being able to trap the relevant data and troubleshoot performance issues is a tool necessary (and yes easy to do) to be able to quickly become a rock-star DBA. I showed how to search for the necessary events (quickly) as well as how to figure out relationships between events in a particular category.

Try it out on one or more of your servers and let me know how it goes.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This has been the eighth article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

Not only may it be very commonplace to experience this due to the nature of the job, but it can also get to feel that way when a new product is released. Suddenly, that icy island is further north into the Arctic Ocean and you are struggling to gasp for a bit of air in the frigid cold.

I am certain this may be some of what you feel as you try to take a look at Extended Events. It is a fancy, new technology and there is not a ton of time to get up to speed while trying to warm yourself against the development chill. As you stew about the situation, you begin to ponder, “How am I even supposed to know what an XE is, let alone what XEs are running on my server?” Never mind the confusion on what an XE is and that it is actually running or deployed sessions on the server. Don’t worry about the terminology just yet. What can we do to figure out what might be out there?

Well, that is where the trusty Google has come to help you. Thanks to a better than adequate ability to search the Internet, you landed on this page that has a script to help you explore what might be running as far as Extended Events on your server. You may be surprised to learn that you have more Extended Events Sessions running than you fathomed. Yes, SQL Server comes with some default sessions. Yes, some of these default sessions are more public than others. And yes, some of these

Come on in out of the cold. Get off that arctic island and warm up with a little XE fun. This script is quite simple but will show you what you may have out there on your servers. Once you see what is there, hopefully your curiosity will get a little piqued and you will want to learn a little more (there is more of that here on this site too).

sessions are “hidden” from you. That is, until now.

And now, for that script:

DECLARE @SessionName VARCHAR(128) = NULL --'sp_server_diagnostics session' --NULL for all
;

SELECT ISNULL(ses.name,xse.name) AS SessionName
		, CASE
			WHEN ISNULL(ses.name,'') = ''
			THEN 'Private'
			ELSE 'Public'
			END AS SessionVisibility
		, CASE
			WHEN ISNULL(xse.name,'') = ''
			THEN 'NO'
			ELSE 'YES'
			END AS SessionRunning
		, CASE
			WHEN ISNULL(xse.name,'') = ''
				AND ISNULL(ses.name,'') = ''
			THEN 'NO'
			ELSE 'YES'
			END AS IsDeployed
	FROM sys.server_event_sessions ses
	FULL OUTER JOIN sys.dm_xe_sessions xse
		ON xse.name =ses.name
	WHERE COALESCE(@SessionName, ses.name, xse.name) = ISNULL(ses.name, xse.name)
	ORDER BY ses.event_session_id;

GO

Quick and simple, and full of that warming XE sensation. This script just shows you what is a private session versus public. What is a private session, you ask? Well, that happens to be all of those database audits you have deployed to your server. It also includes those little things like the “sp_server_diagnostics session” session (yes it sounds redundant but that is because session is also part of the name for that session).

This script will also show you what is deployed to the server versus what is running on the server (not all sessions have to be running). And to top it all off, I set it up so you can query for a specific session or for all sessions.

Try it out on one or more of your servers and let me know how it goes.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This has been the seventh article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

Yesterday i shared the first article in a three part series about the SQL Black Box Recorder, that you can read here, with a promise of at least two more posts. The first article in the series laid the groundwork about the black box recorder in SQL Server. Furthermore, there was a brief introduction into the three components of the black box recorder. The three components as laid out in that article are:

• Default Trace
• system_health Extended Event Session
• sp_server_diagnostics procedure

The first article went into deeper detail about the first leg of this three-legged black box recorder – the default trace. In this article, the focus will be directed to the next leg of the black box recorder – or the system_health Extended Event Session. If by some measure you are in the dark about what Extended Events is, then I recommend you read my personal blog series on the subject. There are numerous articles stepping through the subject in easy-to-digest fashion. You can find many of the articles in the following table of contents – here.

The focus of this article will take on the topic of the third piece of the black box recorder, sp_server_diagnostics. I will show how this procedure is critical in gathering pertinent information that is crucial to gathering an overall picture of the system—from a default black box recorder perspective.

SQL Black Box Recorder

What is the sp_server_diagnostics process?

Beyond being a component of the black box for SQL Server, what exactly is this diagnostics process? The sp_server_diagnostics is much as the name implies—it is a “diagnostics” service that attempts to gather information about various events that may affect the overall health of the SQL Server instance.

The diagnostics process will trap various server related health (diagnostics) information related to the SQL Server instance in an effort to try and detect potential failures and errors. This diagnostics session/process traps information for five different categories by default. There is a sixth category of information for those special servers that happen to be running an Availability Group.

The Six

• system: overall health of the server including information about spinlocks, severe processing conditions, non-yielding tasks, page faults, and CPU usage.
• resource: health of resources, specifically including physical and virtual memory, buffer pools, pages, cache and other memory objects
• query_processing: captures query processing data about worker threads, tasks, wait types, CPU intensive sessions, and blocking tasks.
• io_subsystem: health related data to IO.
• events: errors and events of interest recorded by the server, including details about ring buffer exceptions, ring buffer events about memory broker, out of memory, scheduler monitor, buffer pool, spinlocks, security, and connectivity.
• AvailabilityGroup: basic information about the AvailabilityGroup if there is an AvailabilityGroup enabled on the server.

This is fantastic introductory information. As luck would have it, you can also read more about it from my blog – here.

How to use this process?

What good is any level of information if we don’t quite know how to use or consume that data? Do we consume it via trace? Do we execute the stored procedure? Do we capture the information from some other means? The answer is yes! Wait, yes to what? Yes to all of it! Let’s explore how to consume this data.

Option 1

The first step to learning how to consume this data is to start at probably the most basic of levels. As you have already noted, this is a stored procedure. It was probably the sp_ in the name that gave it away, right? Since it comes in the form of a stored procedure, it makes it pretty easy to figure out the first means to get the data from it—just execute it. Actually, let me clarify that just a bit. sp_server_diagnostics is a CLR stored procedure. Nonetheless, all that needs be done is execute it at this level to get to the data.

EXECUTE sp_server_diagnostics;

Look at that! Pretty slick! I execute the stored procedure and I can see a point in time result of the “health check” from this component of the black box recorder. Notice here that I only have five of the six categories that I had mentioned. This is due to the lack of a configured availability group (AlwaysOn) from this particular instance. Another point of interest is that the data column is not in a very friendly format for DBA consumption. This is flat text that just goes on and on. It is painful to consume in this manner.

Tidy that Code

CREATE TABLE #ServerDiagnosticsResult
	(
		create_time DATETIME
		, component_type sysname
		, component_name sysname
		, state INT
		, state_desc sysname
		, data XML
	);  
INSERT INTO #ServerDiagnosticsResult
		EXECUTE sp_server_diagnostics;  

SELECT *
	FROM #ServerDiagnosticsResult;

DROP TABLE #ServerDiagnosticsResult;

Looking at the code, you can see quickly the difference. I have changed the data column to be XML-based to better match the data being output from the stored procedure. Let’s see that in action:

That is much better. Now I can simply click one of the data results and see a nicely formatted easy to read XML output of that component. Here is an example from the query_processing component:

Option 2

Cool! I have just shown one means of exploring this facet of the black box recorder. Now, let’s take a look at other means to get at this particular data, beyond just executing the stored procedure. Did you know that this procedure is very closely related to Extended Events? (I am sure that you picked up on that nugget due to reading the previous article in the series or possibly the other article I mentioned here). With such a close relationship to Extended Events, it makes sense that sp_server_diagnostics also has a couple of events and is consumed by multiple sessions.

SQL Server comes with various default Extended Event sessions, of those sp_server_diagnostics is consumed by the system_health session as well as the hidden session for SQLDiag that is used by Windows Clustering and Availability Groups to determine server health. You already heard about the system_health (from the previous article), but few have heard about the hidden session. I will take a look at the events as well as the sessions that consume the sp_server_diagnostics data.

Events

First, I will share the details for each of the events related to the sp_server_diagnostics process.

As I have shown here, you can see that these events are a part of the debug channel. In the next couple of images, I want to show some of the payload attached to these events.

These events show what I have shown previously with regards to output of the stored procedure. There is a consistency in the output of the diagnostics process. What I have yet to show in this data is one more piece of information that could be helpful down the road and that pertains specifically to the extended events data. That piece of information I need to share is the keyword associated with each of these events. When looking for related events, I need you to understand that the sp_server_diagnostics events are a part of the scheduling classification. There, I have said it. I just gave you a big clue to help you in your troubleshooting endeavors and the use of the diagnostics processes/events.

Which Sessions then?

Now that I have shown which events are related to diagnostics, I can finally show which sessions are dependent upon the diagnostics data. To show that kind of information, I have this next query to help:

/* session that is running */
SELECT xs.name AS SessionName
		, xse.event_name
	FROM sys.dm_xe_session_events xse
		INNER JOIN sys.dm_xe_sessions xs
			ON xse.event_session_address = xs.address
	WHERE xse.event_name IN ( 'sp_server_diagnostics_result_set',
								'sp_server_diagnostics_component_result' );

/* sessions on the server but may be stopped */
SELECT ses.name AS SessionName
		, sese.name AS event_name
	FROM sys.server_event_session_events sese
		INNER JOIN sys.server_event_sessions ses
			ON ses.event_session_id = sese.event_session_id
	WHERE sese.name IN ( 'sp_server_diagnostics_result_set',
							'sp_server_diagnostics_component_result' );

When I execute the statements in the preceding query, I will receive a single result—system_health. That said, do not believe everything I just showed you with that query. SQL Server does a decent job at hiding some of the details. When I run the next query, I will see a little bit more information.

SELECT *
	FROM sys.dm_xe_sessions;

When I run the preceding query, I will see the following two sessions (given that I have not change the default settings):

Take note of the names of those two sessions. I showed that the system_health session is reliant upon the diagnostics events. Now, I am showing that there is another event session called “sp_server_diagnostics session“. Wait, there is more! SQL Server continues to hide stuff from us.

Hidden Sessions

When you have AlwaysOn configured, there is an additional hidden session that is not visible within the sessions lists inside of SQL Server. To view this hidden session, you must follow the directions shown in this article on msdn. Once you have navigated to the folder containing the hidden XEL files, you can open them in Management Studio and view the data. It is while viewing the data that you will be able to see the relationship between sp_server_diagnostics and this newly discovered trace file.

The payload from this image demonstrates the relationship between sp_server_diagnostics and the hidden trace file. I challenge you to take some time to explore this hidden trace file on one of your servers where AlwaysOn is configured. I know you will be thankful for the investment at some point down the road.

Conclusion

I have just shown a ton of info about the third component of the black box recorder—sp_server_diagnostics. This procedure may be lesser known than the first two components, but it is highly integral into the overall monitoring and health of your SQL Server instance. In this article, I have not only shown how to use sp_server_diagnostics (albeit quickly), but I have also shown that the diagnostics are heavily relied upon by other components of the black box recorder. In addition, I have shown how this component is integrated into other hidden traces that are essential in the monitoring and troubleshooting of your instance.

The black box recorder is real. You need to invest time in getting to know the black box recorder components and how they can be of help to you in becoming a superhero DBA.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This has been the sixth article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

Yesterday i shared the first article in a three part series about the SQL Black Box Recorder, that you can read here, with a promise of at least two more posts. The first article in the series laid the groundwork about the black box recorder in SQL Server. Furthermore, there was a brief introduction into the three components of the black box recorder. The three components as laid out in that article are:

• Default Trace
• system_health Extended Event Session
• sp_server_diagnostics procedure

The first article went into deeper detail about the first leg of this three-legged black box recorder – the default trace. In this article, the focus will be directed to the next leg of the black box recorder – or the system_health Extended Event Session. If by some measure you are in the dark about what Extended Events is, then I recommend you read my personal blog series on the subject. There are numerous articles stepping through the subject in easy-to-digest fashion. You can find many of the articles in the following table of contents – here.

The focus of this article will not be to introduce the topic of Extended Events. Rather, it will be to introduce the system_health session and dive into it a fair bit.

SQL Black Box Recorder

What is the system_health session?

Beyond being a component of the black box for SQL Server, what exactly is this event session? The system_health is much as the name implies – it is a “trace” that attempts to gather information about various events that may affect the overall health of the SQL Server instance.

The event session will trap various events related to deadlocks, waits, clr, memory, schedulers, and reported errors. To get a better grasp of this, let’s take a look at the event session makeup based on the available metadata in the dmvs and catalog views.

DECLARE @SessionName VARCHAR(256) = 'system_health';

/* note: this only works if the session is running and the source is "server" not internal or something else.
*/

SELECT xs.name AS SessionName
		, xs.session_source
		, xse.event_name
		, xo.description AS EventDesc
		, xsea.action_name
		, xoa.description AS ActionDesc
		, xse.event_predicate
		, sese.predicate, sese.predicate_xml
	FROM sys.dm_xe_sessions xs
		INNER JOIN sys.dm_xe_session_events xse
			ON xs.address = xse.event_session_address
		LEFT OUTER JOIN sys.dm_xe_session_event_actions xsea
			ON xse.event_session_address = xsea.event_session_address
				AND xse.event_name = xsea.event_name
		INNER JOIN sys.dm_xe_objects xo
			ON xse.event_name = xo.name
				AND xse.event_package_guid = xo.package_guid
				AND xo.object_type = 'event'
		LEFT OUTER JOIN sys.dm_xe_objects xoa
			ON xsea.action_name = xoa.name
				AND xsea.action_package_guid = xoa.package_guid
				AND xoa.object_type = 'action'
		INNER JOIN sys.server_event_sessions ses
			ON xs.name = ses.name
		LEFT OUTER JOIN sys.server_event_session_events sese
			ON ses.event_session_id = sese.event_session_id
			AND xse.event_name = sese.name
	WHERE xs.name = @SessionName
	ORDER BY xs.name,xse.event_name, xsea.action_name;

In addition to the types of events I mentioned, there are also a few more interesting things to discover from the results of this particular query. For instance, the actions that are being employed help to gather even more information such as the callstack, tsql_callstack, database id, and sql text of the query that was executing at the moment of the event that is trapped.

One very peculiar event that is trapped is the sp_server_diagnostics_component_result event. Remember from earlier in this article that I noted the third leg of the black box record happens to be a very similarly named session – “sp_server_diagnostics procedure”. When I cover this third leg, it will make a little more sense. Understand, for now, that your suspicion of it being a bit more integral to the black box (than merely 1/3 of the overall black box) is confirmed.

This is a good set of information that can be trapped. Furthermore, this is a good set of information which can give a fairly good diagnosis of several different potential problems within your SQL Server Instance! (Yes it may seem repetitive, but I wanted to underscore the importance of the data that can be trapped.) That said, compare the types of events to those of the default trace. Take note of how many events actually overlap between the two components of the black box recorder.

For ease of comparison, let’s simplify that previous query to get a set of events to read without the duplication due to the actions on each event.

DECLARE @SessionName VARCHAR(256) = 'system_health';

SELECT xs.name AS SessionName, xse.event_name
	FROM sys.dm_xe_sessions xs
		INNER JOIN sys.dm_xe_session_events xse
			ON xs.address = xse.event_session_address
	WHERE xs.name = @SessionName
	ORDER BY xse.event_name;

And since I am feeling particularly generous, I have lumped the events from the two traces together in the following query:

DECLARE @SessionName VARCHAR(256) = 'system_health';

SELECT xs.name AS SessionName
		, xse.event_name AS EventName
		, mv.map_value AS CategoryName
	FROM sys.dm_xe_sessions xs
		INNER JOIN sys.dm_xe_session_events xse
			ON xs.address = xse.event_session_address
		INNER JOIN sys.dm_xe_objects xo
			ON xse.event_package_guid = xo.package_guid
			AND xse.event_name = xo.name
			AND xo.object_type = 'event'
		INNER JOIN sys.dm_xe_object_columns oc
			ON oc.object_name = xo.name
			AND oc.object_package_guid = xo.package_guid
			AND oc.name = 'KEYWORD'
		INNER JOIN sys.dm_xe_map_values mv
			ON oc.type_name = mv.name
			AND oc.column_value = mv.map_key
			AND oc.object_package_guid = mv.object_package_guid
	WHERE xs.name = @SessionName
UNION ALL
/* events */
SELECT 'Default Trace' AS SessionName
		, te.name AS EventName
		, tca.name AS CategoryName
	FROM sys.traces t
		CROSS APPLY ( SELECT DISTINCT gei.eventid
							FROM sys.fn_trace_geteventinfo(t.id) gei
					) ca
		INNER JOIN sys.trace_events te
			ON te.trace_event_id = ca.eventid
		INNER JOIN sys.trace_categories tca
			ON te.category_id = tca.category_id
	WHERE t.is_default = 1
	ORDER BY EventName;

That should help to compare the 17 events from the system_health session and the 34 from the default trace with a little more ease and speed. The events from the two sessions are rather complimentary to each as they help to construct the SQL Server Black Box recorder.

How to use this session?

With the events trapped by this session, there are a few ways to use the trapped data that should stand out pretty easily. To me, the quick application of the data from this session can come from the review of deadlocks, reported errors, or even waits analysis. Using the deadlock as probably the most common problem people will want to troubleshoot, this information is suddenly much more valuable and easy to access (no more need for that trace flag). Since the system_health is now trapping the deadlock by default, one could quickly access the deadlock graph from the event data and proceed to troubleshooting. More information on getting that deadlock graph can be found here and here.

Learn to use the data made available through this member of the black box recorder. Becoming familiar with this tool (and additionally with Extended Events) will make you a rock-star in the office. The information needed to troubleshoot most problems is right at your fingertips through the use of the different pieces of the black box recorder. If it is not available there, then more than likely it can be made available through a bit of tinkering with Extended Events. Get to know the tools and be that rock-star DBA.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This has been the fifth article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

Across many professions and industries there is often a need to have some sort of device that “audits” everything that happens with a device or process. We are probably all quite familiar with the infamous black box recorders used by the passenger airline and train industries. It is also quite possibly fairly common knowledge that many pace makers perform the same sort of functionality – on a smaller scale. Various metrics and data points are registered and recorded somewhere. Whether to a local storage device or whether the device phones home to some other remote location, it is recorded.

We often pontificate about the what-ifs for a black box recorder within SQL Server, right? We wish and dream and go about creating our own little recorder to suit the requirements we deem necessary for successful monitoring of the server for the just-in-case scenario. Well, the truth of the matter is that we really don’t need to go to such great lengths to create a “black box recorder” because Microsoft has already done all of that for us.

Wait, what? Yes, that’s right! Truth be told this shouldn’t be much news for most Senior Level data professionals working with SQL Server for the past few years. But if you are new to the product, this might be good news for you. Then again, it might just be bad news depending on your take.

Very much like what you may find with the airline industry, the black box recorder in SQL Server is not just a single method (device) implemented to capture all of the desired data points. On a passenger jet, you may find that there are three or more such devices that contribute to the capture and recording of the in-flight data. In SQL Server, there are three major processes that help capture our in-flight data. Over the next few articles I will discuss each of these processes. These processes include:

• Default Trace
• system_health Extended Event Session
• sp_server_diagnostics procedure

This multi-tiered approach does provide a sort of fail-safe system. If one should be disabled, there still remain up to two more processes that may be running. That said, each of these can be disabled but it does take a bit of an overt effort. And since it does require somebody to put out effort to try and disable each of the black box components, you could potentially capture the culprit via audits from the other components or via an explicit audit that you create for these types of purposes.

Since this will be something to discuss over a few articles, I will break out each process into an individual article. For today, I will discuss the default trace.

Default Trace

The default trace by itself is something that can be turned off via configuration option. There may be good reason to disable the default trace. Before disabling the default trace, please consider the following that can be captured via the default trace. I will use a query to demonstrate the events and categories that are configured for capture in the default trace.

/* events */
SELECT te.name AS EventName
		,tca.name AS CategoryName
		, CASE tca.type WHEN '0' THEN 'Normal'
				WHEN '1' THEN 'Connection'
				WHEN '2' THEN 'ERROR' END AS CategoryType
		, t.path AS TracePath
		, oa.logical_operator,oa.comparison_operator, oa.value AS FilteredValue
	FROM sys.traces t
		CROSS APPLY (SELECT DISTINCT gei.eventid FROM sys.fn_trace_geteventinfo(t.id) gei) ca
		INNER JOIN sys.trace_events te
			ON te.trace_event_id = ca.eventid
		INNER JOIN sys.trace_categories tca
			ON te.category_id = tca.category_id
		OUTER APPLY (SELECT gfi.columnid,gfi.logical_operator,gfi.comparison_operator,gfi.value FROM sys.fn_trace_getfilterinfo(t.id) gfi) oa
	WHERE t.is_default = 1

In this query, I have requested a few more data points than necessary to illustrate the point. That is to help illustrate an additional point that the default trace isn’t filtering out any data for these events. If the event fires (in this trace), it is recorded. Let’s divert right back to the events and categories for now. The results of that query will produce the following sample list of events for me on SQL Server 2014:

That is quite a range of events covered by this particular recording device. From changes to objects down to security related events and even errors and warnings. If somebody drops an object, the default trace can catch it. If one of the various DBCC statements is executed, it will be trapped in this trace.

There is one thing that is captured by this trace that is not overly obvious. In fact, it is rather annoying in how it is recorded in my opinion. Server configuration settings such as “cost threshold of parallelism” or the lesser utilized “user options” are not registered as an “Object:Alter” event but rather as an ErrorLog event. I wouldn’t call this type of “change” an error and don’t necessarily like seeing these registered with other legitimate errors. That said, it would be nice to see these logged differently (they are currently logged the same way with Extended Events). So, if somebody is dorking around with server configurations, it becomes a bit more convoluted to figure it out, but we can get there. Let’s see how that works:

-- server configs Audited via Def Trace
SELECT 	T.StartTime
		, T.SPID
		, T.LoginName
		, T.HostName
		, T.ApplicationName
		, T.DatabaseName
		--, ObjectName,sv.number AS ObjTypeVal, sv.name [ObjectType]
		--, T.TextData
		, ConfigOption = SUBSTRING(T.TextData,CHARINDEX('''',T.TextData)+1,CHARINDEX(' changed from ',T.TextData)-CHARINDEX('''',T.TextData)-2)
		, PrevValue = SUBSTRING(T.TextData,CHARINDEX('from ',T.TextData)+5,CHARINDEX('to ',T.TextData)-CHARINDEX('from ',T.TextData)-5)
		, NewValue = SUBSTRING(T.TextData,CHARINDEX('to ',T.TextData)+3,CHARINDEX('. Run',T.TextData)-CHARINDEX('to ',T.TextData)-3)
		, EventName = te.name
		, T.EventClass
	FROM sys.fn_trace_gettable(CONVERT(VARCHAR(150), 
			( SELECT REVERSE(SUBSTRING(REVERSE(path),
					CHARINDEX('\',REVERSE(path)),256)) + 'log.trc'
				FROM    sys.traces
				WHERE   is_default = 1)), DEFAULT) AS T  
		INNER JOIN sys.trace_events AS te
			ON T.EventClass = te.trace_event_id
	WHERE T.EventClass = 22
		AND T.TextData LIKE '%config%'
	ORDER BY T.StartTime DESC;

Looking at my system, I get the following sample results:

You can see there are plenty of settings that I have been playing with captured by the default trace. If you happen to have a junior DBA or maybe a cowboy DBA that loves to shoot from the hip and make changes, this will help you find all the necessary details to bring to that DBA for discussion.

The default trace comprises one third of the black box recorder. I have shown a quick use for the default trace and have also shown some of the various events that are captured from this trace. I recommend getting to know your default trace just a little bit better. You never know when you may need to resort to the flight recorder data held in the default trace. Being familiar with the default trace before you need to use it will help improve your comfort level when under the stress of trying to figure out what happened just before the server went belly up!

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This has been the fourth article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

I have been doing a lot of posts of late about extended events but have failed to share some of the essential building blocks. I am not looking to do a series of definitions, but just share a couple of quick tips and examples here and there that will make the use of extended events a little easier to undertake.

The first one to tackle is the use of the file target. Using a file target is not difficult from a creation point of view (event session creation). But, some basics are critical or you could end up with fewer follicles.

Let’s take a quick peek at some of these tips.

File System Prep

This first little tip comes from a painful experience. It is common sense to only try and create files in a directory that exists, but sometimes that directory has to be different on different systems. Then comes a little copy and paste of the last code used that worked. You think you are golden but forgot that one little tweak for the directory to be used. Oops.

Take this example session. When you run this code, you will not see an error (don’t change anything).

-- Create the Event Session
IF EXISTS(SELECT * 
          FROM sys.server_event_sessions 
          WHERE name='FileTargetDemo')
    DROP EVENT SESSION FileTargetDemo 
    ON SERVER;
GO
CREATE EVENT SESSION FileTargetDemo
ON SERVER
--+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
/* Pick an event for this demo */
--ADD EVENT sqlserver.buffer_node_page_life_expectancy()
--+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
--+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
ADD EVENT sqlserver.buffer_node_page_life_expectancy()
/* Note two targets exist here */
ADD TARGET package0.ring_buffer,
ADD TARGET  package0.asynchronous_file_target(
     SET filename='C:\Database\XEs\FileTargetDemo.xel',max_file_size = 5,max_rollover_files = 4)
WITH (MAX_DISPATCH_LATENCY = 5SECONDS)
GO

However, if you try to start the session, you will receive a pretty little error. Here is the code to try and start the session, along with it’s associated error.

-- Start the Event Session
ALTER EVENT SESSION FileTargetDemo 
ON SERVER 
STATE = START;
GO

Msg 25602, Level 17, State 22, Line 25
The target, “5B2DA06D-898A-43C8-9309-39BBBE93EBBD.package0.event_file”, encountered a configuration error during initialization. Object cannot be added to the event session. The operating system returned error 3: ‘The system cannot find the path specified.
‘ while creating the file ‘C:\Database\XEs\FileTargetDemo_0_130670484928250000.xel’.

If you double check in the SSMS GUI, you will see that the session is there. It is just stopped. You can also check that the session exists if you recall my query from a previous post where I show the status of the session.

DECLARE @SessionName VARCHAR(64) = 'FileTargetDemo';
 
SELECT sn.SessionName
	, CASE WHEN ISNULL(es.name,'No') = 'No'
		THEN 'NO'
		ELSE 'YES'
		END AS XESessionExists
	, CASE WHEN ISNULL(xe.name,'No') = 'No'
		THEN 'NO'
		ELSE 'YES'
		END AS XESessionRunning
	FROM (SELECT @SessionName AS SessionName) sn
		LEFT OUTER JOIN sys.server_event_sessions es
			ON sn.SessionName = es.name
		LEFT OUTER JOIN sys.dm_xe_sessions xe
			ON es.name = xe.name
	;

If you take it a step further and try to read the log file that should be created, you will be greeted once more with a reminder that there is a problem. Try this (sans change again).

/* read the session data */
SELECT CAST(event_data AS xml) AS TargetData
            FROM sys.fn_xe_file_target_read_file('C:\Database\XEs\FileTargetDemo*.xel',NULL,NULL, NULL)

And you will receive the following message:

Msg 25718, Level 16, State 3, Line 31
The log file name “C:\Database\XEs\FileTargetDemo*.xel” is invalid. Verify that the file exists and that the SQL Server service account has access to it.

The beauty here is that all is not lost. These are easy problems to get around and resolve. All it takes is to adjust the file path to be the correct path or a path that exists (to which you have permissions).

Read the Log File

If you have been reading this article, then this might seem like a bit of redundancy. That is by design. DBAs love redundancy—well at least when it means there is a means to recovery due to the redundancy.

Unlike reading from the ring_buffer, when you read from the file target, you will use different code. The file target is stored in a binary representation on disk that a function will produce as XML format for human readability. So to read a file target, you would need to do something like the following.

/* read the session data */
SELECT CAST(event_data AS xml) AS TargetData
            FROM sys.fn_xe_file_target_read_file('C:\Database\XEs\FileTargetDemo*.xel',NULL,NULL, NULL)

And you definitely do not want to try the following with a file target. That is unless of course you are looking to go bald at an earlier than expected age in this profession.

/* do not do this */
 SELECT CAST(target_data AS xml) AS TargetData
            FROM sys.dm_xe_sessions AS s
            INNER JOIN sys.dm_xe_session_targets AS t
                ON s.address = t.event_session_address
            WHERE s.name = 'FileTargetDemo'
              AND t.target_name = 'event_file';

If you find yourself doing this version, all you will get for results is a short xml string similar to the following:

<EventFileTarget truncated="0">
  <Buffers logged="1" dropped="0" />
  <File name="C:\Database\XE\FileTargetDemo_0_130670493477970000.xel" />
</EventFileTarget>

This is not what you would be looking for when trying to read the event session data. That said, it at least provides some information that could be of use. You will see that buffers are tracked and the file name is tracked. For this last demo, I corrected the FileTargetDemo session to use a directory on the file system that existed and to which I had the appropriate level of permissions.

I just covered two quick tips to help make life a little bit easier when dealing with file targets and extended event sessions. I will try to continue little tips like this over the coming months. I hope you have found this useful and that you will indeed be able to put it to use.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in an article demonstrating the power of XE over Profiler? Check this one out!

This has been the third article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

I just shared an article describing how to use Extended Events to perform your SQL Server Profiler duties. If not, you can read all about it here.

In that article, I showed one common use-case for SQL Server Profiler and how to achieve the same result via Extended Events. What I didn’t show in that article was how to correlate all of your favorite trace/profiler options into the shiny new XE equivalents. Is there even a way to do that?

As luck would have it, there is a means to correlate trace/profiler events to Extended Event events (yeah that sounds weird).  With the release of SQL Server 2012, Microsoft introduced a couple of catalog views to help with this correlation effort. Those views have a pretty straightforward naming convention. For example, one of the views is named sys.trace_xe_event_map while the other is sys.trace_xe_action_map. For this article, I will be focusing on the former.

Making the Upgrade

When looking to finally make the upgrade away from trace/profiler, a big key is to figure out if your favorite trace events are even an option at the next level. I have talked about finding events in XE in previous articles by trying to search for a specific topic that might apply to your current condition.

But if you are already familiar with specific trace events, you may just want/need to know what the new name is in XE. This is where those catalog views come into play. And in support of that, here is a query that can help in that upgrade effort:

SELECT te.trace_event_id,tc.name AS CategoryName
	, CASE tc.type
		WHEN 0 THEN 'Normal'
		WHEN 1 THEN 'Connection'
		WHEN 2 THEN 'ERROR'
		END AS TraceEventType
	,te.name AS TraceEventName
	, xem.package_name, xem.xe_event_name
FROM sys.trace_xe_event_map xem
	RIGHT OUTER JOIN sys.trace_events te
		ON te.trace_event_id = xem.trace_event_id
	INNER JOIN sys.trace_categories tc
		ON te.category_id = tc.category_id
ORDER BY tc.category_id,te.trace_event_id ASC;

Looking through the results, one would notice that not every trace event maps to an Extended Event event. The events that don’t map, in general, deal with audits (which is actually driven by Extended Events).

This is a really good start to getting you on your way to that much needed upgrade away from profiler/trace. What if there are several traces that are already in use or scripted for various reasons in the environment?

Decision Time

If you happen to have traces that are already deployed in the environment or in script form, it can be a bit of a tedious pain to convert those to Extended Events. Do you manually recreate the traces as XE sessions? Do you abandon the upgrade due to the effort and annoyance it will create? Or do you find some automated means of performing the conversion?

Of those three options, only two are valid options. Those options involve performing the conversion of the traces to XE sessions. There are pros and cons for each. You may opt to take your time and learn more about Extended Events by performing the manual upgrade, or you may choose to save time by using an automated routine.

Should you decide to try the automated routine, there is one already out and available to help you on your way. Jonathan Kehayias wrote the script and you can download it here.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in another article demonstrating the power of XE over Profiler? Check this one out!

This has been the second article in the 2018 “12 Days of Christmas” series. For a full listing of the articles, visit this page.

A common excuse for not delving into extended events seems to be the jump from Profiler to extended events.  There appears to be an inherent fear with how to use extended events.  In addition to that fear, is the fear that extended events does not do what Profiler can do so easily.

Today, I would like to share a short tutorial on how to use extended events as if it were a profiling session.  I am only going to show one of the many possible means to profile via extended events.  Just understand that this means you have flexibility and a way to do more with this simple example than what I am going to share.

Requirement

You need to capture TSQL queries that are occurring against your instance in order to try and determine the code that is being passed to SQL Server from the application.

How-To

Rather than jump to Profiler, you are going to need to jump to a fresh, clean query window in SSMS.  Just like in Profiler, you will want to capture certain “statement” oriented events.  For this session, I want to start with sp_statement_starting and sql_statement_starting.  I am not going to use the _completed forms of those events because I want to capture as much as I can – without capturing too much noise.  Sometimes, there may be a _starting without a coordinated _completed and that could throw a wrench in the works.

With the basic information in hand, we are ready to implement some of the things learned in previous XEvents posts (here and here).  Let’s go ahead and start setting up the session that will capture the information we seek.

USE master;
GO
-- Create the Event Session
IF EXISTS(SELECT * 
          FROM sys.server_event_sessions 
          WHERE name='TraceIncomingQueries')
    DROP EVENT SESSION TraceIncomingQueries 
    ON SERVER;
GO
CREATE EVENT SESSION TraceIncomingQueries
ON SERVER
ADD EVENT sqlserver.sp_statement_starting (
	SET collect_object_name=(1)
	ACTION(sqlserver.database_name,sqlserver.nt_username,sqlserver.session_id,sqlserver.client_hostname,sqlserver.client_app_name)
WHERE sqlserver.client_app_name <> 'Microsoft SQL Server Management Studio - Transact-SQL IntelliSense'
	),
ADD EVENT sqlserver.sql_statement_starting(
    ACTION(sqlserver.database_name,sqlserver.nt_username,sqlserver.session_id,sqlserver.client_hostname,sqlserver.client_app_name)
WHERE sqlserver.database_name='AdventureWorks2014'
	AND sqlserver.client_app_name <> 'Microsoft SQL Server Management Studio - Transact-SQL IntelliSense'
	)
ADD TARGET package0.event_file(SET filename=N'C:\Database\XE\TraceIncomingQueries.xel')

/* start the session */
ALTER EVENT SESSION TraceIncomingQueries 
ON SERVER 
STATE = START;
GO

One caveat to consider is in the sp_statement_starting event, I have specified a specific set option be used.

SET collect_object_name=(1)

This particular option is disabled by default.  If you want to know the object_name of code that is being used (triggers, stored procedures, etc), then this option must be enabled.

I have specified the same actions to be used for each of the events I am trapping, but this is purely up to you.  If you do not wish to capture these actions, then you can remove them or add different actions as necessary.  I have also specified a different predicate for each of the events.  This is something that works so much better in XEvents than Profiler – the filtering that can be applied is much more flexible.

Once the session is started, then it is merely a matter of figuring out how to look at the data.  I generally use TSQL to parse the data from the file, but you could also fiddle with the GUI as well.  You can access that by right-clicking the event session in management studio and then selecting “Watch Live Data”.  If everything is configured properly, then you will start to see data after the first event is triggered.  From within the GUI, you can pick and choose which columns to display in your view.  You can pause the display of the session data, or you can even “erase” the data from the current view (just like in Profiler).  You can even filter, aggregate or group the data in the view from the GUI.

The use of the GUI is up to you.  I still prefer to use the script route.  With that route, here is a sample of what you may need to write in order to display the data from the session data file that has been captured.

use master;
GO

SELECT
event_data.value('(event/@name)[1]', 'varchar(50)') AS event_name,
    event_data.value('(event/@timestamp)[1]', 'varchar(50)') AS [TIMESTAMP],
	event_data.value('(event/action[@name="database_name"]/value)[1]', 'varchar(max)') AS DBName
	,event_data.value('(event/data[@name="statement"]/value)[1]', 'varchar(max)') AS SQLText
	,event_data.value('(event/data[@name="object_name"]/value)[1]', 'varchar(max)') AS ObjName
	,event_data.value('(event/action[@name="session_id"]/value)[1]', 'varchar(max)') AS SessionID
	,event_data.value('(event/action[@name="nt_username"]/value)[1]', 'varchar(max)') AS ExecUser
	,event_data.value('(event/action[@name="client_hostname"]/value)[1]', 'varchar(max)') AS Client_HostName,
	event_data.value('(event/action[@name="client_app_name"]/value)[1]', 'varchar(max)') AS Client_AppName
FROM(
SELECT CONVERT(XML, t2.event_data) AS event_data
 FROM (
  SELECT target_data = convert(XML, target_data)
   FROM sys.dm_xe_session_targets t
    INNER JOIN sys.dm_xe_sessions s 
        ON t.event_session_address = s.address
   WHERE t.target_name = 'event_file'
    AND s.name = 'TraceIncomingQueries') cte1
   CROSS APPLY cte1.target_data.nodes('//EventFileTarget/File') FileEvent(FileTarget)
   CROSS APPLY  sys.fn_xe_file_target_read_file(FileEvent.FileTarget.value('@name', 'varchar(1000)'), NULL, NULL, NULL) t2)
    AS evts(event_data);

It really is that simple.  Now you have a means to run a Profiler-like trace on your server without the impact of Profiler.  XEvents provides the means to run a more evolved Profiler session on your data to capture things like queries from an application.  XEvents does all of this without the severe penalty of Profiler and with many additional bonuses.  Try it out and enjoy!

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in another article demonstrating the power of XE over Profiler? Check this one out!

This is the first article in the 2018 “12 Days of Christmas” series. For the full list of articles, please visit this page.

The Cloud

It has been a minute since I gave much love or attention to Extended Events in Azure SQL DB. Things have changed a touch since then. We will see about some of those changes in a future article. As for this time, let’s focus on how to get a session in Azure SQL DB up and running really quick and easy.

Create a Session

I am going to keep this as easy as possible for creating a session and thus I will demonstrate how to do it from the GUI. And for those more in tune with their scripting side, there is a future article on that as well as a glimpse of a script near the end of this article.

Recall from the previous article that things in Azure SQL DB are different for Extended Events. XE is database scoped rather than server scoped (as is the case with your on-premises servers). Due to this change, finding the GUI for XE is a little different.

In order to find the GUI for XE, you must drill down into the database and then you will see “Extended Events.” (Side note, there is no XE Profiler for Azure SQL DB as of this writing.) If you right-click Sessions, you will get a menu with the option to create a New Session.

After clicking “New Session…” the familiar window for a new session will appear.

Here, you see a session that I started creating already. For this session, I opted to go with the template called “Query Detail Tracking” (more on templates for Azure SQL DB in a future article). I also like to enable “Causality tracking” so that checkbox is ticked in this example. The template has the events I wish to capture already and I am only going to use the ring buffer (file target has some additional requirements for an Azure SQL DB session and should be treated in an entirely separate article) so I will skip those screens straight to the advanced screen before completing this session.

From the advanced screen, all I want to do for now is to decrease the dispatch latency from the default 30 seconds down to 10 seconds. For this session, I just want to capture the events quickly in the target and be able to review them as soon as possible. I won’t be able to utilize the “Watch Live data” option for this session (as shown below), so a quick dispatch is essential.

After I have selected all of the configurations I desire for this session, then I click the script button at the top of the window. From there, I will select “New Query Editor Window.” Once scripted, I will receive the following script for this particular session.

CREATE EVENT SESSION [AzureDB_Demo]
ON DATABASE
	ADD EVENT sqlserver.error_reported
	(ACTION
	 (
		 sqlserver.client_app_name
	   , sqlserver.database_id
	   , sqlserver.query_hash
	   , sqlserver.session_id
	 )
	 WHERE (
			   ([package0].[greater_than_uint64]([sqlserver].[database_id], (4)))
			   AND ([package0].[equal_boolean]([sqlserver].[is_system], (0)))
		   )
	)
  , ADD EVENT sqlserver.module_end
	(SET collect_statement = (1)
	 ACTION
	 (
		 sqlserver.client_app_name
	   , sqlserver.database_id
	   , sqlserver.query_hash
	   , sqlserver.session_id
	 )
	 WHERE (
			   ([package0].[greater_than_uint64]([sqlserver].[database_id], (4)))
			   AND ([package0].[equal_boolean]([sqlserver].[is_system], (0)))
		   )
	)
  , ADD EVENT sqlserver.rpc_completed
	(ACTION
	 (
		 sqlserver.client_app_name
	   , sqlserver.database_id
	   , sqlserver.query_hash
	   , sqlserver.session_id
	 )
	 WHERE (
			   ([package0].[greater_than_uint64]([sqlserver].[database_id], (4)))
			   AND ([package0].[equal_boolean]([sqlserver].[is_system], (0)))
		   )
	)
  , ADD EVENT sqlserver.sp_statement_completed
	(SET collect_object_name = (1)
	 ACTION
	 (
		 sqlserver.client_app_name
	   , sqlserver.database_id
	   , sqlserver.query_hash
	   , sqlserver.query_plan_hash
	   , sqlserver.session_id
	 )
	 WHERE (
			   ([package0].[greater_than_uint64]([sqlserver].[database_id], (4)))
			   AND ([package0].[equal_boolean]([sqlserver].[is_system], (0)))
		   )
	)
  , ADD EVENT sqlserver.sql_batch_completed
	(ACTION
	 (
		 sqlserver.client_app_name
	   , sqlserver.database_id
	   , sqlserver.query_hash
	   , sqlserver.session_id
	 )
	 WHERE (
			   ([package0].[greater_than_uint64]([sqlserver].[database_id], (4)))
			   AND ([package0].[equal_boolean]([sqlserver].[is_system], (0)))
		   )
	)
  , ADD EVENT sqlserver.sql_statement_completed
	(ACTION
	 (
		 sqlserver.client_app_name
	   , sqlserver.database_id
	   , sqlserver.query_hash
	   , sqlserver.query_plan_hash
	   , sqlserver.session_id
	 )
	 WHERE (
			   ([package0].[greater_than_uint64]([sqlserver].[database_id], (4)))
			   AND ([package0].[equal_boolean]([sqlserver].[is_system], (0)))
		   )
	)
	ADD TARGET package0.ring_buffer
WITH
(
	MAX_MEMORY = 4096KB
  , EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
  , MAX_DISPATCH_LATENCY = 10 SECONDS
  , MAX_EVENT_SIZE = 0KB
  , MEMORY_PARTITION_MODE = NONE
  , TRACK_CAUSALITY = ON
  , STARTUP_STATE = OFF
)
GO

Notice that there is nothing about starting this session in the script to start the session.

ALTER EVENT SESSION [AzureDB_Demo] ON DATABASE
STATE = START;

Note that “on database” has been specified instead of the traditional “on server” for the session state command. In order to stop the session, it is just as easy as follows.

ALTER EVENT SESSION [AzureDB_Demo] ON DATABASE
STATE = STOP;

Conclusion

This has been a very simple introduction into the creation of an Extended Event session using a template for Azure SQL DB. I demonstrated the use of the GUI to configure the session quickly and then to subsequently script that configuration before creating the session.

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Interested in seeing the power of XE over Profiler? Check this one out!

This is a quick pre-cursor to the 2018 “12 Days of Christmas” series. The series will begin on December 25th (the first day of Christmas) and you can easily follow it via this page.

We have all run into a need or a desire to clean up old stale files from the file system, whether it be to remove old backup files or flat files that are created from one process or another.  And based on this need/desire, we have all come up with a method to help with achieve that goal.

Some of the methods might be to include a flag in a maintenance plan that may be used.  Other methods may be to use a SQL script employing xp_cmdshell and delete statements.  Yet another may utilize the sp_oa stored procs and DMO.  And still others may have ventured into powershell to accomplish the same task.  The point is, there are many methods.

I am adding yet another method to the mix.  Why?  I didn’t much like the option of using the sp_oa method or the xp_cmdshell route.  I am very novice with powershell and it would take a lot more tinkering to get the script working properly.  Also, I felt pretty comfortable with SSIS and had approval to try and get this done using that method.  And just because I am a novice with powershell, does not mean that I will not circle back around to try and accomplish this task via that means.

Note: This article was originally written in 2011 and got stuck in an unpublished state. Things have changed since then so i will definitely be circling back around for a powershell version.

Requirements

The method employed needs to be able to do the following:

1. Remove multiple file types
2. Be configurable
3. Clean out files from numerous directories
4. Remove files older than a specified number of days.

Setup

The solution I chose utilizes SSIS.  It also requires that there be a table in a database that helps to drive the package.

The table looks like the following.

USE [AdminDB_Test]
GO

/****** Object:  Table [dbo].[FilePaths]    Script Date: 12/05/2011 23:36:26 ******/
IF  EXISTS (SELECT * FROM sys.objects WHERE object_id = OBJECT_ID(N'[dbo].[FilePaths]') AND type in (N'U'))
DROP TABLE [dbo].[FilePaths]
GO

USE [AdminDB_Test]
GO

/****** Object:  Table [dbo].[FilePaths]    Script Date: 12/05/2011 23:36:26 ******/
SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO

SET ANSI_PADDING ON
GO

CREATE TABLE [dbo].[FilePaths](
	[PathID] [int] IDENTITY(1,1) NOT NULL,
	[Process] [varchar](32) NULL,
	[FilePath] [varchar](256) NULL,
 CONSTRAINT [PK_FilePaths] PRIMARY KEY CLUSTERED
(
	[PathID] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]
) ON [PRIMARY]

GO

SET ANSI_PADDING OFF
GO

The filepath column holds the FileSystem Path for each directory that needs to be cleaned.  Paths that are supported are local (e.g. C:\temp ) and unc paths (\\machine\c$\temp).  I set this attribute to a length of 256, but if you have a longer path, you will want to adjust the length.

The Process column will hold a value describing what that path relates to, such as MaintainDirectory.  In my example, I am using MaintainDirectory to control which directories hold files that potentially need to be deleted.

Here is an example of the contents of that table I am using currently.

The last piece of the setup before we start working on the SSIS package is the need for a string splitting function.  Pick the string splitter of your liking.  I have one that I like and am sure you have one that you prefer.  The SSIS package relies on the return field from the splitter being named “Item.”  If it is named something else, please make the adjustments in the package as necessary.

The Package

The package I created has been created in SSIS 2008.  To meet the requirements already set forth, I utilized the following objects: ADO.Net Data Source, 2 Execute SQL Tasks, 2 ForEach Loop Containers, a Script Task, and 8 variables.  Let’s take a look at these starting with the variables.

Variables

• SQLServerName – The value held here is used in an Expression for the Data Source.  This will overwrite the ServerName value in the Data Source.
• DatabaseName – Used alongside the SQLServerName variable in an Expression for the Data Source.  This value will overwrite the InitialCatalog value in the Data Source.  This should be the name of the database where the FilePaths table and String Split function exist.
• DaysToKeep – This value is the cutoff point for which files to keep and which files will be deleted.  This variable is used as a ReadOnly variable in the Script Task.
• obj_FileExtension – This object variable is used to store the result set from one of the Execute SQL tasks and the results of the string split function from the FileExtensionList variable.
• FileExtensionList – This is a delimited list of file extensions that need to be evaluated for deletion.  It is important to note that the file extensions that are to be processed are case sensitive.  The extension must appear in this list as it appears in the file system.
• FileExtension – to be used in one of the ForEach loops.  This variable will receive the FileExtension from the obj_FileExtension variable one at a time.
• obj_ListOfDirectories – This variable will receive the result set of an Execute SQL Task to be later consumed by one of the ForEach loops.
• DirectoryToMaintain – receives one at a time the Directory to process for file deletion.  The ForEach loop stores a value from obj_ListOfDirectories in this variable for processing.

Execute SQL Tasks

The two Execute SQL Tasks are simple in function.  One is to get the list of directories to maintain from the FilePaths table.  The other is strictly to split the string for the FileExtensionList variable.

The first is named “Get Directory List” and should receive the Full Result Set from the following query.

select fp.PathID,fp.Process,fp.FilePath
	from dbo.FilePaths FP
	where fp.Process = 'MaintainDirectory'
	order by fp.PathID

The Result Set tab of this task also needs to be modified.  it should look like this.

From this task, we flow to the next Execute SQL Task named “Split FileList.”  The setup of this task is very much like the previous task.  We want to receive the full result set.  We have a configuration to make on the result set tab.  We also need to map a parameter.  Let’s take a look at those real quick.

Parameter Mapping

Result Set

And this is the query that we will be executing.

Select Item
	From stringsplitter(@FileExtensionList,',')

Notice that the Parameter we named in the Parameter Mapping tab is being used in the function call.  I chose this method because I could see and understand how it works better.

ForEach Loops

The next stop in the flow is the ForEach Loop – Directory object.  As the name implies, this ForEach Loop is designed to work with the obj_ListOfDirectories variable/array.

With this first Loop container, we have two tabs that need to be configured in the properties.  Both Loop containers are similar in that they need the same tabs to be configured.  First, let’s talk about the Collection tab.

On the Collection tab, we need to set the Enumerator option to “ForEach ADO Enumerator.”  Then we need to select the obj_ListOfDirectories from the drop down labeled “ADO Source Object Variable.”  Your screen should look like the following image.

With this tab configured, we can focus our attention to the quick changes that need to be made on the Variable Mappings tab.  On this tab, we are telling the enumerator how to handle the data from the object variable.  We are mapping columns from the result set to variables for further consumption.  When configured, it should look like the following.

Inside of this ForEach loop container, we have another ForEach loop container.  This second ForEach loop container handles the file extensions that we listed out in delimited fashion in the FileExtensionList variable.  I have called this container “ForEach Loop – FileExtension” (just keeping it simple).

The collection tab follows the same configuration setup.  The difference of course being that this container will use the obj_FileExtension object from the source variable dropdown menu.

The variable mapping tab is also slightly different.  We will be mapping column 0 of the object to the FileExtension variable.  The explanation for the different number between the two loop container variable mappings is simple.  In obj_ListOfDirectories, we have multiple columns being returned.  In obj_FileExtension, we have but one single column being returned.

This inner Loop container will loop through each of the extensions for each of the directories that have been returned to the outer loop container.  the inner loop container has the remainder of the workload in it via the Script Task.

Script Task

It is via the script task that we actually get to start deleting files.  This was the most difficult piece of the entire package – though the script is not very large.

For the script task, I chose to implement it via the Visual Basic option (instead of C#).  I have three ReadOnlyVariables employed by the script.  Those variables are: User::DaysToKeep,User::DirectoryToMaintain, and User::FileExtension.

Once you have set those on the script tab, the next step is to click the Edit Script… button where we need to place the following script.

' Microsoft SQL Server Integration Services Script Task
' Write scripts using Microsoft Visual Basic 2008.
' The ScriptMain is the entry point class of the script.

Imports System
Imports System.IO
Imports System.Data
Imports System.Math
Imports Microsoft.SqlServer.Dts.Runtime

<System.AddIn.AddIn("ScriptMain", Version:="1.0", Publisher:="", Description:="")> _
<System.CLSCompliantAttribute(False)> _
Partial Public Class ScriptMain
	Inherits Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase

	Enum ScriptResults
		Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success
		Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
	End Enum

    Public Sub Main()
        Dim SourcePath As String
        Dim PurgeDays As Integer
        Dim FileExtension As String
        Dim thisFileInUse As Boolean = False

        PurgeDays = CInt(Dts.Variables("User::DaysToKeep").Value)
        SourcePath = CStr(Dts.Variables("User::DirectoryToMaintain").Value)
        FileExtension = CStr(Dts.Variables("User::FileExtension").Value)

        For Each file As FileInfo In New DirectoryInfo(SourcePath).GetFiles()
            If ((Now - file.LastWriteTime).Days > PurgeDays) And (file.Extension = FileExtension) Then
                Try
                    file.Delete()

                Catch ex As Exception

                End Try
            End If

        Next

        Dts.TaskResult = ScriptResults.Success
    End Sub

End Class

An important note of interest is the need for the Try…Catch.  Without this block as it is, you could run into an issue where the file (such as those pesky temp files) may be in use by some process and cause the package to error.  The Try…catch will move past that nasty error and delete the files that it can.

Inside this script, you will see that I am comparing the LastWriteTime to the PurgeDays and ensuring that the file extension matches one that is in the list.  Then we move into the try…catch and either delete the file that matches those criteria or throw an exception and move on to the next file.

When all is said and done, your package should look something like this.

You should also have a variable list that looks like this.

Each variable that is not an Object has a value assigned to it at this point.  These values will be overwritten where applicable.

Next Steps

Having this package is a good start.  But unless you are prepared to manually run this on a daily basis, it needs to be added to a job and scheduled.  There are two ways to go about scheduling this package.

The first option is to configure the FileExtensionList and DaysToKeep variables and save the package with those values.  Then run this package through SQL Agent with those values every time.  The drawback to this method is that if you need to add or remove a file extension (as an example) then you need to edit the package and re-save it.

The alternative option is pass the values through the job to overwrite those variables as the job runs.  Should you need to remove or add a file extension, it would just be done at the job definition level.

Let’s take a look at this second option.  I will skip past how to create the job as an SSIS job in SQL Server and we will look directly how to modify those variables from the job properties.

To configure these variables directly from the SQL Agent job, open the Job properties and click on the Set Values tab (assuming you have defined this job as an SSIS Job type).  You should get a screen similar to this (void of the set values shown in the pic).  Just add the parameters (variables) we have discussed to this point with appropriate values to fit your needs/environment.

I have chosen to only include the four variables shown above since the remaining variables are either objects or get overwritten in the ForEach loops during processing.  The only thing remaining now is to set the schedule for the job.  Once set, the job (and package) will take care of the rest.

Conclusion

I have now shown you how to maintain some of the directories on your system through the use of SSIS and SQL server.  There are many methods to accomplish this goal, it is up to each of us to choose the best method for our environment and comfort level (by means of supporting the chosen solution).

If you would like to read more interesting stuff concerning SSIS, you might want to check any of these articles: lost password, expected range errors, and synonyms extending SSIS.

Years ago the TSQL Tuesday party was started by Adam Machanic (b|t). The premise of the monthly event is to get people a topic each month to try and ease some of the difficulty with writing (figuring out what to write) and to get more involved in the community. The party started in December 2009 and is now embarking on the 10th year. The party is running strong and will likely continue for many years to come.

I have personally participated in quite a large number of these events. I am far from a perfect participation record like Rob Farley (b | t) but I do participate when I can. Every now and again I think about what I might have participated in or what I have written in the past for a TSQL Tuesday. This post will serve as a table of contents for all of those articles. This is to help make it easier to find for me (as well as give me a chance to review and fix problems from really old posts). This post will provide a quick synopsis of my article (if one exists) along with topic and links to original invite and roundups for the monthly party.

T-SQL Tuesday #001: Date/Time Tricks (Dec. 2009)

Invite and roundup

My Article: N/A

T-SQL Tuesday #002: A Puzzling Situation (January 2010)

Invite and roundup

My Article: TSQL Tuesday – But I was late

This was the first time I participated in TSQL Tuesday. I was very new to the blogging concept – and it shows. The puzzling problem I encountered in the article was a fun little job that kept giving different results than running the same code from SSMS. Check it out!

T-SQL Tuesday #003: RELATIONSHIPS (February 2010)

Invitation and summary

My Article: Relationships

I covered all sorts of different relationships that affect SQL Server from work, to marital, to table relationships in a database. It all ties together in requiring an individual to constantly tune the different types of relationships.

T-SQL Tuesday #004: IO (March 2010)

Invitation and roundup.

My Article: IO IO IO

What if we could create a numbers table without IO?  What if we could perform several different kinds of queries without IO?  Itzik Ben Gan proposed a cascading CTE solution that does this sort of thing. This article shows that method in use to help improve certain IO conditions.

T-SQL Tuesday #005: Reporting (April 2010)

Invitation and roundup.

My Article: IP and Default Trace…T-SQL Tuesday #005

Having been introduced to a requirement to report on IP addresses of connections, I dive into a solution that will help show hostname, ip address and a few more tidbits.

When it is necessary to provide reports on activity occurring on the server, it pays to do a little prep work.  Be Prepared.  It is not an easy task to be able to go back in time and report on data that isn’t captured.  The little prep work that one may need to do is well worth the effort in the end.

My 2nd Article: BLOB Report T-SQL Tuesday #005- Reporting

Learning how to decrypt the individual object sizes broken down into type and filegroup. Then report on the data that was retrieved from the DMOs.

T-SQL Tuesday #006: “What About BLOB?” (May 2010)

Invitation and roundup.

My Article: T-SQL Tuesday #006: A Blobbing We Will Go

In the DMVs for SQL 2005 and SQL 2008 there is more than one place you can find the information about the size of your LOB data?  And it goes without saying that there is more than one way to find information about LOBs in your database.

T-SQL Tuesday #007 – Summertime in the SQL (June 2010)

Invitation and roundup.

My Article: Data Compression

I see database compression, as offered with SQL 2008, to be more like these file compression utilities than DriveSpace.  Data compression in SQL 2008 is not an all or none implementation.  You get to pick and choose what gets compressed.  That is a big time bonus for me.

T-SQL Tuesday #008: Gettin’ Schooled

Invitation and roundup.

My Article: Gettin’ Skewled

I am learning that learning is not just formalized education in a classroom or in specific settings.  There are things to be learned from all aspects of life.  This can be learned if only a little observation is used.

T-SQL Tuesday #009: Beach Time (August 2010)

Invitation and roundup.

I hosted this particular event.

My Article: R & R

I find it completely useless to go on vacation if I am going to be checking email or project statuses every 10 minutes.  There is no rest or relaxation in doing those things while I am supposed to be doing something else.  Vacation should be fun and enjoyable.  Thus, if I am to enjoy vacation, I need to do a few extra things in the office prior to leaving.

T-SQL Tuesday #010 – Indexes (September 2010)

Invitation and roundup.

My Article: TSQL Tuesday Indexes and Blobs

How does one find what columns were LOB columns in the database.  I knew I had some past blog posts about various aspects of LOBs, but I had never broken it down to find the columns in the LOB.  Even better was that I wanted to know what columns were in what index that were also a BLOB.

T-SQL Tuesday #011 – Misconceptions in SQL Server (October 2010)

Invitation and roundup.

My Article: A Haunting TSQL Tuesday Tale

I chose the myth that truncate is unrecoverable. Check it out and see how a truncate is most certainly capable of a rollback or being recoverable.

T-SQL Tuesday #012 – Why are DBA skills necessary? (November 2010)

Invitation and roundup.

My Article: T-SQL Tuesday #012 – Skills

As a DBA, we occasionally have the opportunity of attracting a new client or a new job or a new database.  It seems that more often than not, some of the skills requisite (at least they should be) of owning a database are missing. (Nunchuku skills could come in handy from time to time too!)

T-SQL Tuesday #13 – What the Business Says Is Not What the Business Wants (December 2010)

Invitation and roundup.

My Article: T-SQL Tuesday #13 – Business Requirements

I think a common area that is easily overlooked when it comes to requirements and interpretation of requirements is report creation.  A common problem is that there are no defined or written requirements for the creation of a report.

T-SQL Tuesday #014 – RESOLUTIONS (January 2011)

Invitation and roundup.

My Article: TSQL Tuesday 14: Committed

This month was all about resolutions and goals. My list of goals were pretty good and entirely profession based.

T-SQL Tuesday #015 – Automation in SQL Server (February 2011)

Invitation and roundup.

My Article: T-SQL Tuesday #15 DBA Automaton

I shared a script that will load a sample of data from every table in every database for every column and give you the length of the pertinent columns (I have excluded obvious columns such as numeric types and certain LOB types).

T-SQL Tuesday #016 – Aggregate Functions (March 2011)

Invitation and roundup.

My Article: T-SQL Tuesday #016: Aggregates and Statistics

Super geeky article on stats, quartiles and the like. For my data analysis and trending, I wanted to find a simple distribution across quartiles.

T-SQL Tuesday #017 – APPLY Knowledge (April 2011)

Invitation and roundup – unavailable on origin site.

My Article: T-SQL Tuesday #17 – APPLY Knowledge

In this article I take an unexpected twist in my application of the APPLY operator. In that twist, I show a quick and simple script to grab role members.

T-SQL Tuesday #018 – CTEs (May 2011)

Invitation and roundup.

My Article: T-SQL Tuesday #18 – CTEs

I shared a rather complex series of CTEs that are used in the same way a hierarchy and ancestry tree would be generated – except for Foreign Key relationships. That script was a load of fun.

T-SQL Tuesday #019 – Disasters and Recovery (June 2011)

Invitation and roundup.

My Article: T-SQL Tuesday #19 – Disasters & Recovery

Who needs a single disaster when you can enjoy multiple disasters in a single sitting? I share three different types of disasters. I am sure many have had experiences with all three and probably many more types of disasters.

T-SQL Tuesday #020 – T-SQL Best Practices (July 2011)

Invitation and roundup.

My Article: N/A

I missed the announcement or something like that.

T-SQL Tuesday #021 – A Day Late and Totally Full Of It. (Aug. 2011)

Invitation and roundup.

My Article: TSQL Tuesday 21 – FAIL FAIL FAIL

I don’t hide much as I share my feelings about crap code and then show my own crap code followed by a little bit of explanation on an improved version of the same code.

T-SQL Tuesday #022 – Data Presentation (September 2011)

Invitation and roundup

My Article: T-SQL Tuesday #22 – Data Presentation

I chose to touch on several aspects of data presentation – Performance, Accuracy, Display, and Business Requirements.

T-SQL Tuesday #023 – Joins (October 2011)

Invitation and roundup.

My Article: N/A

D’oh – missed another one in 2011.

T-SQL Tuesday #024 – Prox ‘n’ Funx (November 2011)

Invitation and roundup.

My Article: T-SQL Tuesday #024: Prox ‘n’ Funx

Big takeaway from this month was the value of a very good string splitter.

T-SQL Tuesday #025 – Invitation to Share Your Tricks (Dec. 2011)

Invitation and roundup.

My Article: T-SQL Tuesday #025 – Holiday Gifts

In the spirit of the Holidays, I wanted to share some tricks and tips.  They can be my gifts to you during the holidays.  And maybe they can help you give to somebody else.

T-SQL Tuesday #026 – Second Chances (January 2012)

Invitation and roundup. (Original site toasted).

My Article: TSQL Tuesday #26 or #23 – Identity Crisis

Having missing TSQLTuesday 23 and the topic being about Second Chances, I chose to write about JOINs and sort of fulfilling the requirements for both TSQL Tuesdays in one shot.

T-SQL Tuesday #027 – Invitation to The Big Data Valentine’s Edition (February 2012)

Invitation and roundup.

My Article: N/A

After that second chance I blew it the very next month. Luckily I have a good reason – Valentines Day!

T-SQL Tuesday #028 – Jack of All Trades or Master of None (March 2012)

Invitation and no roundup.

My Article: T-SQL Tuesday #028 – Jack of All Trades, Master of None?

This one brings up bad memories of when as the DBA it was also required to be the janitor.

More updates coming soon!

What is T-SQL Tuesday?

T-SQL Tuesday is a monthly blog party hosted by a different blogger each month. This blog party was started by Adam Machanic (blog|twitter). You can take part by posting your own participating post that fits the topic of the month and follows the requirements below. Additionally, if you are interested in hosting a future T-SQL Tuesday, contact Steve Jones via the tsqltuesday website – here.

How to Participate

• Your post must be published between 00:00 GMT Tuesday and 00:00 GMT Wednesday.
• Your post must contain the T-SQL Tuesday logo from above and the image should link back to this blog post.
• Trackbacks should work. But, please do add a link to your post in the comments section below so everyone can see your work.
• Tweet about your post using the hash tag #TSQL2sDay.

This month I am in charge of the topic for TSQLTuesday. The invite was published here a couple of weeks ago. I knew I was picking something rather difficult for people in general and probably more difficult for technologists due to the notion of jumping head first into a discussion with somebody about yourself. I have been asked by previous employers to do my own personal annual job review. This idea is very much like that – but more personal.

Some say it eats at the core of being prideful to discuss this topic or to broach the “review” with others. I like to take a different angle on it. I see it as a means to personal growth and has nothing to do with bragging. I tend to find that people gifted with humility will ask others how they could improve professionally or personally. Being able to find where you are doing well or not so well is key to progress. I would hope those that participated found this difficult exercise helpful in uncovering a new angle on personal and/or professional improvement.

Discomfort

There is no hiding from the fact that this task provided a great deal of discomfort and probably angst as demonstrated from this tweet.

Multiple participants, as you will see as you read their articles, shared their feelings on the level of discomfort and difficulty they had with the topic. I expected the discomfort, I felt it too.

As I read the articles, I could feel the discomfort alongside the author. I get it – it’s hard because none of us want to come off as bragging about ourselves. Take into account that discomfort and then the message that each of these people has to share with you!

WHO?

Steve Jones (b | t) – Influence. Steve is not comfortable writing about this topic as he dives in to share with us. As he does he hits a very strong note for me. We, the SQL Community, give a lot of ourselves and that giving grows exponentially. We give of ourselves and many times we end up the recipient of this exponential experience.

It can be scary and intimidating to share knowledge with others publicly, but it is also immensely rewarding! -Steve Jones

I believe the same can be said of sharing a personal review publicly.

Wayne Sheffield (b | t) – Influence Somebody. I personally know that Wayne had a hard time with this topic and sharing an article. Wayne shares a couple of stories outlining some very small actions on his part. He was largely unaware of his influence and in turn ended up dumbfounded and awestruck at what his influence meant to other people.

Small actions. That’s all it takes to make a positive difference in a person’s life. Be interested, and encouraging. You never know what kind of effect you are truly having on people. And when you do find out, prepare to be astonished. – Wayne Sheffield

Malathi Mahadevan (b | t) – To Influence and be Influenced. Mala is a very humble person. Mala shares a personal experience about her influence several other people as she worked on her very first book project this past year. The really cool part is the joy she got from seeing many people filled with the joy of having the opportunity to be in the book.

It is a positive way to end the year on a tone of gratitude to people who have influenced you – and it is doubly positive/uplifting to see/hear of what you have done to other people too. – Mala

I am grateful for the friendship I have with Mala, she is a genuine and good person.

Shane O’Neill (b | t) – Influence Somebody. Shane shares how difficult this task was going in and even stumbles on a bit of writers block. Shane doesn’t go out daily to try and find somebody to profoundly impact. He goes about his day in an unassuming way as he just does what he does – shares knowledge just because it needs to be shared. Shane is aware of the impact as people have met him and thanked him personally for his contributions to the community. That’s a win win!

I’ve had some great surprises from people in the community about stuff they’ve seen on here.

It’s nice to know that when you’re shouting into the void, that the void sometimes shouts back.

Each token of appreciation, each expression of gratitude is a replenishment to my morale.

Just a way to say that all efforts put in, while not showing straight away, build up. – Shane O’Neill

I am sure many of us have felt that replenishment to morale from the various tokens of gratitude. The sincere and personal sharing of appreciation carries so much more weight and replenishes the morale so much more completely.

Rob Farley (b | t) – People I Influence. Rob is a humble kind and caring person! He struggled with the topic as he illustrated on twitter. All of that aside, he hit on a note I was hoping somebody would hit on – employees and coworkers. Employees are a reflection of leadership. If the leader is a class act, s/he will impact the employees in a positive manner for the better of the company and the employee.

That aside, Rob is hopeful of being an influence to other people outside of work. Rob is not comfortable with thinking about whether he is influential or not and I get it. I don’t think many of us think regularly about our sphere of influence – until somebody gives us that token of gratitude for some small action we performed.

For a while now, I’ve thought that I would rather employ MVPs than be one myself (although I hope this doesn’t actually cause me to lose my status), and I have often looked for opportunity to get my team in front of people.

As an employer who encourages his staff to be community minded and on the path to being MVPs like Rob is, I think this an outstanding level of influence that is merely demonstrative of the humility that resonates within Rob.

Jason Brimhall (me) – To Influence or be Influenced. I share a couple of experiences in my entry for the month. More important than the stories is the message that as one gives of themselves, there is a gain that will be returned to that person at some point that is likely exponential in nature to the little token of giving that happened in the first place. Give and you will receive! Steve hit on that same note.

And in case you are still curious what this TSQL Tuesday thing is, here are the details…

What is T-SQL Tuesday?

T-SQL Tuesday is a monthly blog party hosted by a different blogger each month. This blog party was started by Adam Machanic (blog|twitter). You can take part by posting your own participating post that fits the topic of the month and follows the requirements below. Additionally, if you are interested in hosting a future T-SQL Tuesday, contact Steve Jones via the tsqltuesday website – here.

How to Participate

• Your post must be published between 00:00 GMT Tuesday, December 11e, 2018, and 00:00 GMT Wednesday December 12e, 2018.
• Your post must contain the T-SQL Tuesday logo from above and the image should link back to this blog post.
• Trackbacks should work. But, please do add a link to your post in the comments section below so everyone can see your work.
• Tweet about your post using the hash tag #TSQL2sDay.

When SQL Server 2005 was released, a nifty little feature was included called synonyms.  Despite being around since SQL Server 2005, I think this feature is often under-utilized or, more importantly, it is implemented in a very bad way.

Today I want to share a couple of examples.  We will take a look at examples of both good and bad implementations of synonyms.

First, let’s take a look at the syntax for creating a synonym.  Per BOL (and for the fun of it, here is the 2005 link).

-- SQL Server Syntax

CREATE SYNONYM [ schema_name_1. ] synonym_name FOR <object>

<object> :: =
{
    [ server_name.[ database_name ] . [ schema_name_2 ]. object_name 
  |  database_name . [ schema_name_2 ].| schema_name_2. ] object_name
}
-- Windows Azure SQL Database Syntax

CREATE SYNONYM [ schema_name_1. ] synonym_name FOR < object >

< object > :: =
{
    [database_name. [ schema_name_2 ].| schema_name_2. ] object_name
}

So a sample implementation of a Synonym could be something like the following.

CREATE SYNONYM [syn].[Staging_Cardumum] FOR [Cardamum_UAT].[dbo].[cardamum_staging]
GO

Before we delve into that sample synonym, lets look at an example of a really bad implementation.

The BAD

While working with a client, I received a request to look into why a linked server query was failing.  (This isn’t necessarily the bad, just hang in there for a bit.)  The message was something like this:

The OLE DB provider “SQLNCLI10” for linked server “blahblahblah” indicates that either the object has no columns or the current user does not have permissions on that object.

The error message seems at least semi-descriptive and gives a starting point.  In this case, I decided to verify the linked server was created properly, verified that the permissions were done properly and even tested the linked server.  On the source (linked) and destination server (let’s call the Source server ServerA and the Destination server we will call ServerB), I verified that permissions were in place for each database to be touched.  Still no dice!

Well, let’s go take a look and see if that referenced table actually exists.  It did not!  Does it exist as a view?  It did not!  Alas, the table existed as a synonym.  This is where it gets wonky.  In looking at the definition of the synonym, I found that the table defined in the synonym had a linked server table as its source.  To top things off, the linked server was back on the originating server that was coming across the link in the first place.  So yes, that would be ServerB initiated a query against ServerA to pull data back to ServerB.  But the data needed (as defined by the vendor) was available on ServerA – supposedly.  Reality had that data actually sitting on ServerB the whole time.

At any rate, thanks to having a synonym for each and every table sitting on ServerA that referenced a table across a linked server on ServerB, we had mass confusion.  In the end, the query was far simpler to execute by just pulling it from the originating query server (ServerB).

This implementation of a synonym was not the best.  All it did was cause confusion, create documentation inaccuracies and delay the developer from accomplishing her task.  Do you really need 1000s of synonyms in your database?  Do you need all of them going across a linked server?  If you do, did you ever think about the potential for performance issues?  (The vendor in this case used those linked servers and synonyms to perform a data conversion that took 36 hrs each time for a rather small dataset – ouch!!!!).

On the other Hand

Imagine, if you will, two databases sitting on the same box.  One database will be for your user data, and the other for “staging” data for processes such as those related to ETL.  Imagine further that, for some inane reason, the naming standard of your databases must include the suffix denoting the environment of the database.

Now picture an SSIS package that must utilize tables from both databases in data sources somehow.  At least one of the tables has millions of records.  And both of the tables have nearly 100 columns.  Even just pulling in the minimum amount of data using a query from each source can cause memory issues.  Two separate data sources means you will likely have a sort transformation (for each source) as well as a join transformation.

Trying to reduce the amount of data in the larger table source could be done via TSQL.  But to reference a database in one environment versus another environment means a code change with each deployment (due to the TSQL – think three part naming).  So you have been hampered by the environment.  Or have you?

By using a synonym in this situation, the data can be joined in a tsql data source by referencing that synonym.  Let’s look back at the sample synonym posted earlier in this article.

CREATE SYNONYM [syn].[Staging_Cardumum] FOR [Cardamum_UAT].[dbo].[cardamum_staging]
GO

You can see that this synonym follows the same sort of naming standards as was just laid out in the preceding scenario.  If I create a synonym in each environment by the same name, and referencing the appropriate environment named database, I have just opened up a performance tweak for my SSIS datasource.

By implementing this slight tweak, I have been able to gain a 10x performance improvement in package performance.  I am now requiring SSIS to ingest fewer records and thus chew up less memory.  Fewer transformations are required and the package can just fly into the required transformations, rather than tinkering around with the transformations needed to just get the data into a usable state for those transformations.

There are other benefits within SSIS to using synonyms for databases on the same server as well.  Especially when dealing with this kind of naming standard that requires the databases to be named differently in each environment.

Conclusion

How you use a synonym can be a huge asset or it can be a significant dampener to performance.  There are benefits and uses for these nifty little things.  Check them out and let us know how you have been able to put synonyms to use to benefit you.

Check out some of these other articles on synonyms here and here.

We all inherit things from time to time through our profession.  Sometimes we inherit some good things, sometimes we inherit some things that are not so good.  Other times we inherit some things that are just plan annoying.  Yet other times, we inherit things that may be annoying and we probably just haven’t discovered them yet.

Dizzying, I know.

Inheritance

Have you ever taken over a server that had several maintenance plans on it?  Have you ever really checked who the owner of those plans is?  Or, maybe you had a failing job relating to one of these maintenance plans and you changed the job owner, but did you really fix the root cause?  That could be one of those things that you inherited that could be annoying but you just don’t know it yet.

Step by Step

No this is not New Kids on the Block (I think I just threw up in my mouth thinking that).

Let’s create a generic maintenance plan and see what happens.

The first thing we do is navigate to Maintenance Plans under the Management menu in Management Studio.

Right Click the Maintenance Plan folder and select New Maintenance Plan… from the context menu.  This will prompt us with the following dialog box.

In this box, we can type a name for this Maintenance Plan that is to be created.  I chose MaintPlanOwner, since that is the topic of this article.

After clicking ok on this dialog box, you will be presented with a blank canvas with which to design your maintenance plan.  I have chose a simple task for the purposes of this article.

I will create a subplan named Statistics and add the Update Statistics task to the canvas.

You can see this illustrated to the left.  I chose to update the statistics on all databases and left all other options as the default option – for simplicity of this article.

At this point, the only thing left to do is to save this Maintenance Plan.  Once the plan is saved, then we can move on to the next step – some fun with TSQL.

Fun with TSQL

This is the stage of the article where we get to play with TSQL and investigate at a high level the Maintenance Plan we just created.

Within the msdb database, we have some system tables that store information about SSIS packages, DTS packages, and Maintenance Plans.  We will be investigating from a SQL 2008 and SQL 2005 standpoint (it changed in 2005 and then again in 2008).

In SQL 2005, we can query the sysdtspackages90 and sysdtspackagefolders90 to gain insight into who owns these Maintenance Plans.  In SQL 2008 and up, we can query sysssispackages and sysssispackagefolders to gain the same insight.  These system tables are within the msdb database.

In SQL Server, we can use the following to find that I am now the owner of that maintenance plan we just created.

USE msdb;
GO
DECLARE @ServerMajorVersion DECIMAL(4, 2)
		,@SQLProduct VARCHAR(24)
		,@SQL VARCHAR(MAX);

SELECT @ServerMajorVersion = CONVERT(DECIMAL(4, 2), PARSENAME(dt.fqn, 4) + '.'
		+ PARSENAME(dt.fqn, 3))
	FROM ( SELECT CONVERT(VARCHAR(20), SERVERPROPERTY('ProductVersion'))
			) dt ( fqn );
IF OBJECT_ID('tempdb.dbo.#SQLVersions') IS NOT NULL
BEGIN
	DROP TABLE #SQLVersions;
END

CREATE TABLE #SQLVersions
	(VersionID INT IDENTITY(1,1)
		, Product VARCHAR(32)
		, ServerMajorVersion DECIMAL(4, 2) --INDEX CI_ServerMajorVer CLUSTERED
	);
CREATE CLUSTERED INDEX CI_ServerMajorVer ON	#SQLVersions (VersionID);

INSERT INTO	#SQLVersions
		( Product
			, ServerMajorVersion
		)
	VALUES
		( 'SQL Server 2008', 10.00)
		,( 'SQL Server 2008R2', 10.50)
		,( 'SQL Server 2012', 11.00)
		,( 'SQL Server 2014', 12.00)
		,( 'SQL Server 2016', 13.00)
		,( 'SQL Server 2017', 14.00)
		,( 'SQL Server 2019', 15.00)
		,( 'SQL Server 2005', 9.00);

SELECT @SQLProduct = Product
	FROM #SQLVersions sv
	WHERE sv.ServerMajorVersion = @ServerMajorVersion;

SET @SQL = '
SELECT	p.name AS PackageName
		, '''+ @SQLProduct + ''' AS SQLVersion
		, p.createdate
		, pf.foldername AS PackageType
		, sp.name AS PrincipalName
	FROM ' + CASE WHEN @ServerMajorVersion < 10 THEN 'msdb.dbo.sysdtspackages90 p
		INNER JOIN dbo.sysdtspackagefolders90 pf
			ON p.folderid = pf.folderid'
			ELSE 'msdb.dbo.sysssispackages  p
		INNER JOIN dbo.sysssispackagefolders pf
			ON p.folderid = pf.folderid' END + '
		LEFT OUTER JOIN master.sys.server_principals sp
			ON p.ownersid = sp.sid
	WHERE pf.foldername = ''Maintenance Plans''
	ORDER BY PackageName;'

EXECUTE (@SQL);

IF OBJECT_ID('tempdb.dbo.#SQLVersions') IS NOT NULL
BEGIN
	DROP TABLE #SQLVersions;
END

Notice that in this query, I delve out to the sys.server_principals catalog view.  I did this to retrieve the name of the owner of the package that was found in the sysdtspackages90 and sysssispackages tables respective to version of SQL Server. I also am running a dynamic SQL query to support both views dependent on version of SQL Server.  I figured this might be a tad more helpful than the previous version here. This query would yield the following result set for that new “Maintenance Plan” that was just created.

Caveat

Let’s assume that this package is scheduled via a SQL Agent job on a production server.  I then get moved to a different department and no longer have permissions on this particular production server.  The job will start failing due to the principal not having access.  One fix would be to change the owner of the job.

That will work.  However, there is a problem with that fix.  As soon as somebody opens and saves the Maintenance Plan, the owner of the job will revert back to the owner of the Maintenance Plan.  When that happens, then the job will fail again.

A permanent fix is needed.  The permanent fix is to change the owner of the Maintenance Plan.  The following will change the owner to ‘sa’ for both SQL 2005 and SQL 2008 (and up).

SQL 2005

USE msdb;
GO
UPDATE p
SET ownersid = 0x01
FROM msdb.dbo.sysdtspackages90  p
	INNER JOIN dbo.sysdtspackagefolders90 pf
		ON p.folderid = pf.folderid
WHERE pf.foldername = 'Maintenance Plans';

SQL 2008

USE msdb;
GO
UPDATE p
SET ownersid = 0x01
From msdb.dbo.sysssispackages  p
	INNER JOIN dbo.sysssispackagefolders pf
		ON p.folderid = pf.folderid
WHERE pf.foldername = 'Maintenance Plans';

Now if you run the code used earlier to investigate, you will find that the owner has indeed changed.  The results of that query should be similar to the following.

There you have it.  No more hair tugging over something as benign as the owner of a Maintenance Plan.  This is one of those things that should be looked at as soon as you inherit a new server.

The Wrap

In this article I took a rather long route to a simple fix. It’s easy to try each of the steps I showed in this article thinking it will help. It isn’t illogical to try some of those steps. They just don’t work unfortunately. In the end, getting to know the settings in the database and what the errors are really trying to get at is most helpful. Sometimes, it just takes a few more steps to get to the real meaning of the error.

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

Well, have I come up with a doozy of a topic for this month for the TSQLTuesday party. Its not easy to step out of ones own comfort zone. Its even more difficult to fathom the thought and action of performing a self review. That is in essence the task at hand – self review or have somebody else review your personal contributions to the community.

In the invite, I mentioned how many of us have been able to come up with lists of people that have impacted us personally. Using that previous exercise would be a pretty good resource for some to maybe find out more how it was they influenced another person in the community.

To be candid, the invite is not just for those who have influenced others but also for those who have been influenced. If somebody had an impact on your career or your personal identity, take a minute to chat with them and let them know!

Surprises

How great is it to provide a service to somebody and then to receive a legitimate and meaningful thank you? The SQL community gives and gives and gives and all too often there is no personal note of gratitude for the help given. Every once in a blue moon, that thank you comes and a really cool calming feeling may just overwhelm you. That is when you know you have done good and have meant something to somebody in their life or career.

I am not talking about the flippant thumbs ups, or likes as you may see in social media. Granted, seeing those likes and retweets is probably nice to see on occasion. It just doesn’t bring a real valid connection to the people you are trying to help.

These kind of personal connections are the types of things that build friendships, networks, and long lasting relationships. Someday, this relationship may turn into a client or co-worker or even future mentor (we all can learn something new from other people).

I have had this kinds of conversations more than a few times with the most recent just occurring today from a client. The latest opportunity came as a candid conversation after taking a week long vacation. The client asked me over for a quick chat and reminded me of how thankful they are for the effort and long hours I have given them. The stability they now see with the environment has brought peace of mind and comfort knowing the database will be up when they need them. It has been a culture shift in how databases are viewed there and it has been a good and meaningful shift. This organization is now setup to bring on more talent to a database team.

Just as meaningful is the opportunity I had to chat with another MVP. This person pulled me aside at a very popular conference. I must admit this one blew me away because I had no idea. This person proceeded to tell me how thankful he was that I was a good example as a friend and mentor. He was also appreciative of the time I would take to answer his technical or non-technical questions. Unbeknownst to me, this was happening while he was hitting a rough patch (his words).

This second experience was extremely humbling to me. I share this because I was unaware of the profound impact I was having on this person who later became an MVP. I didn’t do anything super special. I just did what I felt was right. The experience reminded me of why I do the things I do in the community. While extremely gratifying to give and help, it is also very humbling to be able to give, help and be reminded of how a little effort can have a profound effect.

For me, this reminder came when I was personally getting a little down. I needed the reminder to keep it humble and continue to give. I enjoy helping in various ways in the SQL community and I was getting a little sidetracked from doing those things I enjoy. So by these people pulling me aside and telling me about the influence I had on them, they unwittingly also had an impact on me. As I have said multiple times, when you give, you also get something in return. That is the nature of service.

Circling Back

Just because you give to the community doesn’t mean you can’t also receive. It also doesn’t mean that it has to be done without ever knowing the true impact of your giving. You simply cannot improve yourself if you have no clue how well you are doing.

Getting feedback from others is that little check that is needed to help progress and move in the right direction. When allowing time and opportunity for feedback from others, you are doing yourself a service as well as them. As I noted, these people that gave me this feedback not only told me about how much I helped them, but they also impacted me at the right moment to help me remember the reasons behind why I like to help. Give a little and you will also get a little!

Then come the times when you are new to an environment and you really need to know some basic information about the servers you have just inherited. Well, truth be told, you don’t necessarily have to be new to a job to inherit some new servers. Sometimes, blackbox servers end up getting dropped into your realm of responsibility.

When you run into something like this, you really need to have some means to figure out what changed, when it changed and who changed it. Having that information as a base, you would then have the tools to inquire as to why it might have been changed. One of the things that would be nice to know is around the trace flags that can be enabled in SQL Server. There are trace flags (or knobs) for so many different things within SQL Server that it would be nice to get a handle on them if they do happen to be enabled or even disabled.

As a DBA in SQL Server, you should know by now that there are multiple different ways to do all sorts of things within SQL Server. Today we will discuss three different methods that you could use to perform this investigation for yourself.

Report Style

Did you know that Management Studio comes with a dashboard report? It does! In that report, one will find a pretty decent amount of overview information as a first pass to get to know your server. If you were to right click the instance within SSMS, you can browse to the Reports/Standard Reports submenus. From there you can select the Dashboard report. The path to the reports is similar to what is displayed in the following screen capture.

The report that is of interest is circled in red in the preceding image.

If you have any trace flags enabled (globally) you can see those listed in the report similarly to the following image:

Once again, I have encircled the relevant data in red. From this instance that I queried, I currently have one trace flag enabled. As it stands, this particular trace flag is the one that disables the noise of successful backups from being recorded to your log.

This is one particularly easy method to retrieve that information. A benefit from this method is that you will be able to see some graphs and charts and so forth by using the report.

SQL Query

As logic would dictate, it would seem that we should be able to trap that information through the use of our most frequently used tool – TSQL. Case in point is that the report itself requires the use of TSQL to produce the dataset for the nice display we would see when the report is rendered.

Using similar code to what is used to produce the Dashboard report, we can get a decent understanding of the trace flags that might be employed on your particular instance.  Let’s take a quick look at what we could do.

CREATE TABLE #tracestatus(
        TraceFlag nvarchar(40)
,       Status tinyint
,       Global tinyint
,       Session tinyint
);

insert into #tracestatus 
	exec('dbcc tracestatus')
update #tracestatus 
	set TraceFlag = 'Traceflag ('+TraceFlag+')'

select 1 as l1
,       TraceFlag as name
,       convert(nvarchar(15), Status) as run_value
,       '0' as default_value
,       1 as msg  
from #tracestatus 
where Global=1 order by name

GO

As you can see, that code is terribly simple. The crux of the code boils down to a DBCC command to return the results of TRACESTATUS. Granted, this is a bit over the top and designed to dump the data into a table for later consumption (for good reason). You can legitimately get away with simply running DBCC TRACESTATUS. The results of this would produce the following for the same instance of SQL Server shown in the Server Dashboard report we already explored. Those results look like the following.

So far, so good. If I had several trace flags enabled globally on this instance then I would see every single one of them in the result set just displayed. Cool!

To this point, this is wonderful for anything that might be currently enabled on the instance. Should any of those trace flags be disabled shortly after being enabled, or even be disabled months after being enabled but before you run either of the two preceding methods, then you are up a creek without a paddle. That is a problem. How would you trap that cowboy sysadmin or cowboy DBA in that case? Well, you could go with a full blown audit. Or you could try a similar approach but on a somewhat smaller scale.

If you are paying close attention, you are probably thinking to yourself that the query just demonstrated seems more complex than it need be.  After all, the documentation for DBCC TRACESTATUS says that I can run that command with a -1 parameter and just have it return the global trace flags.  And in the script I am showing, I filter on Global = 1.  Well, as it turns out, the -1 option in DBCC TRACESTATUS does not work as documented.  The -1 parameter does not return the appropriate result set and that is reproducible.  If you were curious, the MSDN reference is here.  The use of that parameter value still returns session flags as well.

Extended Event

Oh my, we are talking about something really abstract and difficult now! No, not really.  This is a pretty straightforward approach and easy enough to implement. Within XE, there is an event called trace_flag_changed. If you were curious, you could use a query such as the following to find any events that might be related. The key is finding the correct key terms to find what you need/want.

SELECT c.object_name as EventName,o.description as EventDescription
	FROM sys.dm_xe_objects o
		INNER JOIN sys.dm_xe_object_columns c 
			ON o.name = c.object_name
			and o.package_guid = c.object_package_guid
	WHERE object_type='event'
		AND c.name = 'channel'
		AND c.object_name like '%flag%'
	Order By o.package_guid;

Running that query will return just one result. You might find a few more if you expand your search criteria to include the term “flag” within the description field. It’s up to you to explore a bit more. As the description and name imply for this particular event, you can presume correctly that it tracks when a TF is either enabled or disabled (the state of the TF changes). Let’s call this a mini audit for a very targeted type of event.

Knowing the name of the event, let’s go ahead and look at a possible setup for an event session to trap data:

CREATE EVENT SESSION [TrackTFChange] ON SERVER 
ADD EVENT sqlserver.trace_flag_changed(
    ACTION (sqlserver.database_name,sqlserver.client_hostname,sqlserver.client_app_name,
            sqlserver.sql_text,
			sqlserver.session_id)
    WHERE sqlserver.client_app_name <> 'Microsoft SQL Server Management Studio - Transact-SQL IntelliSense')
ADD TARGET package0.ring_buffer
WITH (MAX_DISPATCH_LATENCY=5SECONDS, TRACK_CAUSALITY=ON)
GO

ALTER EVENT SESSION TrackTFChange 
ON SERVER 
STATE = START;
GO

This gets us to a point where we can now trap all changes to a trace flag status (enabled/disabled). At this point, a bit of a caution is that the event appears to fire twice for every time it occurs. There is one case where it will not produce a double result for each change. That case happens to be if the TF is already set to the same setting to which you are attempting to change it to. So if it is enabled, trying to enable it again just records one event to fire. However, trying to enable or disable the TF when it is in the opposite state, you will see two occurrences of the event firing. Here is a sample output to help show what happens.

Displayed in the preceding image, you will notice both scenarios. In the scenario where the TF was not enabled, and I attempted to enable it, you will see two entries. And then when I disabled that TF after enabling it, I once again got a double entry. A corresponding example is encircled by the green. Now, when I tried to disable a TF that was not currently enabled, you see there is a single entry. That occurrence is marked via the orange circle.

A problem does present itself here with this result set and the use of XE.  How in the heck did I get the results from the XE Session so I could peruse it? Here is a sample script that will help parse this session data.

/* Query the XE */

SELECT
    event_data.value('(event/@name)[1]', 'varchar(50)') AS event_name
	,event_data.value('(event/@timestamp)[1]','varchar(max)') as timestamp
    ,event_data.value('(event/data[@name="flag"]/value)[1]', 'bigint') AS TraceFlag
	,event_data.value('(event/data[@name="type"]/text)[1]', 'varchar(max)') AS FlagType
	,case event_data.value('(event/data[@name="new_value"]/value)[1]','int') 
		When 0 then 'Enabled'
		When 1 then 'Disabled'
		END as NewValue
	,event_data.value('(event/action[@name="sql_text"]/value)[1]', 'varchar(max)') AS sql_text
	,event_data.value('(event/action[@name="database_name"]/value)[1]', 'varchar(max)') AS DBQueryExecutedFrom
	,event_data.value('(event/action[@name="client_hostname"]/value)[1]', 'varchar(max)') AS ClientHost
	,event_data.value('(event/action[@name="client_app_name"]/value)[1]', 'varchar(max)') AS appname
	,event_data.value('(event/action[@name="session_id"]/value)[1]', 'varchar(max)') AS session_id
FROM(    SELECT evnt.query('.') AS event_data
        FROM
        (    SELECT CAST(target_data AS xml) AS TargetData
            FROM sys.dm_xe_sessions AS s
            INNER JOIN sys.dm_xe_session_targets AS t
                ON s.address = t.event_session_address
            WHERE s.name = 'TrackTFChange'
              AND t.target_name = 'ring_buffer'
        ) AS tab
        CROSS APPLY TargetData.nodes ('RingBufferTarget/event') AS split(evnt) 
    ) AS evts(event_data)
WHERE event_data.value('(event/@name)[1]', 'varchar(50)') = 'trace_flag_changed'
ORDER BY timestamp asc;

Now isn’t that really helpful?

Conclusion

Today I covered three different means to help isolate the trace flags in use or the changes to the state of those trace flags. This is good information that any DBA should want to know. It is information to help get an understanding of the health and operating environment of your instance. It is also a means to help identify what might be changing under your nose!

For more uses of Extended Events, I recommend my series of articles designed to help you learn XE little by little.

Need a little help with extensive audits of your servers, check out these articles.

It has now been 28 months since the last time I hosted a TSQL Tuesday, that was TSQL Tuesday 81. I recapped that event here with the original invite here. It continues to amaze me how quickly time flies and how easily years can blow by in between hosting this awesome party!

The last time I hosted, I issued a difficult task. I dare say this one might be just a touch more difficult. First, I want to jump back into the time machine and visit a few of  TSQLTuesday topics that might be relevant to this month’s challenge.

In November 2017, Ewald Cress (b | t) invited everybody to talk about and basically give thanks to people that have helped impact their careers or lives. Check out the roundup here.

In December 2017, Mala Mahadevan (b | t) invited everybody to set goals for themselves. These goals were supposed to be about learning. But when you get down to the nitty gritty, anything that helps build character and career really comes from something that must be learned. It all starts with a bit of introspection. It is this introspection that I ask you to use as a building block for the party this month.

Lastly, in May 2018, Riley Major (b | t) asked everybody to reflect a bit on the theme from Ewald. This time though, the task was to give back to the community. Pay it forward, if you will, given that you had previously benefited from the kindness of somebody else.

Invitation

Building on the work of these three fine individuals, here comes the difficult task. You have been the benefactor of some awesome help from somebody else. You even wrote about it and in a way, told that person how they impacted you, your career, or both. You have set goals for yourself to become a better you after some personal reflection, meditation, introspection. Then you have given back to the community in some way.

I am not asking you to be braggy, just aware and cognizant. What have you done to impact somebody else in the last 13 months?

How do you know you have impacted them? This is really the hard question. I want stories of how you impacted somebody else for the better. This may mean you will need to talk to some people and have a little retrospective with them.

Why?

This past year we lost some real juggernauts in the SQL Community such as Robert Davis (blog). We all know he impacted many people. We can also assume that he knew he impacted peoples lives. How great would it have been to sit down and have a personal discussion with him to let him know for certain how he impacted your career?

At PASS Summit, I had the opportunity to have such a discussion with somebody completely out of the blue. I know how much that meant to me. I also know that I was rather unaware of the influence I had on this individual.

How?

If you have not already had the opportunity to discuss your influence, make the opportunity. If you have mentored somebody, have a chat with them. If you work with somebody that you might have influenced, have a candid chat. Ask them directly if you have been able to be a good influence to them. Ask them how you might be able to better help them.

I know, this gets us all out of our comfort zones – but we need to do things like this. It is a method of both giving thanks as well as just giving (it is the season).

If you are reading this and don’t feel you have influenced somebody, then talk to somebody that has influenced you. Let them know how you influenced them.

Then, after having this candid chat, please write about both the experience (even the awkwardness), anything you learned from the conversation, as well as some details around what it is you did that impacted said individual.

Doing this little exercise will not only help you to become a more involved team member, community member, and leader it will also help you improve on some of the interpersonal skills used for networking as well as public speaking.

What is T-SQL Tuesday?

T-SQL Tuesday is a monthly blog party hosted by a different blogger each month. This blog party was started by Adam Machanic (blog|twitter). You can take part by posting your own participating post that fits the topic of the month and follows the requirements below. Additionally, if you are interested in hosting a future T-SQL Tuesday, contact Steve Jones via the tsqltuesday website – here.

How to Participate

• Your post must be published between 00:00 GMT Tuesday, December 11e, 2018, and 00:00 GMT Wednesday December 12e, 2018.
• Your post must contain the T-SQL Tuesday logo from above and the image should link back to this blog post.
• Trackbacks should work. But, please do add a link to your post in the comments section below so everyone can see your work.
• Tweet about your post using the hash tag #TSQL2sDay.

One of the really super cool things about the SQL community is the frequent opportunity one could have to get involved.

One of my favorite methods of getting involved is the varied ways to participate on a book project. To be involved in a book project does not require that one be an author. One can be a contributor, technical editor, technical consultant, author, or even provide technical reviews after the book is ready for publication.

Each method does come with some sort of invite from either the principle author or the publisher.

I have had a number of various opportunities to be involved with a book. As one ages, it becomes a little more difficult to remember all of the contributions so I just wanted to bring it all together in a single place to help me remember.

Author

SQL Server 2012 T-SQL Recipes. This book came out in 2012 and I had a few people helping me with this first book. I enjoyed this experience even with the difficulties of learning the whole publication process and just getting through the massive job that it was.

DBA Jumpstart.  This was a community effort headed up by John Sansom in 2013. You can check out my contribution here.

SQL Server T-SQL Recipes. This book was released for SQL Server 2014 in 2015. Yeah it came a little late and that is because we had a bit of a late start. This was a follow up edition of the recipes book, only this time with a much smaller bunch of authors.

Consultant

Healthy SQL. This book was released in 2015. I helped Robert get it started and helped him get it across the finish line. This kind of effort included discussions about topics, scripts, and technical edits. If not for time, I was supposed to do a chapter or two and just got too busy to help as an author.

Technical Editor

SQL Server 2012 Data Integration Recipes.  I came fresh off the writing of the T-SQL recipes book in 2012 to immediately plunge into tech editing this book. Tech editor is a pretty good way to get involved because you are immersed in the publication process and you get to validate the content before printing.

Contributor

This is kind of a weird category. Each of the books here has a section from me. So in a sense it is kind of like a small author contribution.

Data Professionals at Work. This book was released in 2018 and is a compilation from several Microsoft MVPs about their personal experiences. I wrote a review about the book here.

How to be an MVP in Life. This book was also published in 2018. This book is another group of MVPs from tech and sport discussing various qualities of an MVP. You can read more of what I said about the book here.

Technical Reviewer

This type of participation comes after publication when a publisher asks for your assistance with a technical review.

SQL Server Transaction Log Management. This book came out in 2013 and was a project by a close friend – Gail Shaw. I do recommend this book to anybody who is working on SQL Server. You can read my review here.

Defensive Database Programming. My first experience with publishers and books came back in 2010 when I was asked to review this book. There are ten articles in the series – starting with chapter 10 here.

An old friend Malathi Mahadevan (b | t) asked me one day if I would be interested in helping with her first book project. Anybody who has participated in the writing of a book knows this can be a rather daunting task – no matter the level of participation. Certainly, the principal author on a book has the grunt of the work, but there is plenty of work to be done for all.

I must say, I gladly accepted the opportunity. Firstly, this was a friend asking. Secondly, I liked the topic and method Mala had chosen. Lastly, I sometimes enjoy a little challenge. Ok, I probably enjoy challenges a little more than the average Joe.

Wowzas

Mala was able to gather quite a stunning group of people together to bring this book to reality. Check out this list (hyperlinks take you to the individuals chapter at Safari): Mindy Curnutt, Julie Smith, Kenneth Fisher, Andy Leonard, Jes Borland, Kevin Feasel, Ginger Grant, Vicky Harp, Kendra Little, Jason Brimhall, Tim Costello, Andy Mallon, Steph Locke, Jonathan Stewart, Joseph Sack, John Q. Martin, John Morehouse, Kathi Kellenberger, Argenis Fernandez, Kirsten Benzel, Tracy Boggiano, Dave Walden, Matt Gordon, Jimmy May, Drew Furgiuele, Marlon Ribunal, Kevin Kline and Joseph Fleming.

Kevin Kline out of that bunch contributed the foreword. If you follow the links, you will notice that the authors appear in order of their chapters.

All of these authors came together to share experiences and insights into the world of a data professional. None of the stories are the same. None of the experiences are the same (not on the micro level anyway). That said, all of the stories do bring the reader an opportunity to learn from the people that have been there and experienced some cool stuff in their careers.

Not only will you encounter some interesting stories, but I believe you will also encounter some interesting insight into some of the contributors. Each person unravels a little bit of their own unique personality while trying to share their passion and love for this field we all excel at (at least sometimes).

If you read closely enough, you might even find a steamy romance hidden between the covers of the book. Then again, maybe that can only be found on a Little blog.

Oh, did you know you can also reach each of the authors on twitter? Check out this list here! Missing from the list is Kevin Kline, but you can find him on twitter here.

Go Get the Book!

With so much great stuff to learn from so many awesome people, you must be asking where you can find the book?

That’s easy! Head on over to Amazon and snag the book (in ebook or paper formats) now! Even with sooo many people contributing to this book, it is a rather short read, coming in at just under 400 pages. It’s also lightweight enough, it could be easily carried around during the lunch break or to and from work.

One of the fun things about being involved in the community is the opportunity to meet new people. Sometimes, meeting new people leads to new opportunities. Most of the time, those meetings turn into friendships of varying degrees.

One person I met at an event is David Lundell (b | t). We had an opportunity to sit down and get to know each other just a little bit. Then we ran into each other at a few more events – funny how that happens. We have a lot in common such as our mindset toward community and being involved in many different ways (coaching sports, scouting, speakings, writing, tech, MVP etc).

After meeting David, I had the opportunity to participate in one of his projects. David was looking to write a book about the MVP mindset and character. Well, the book is now finished! You can find this book on Amazon – here!

David put a lot of work and effort into this book. Not only does he interview Microsoft MVPs, but there is also an interview with the 2016 World Series MVP Ben Zobrist. He strives, in this book, to illustrate the MVP character and how that character is something that helps to build up a team. An MVP may receive a lot of recognition, but usually that recognition also includes higher achievement for the team and more recognition for the team overall.

This book will help you learn how to elevate yourself and your team as you uncover some of the traits related to those MVPs in life, sports, or even work. This book is not just about how to be an MVP for Microsoft, but rather how to elevate yourself or your team to a higher standard in life. Check it out and bring a whole new attitude to how you balance your life and efforts.

Just for techies right?

While the book is predominantly comprised of experiences and interviews from Microsoft MVPs, don’t be fooled into thinking this is a tech book or that it is just for the IT crowd. This book also contains stories and interviews from sports MVPs as well.

Combining the geek crowd and the sports crowd into one book that can relate to all is quite a feat. Those two groups are at pretty disparate ends of the spectrum if you recall your school days at all.

When it comes to characteristics of an MVP, the two groups (geek and jock) are no longer at the opposite ends of the spectrum. Rather, they are very similar in nature which makes this such a great tool to help relate to many teams of different types. Business leaders, youth leaders, coaches, and the it crowd can all benefit from reading this book. The book is just under 200 pages and is available in e-book format on Amazon right now!

Beyond just trapping when these events occur, the DBA usually also wants to know what caused the file to grow or shrink. This has been a problem that has plagued many a DBA over time. Sure there are plenty of methods to assist in the capture of such things. I even wrote about that quite some time ago here. In that article, I gave the tools to help track when a transaction log grew. Today, I am looking to share a different method on how to trap the executing SQL that causes a file to grow. And unlike that previous method, this method will also help to track when a file decreases in size.

I do want to point out that there is an alternate method to capture the shrink events if you wish to read more. I wrote previously about using the default trace to capture when a shrink occurs within the instance. This can be captured thanks to the default trace. You can read all about that over here. Despite the availability of the information within the default trace, I think the method I share here will be lightweight enough that it won’t hurt to have it running—at least occasionally.

Enter the Database Audit

If you have become accustomed to reading my articles, you will probably surmise that there is some degree of setup that needs to be done before I get to the meat of the matter. That holds true today. It just wouldn’t be right to share a method to do something without a harness to show how to get there. So, let’s get the setup out of the way.

Use master;
Go
IF DB_ID('Sandbox2') IS NULL 
BEGIN
	EXECUTE ('CREATE DATABASE Sandbox2');
END
GO

/*We Need The database ID of the newly created Sandbox2 Database - just file it away */
SELECT db_id('Sandbox2') as DatabaseID

Since we will be wanting to create something that will track when a file changes in size, we will need to have a baseline of the size of the files for the newly created Sandbox2 database. Let’s capture that with the following query. Small note on this query is that I am including the tempdb related file sizes. You don’t need to include that, but it may be of interest for future testing.

SELECT DB_NAME(database_id) AS DBName,name AS FileName,size/124.0 AS size_mb
	FROM master.sys.master_files
	WHERE database_id in (2, DB_ID('Sandbox2'));

That is all the setup that is needed at this point. We are now ready to implement a tool that can help us monitor these changes. This tool is lightweight and pretty efficient at capturing the info that would be essential for the exceptional DBA to keep on top of the changes occurring in his/her environment. This tool comes to us in the form of an extended event. Let’s do a little checking into what is available before we dive into the actual XE session itself.

First, when you are curious, if there is an extended event that may be sufficient for your needs, you should check the event store to see what is there. We can do that via a query similar to the following.

SELECT c.object_name as EventName,p.name as PackageName,o.description as EventDescription
	FROM sys.dm_xe_objects o
		INNER JOIN sys.dm_xe_object_columns c 
			ON o.name = c.object_name
			and o.package_guid = c.object_package_guid
		INNER JOIN sys.dm_xe_packages p
			on o.package_guid = p.guid
	WHERE object_type='event'
		AND c.name = 'channel'
		AND (c.object_name like '%file_size%'
			or c.object_name like '%growth%')
	Order By o.package_guid;

This query may return something along the lines of what we see in the following image:

From that list, we can immediately see that we have some options to help us try and capture what is happening with our databases when our back is turned. From here, we can query to find what kind of data is captured with each of these events. To do that, we simply need to run a query such as the following query:

/* Let's take a look at the columns available for each of the events picked out
*/

SELECT * FROM sys.dm_xe_object_columns
WHERE object_name = 'databases_data_file_size_changed'
AND column_type != 'readonly';

SELECT * FROM sys.dm_xe_object_columns
WHERE object_name = 'databases_log_file_size_changed'
AND column_type != 'readonly';

SELECT * FROM sys.dm_xe_object_columns
WHERE object_name = 'databases_log_growth'
AND column_type != 'readonly';

Looking at the available data points in each of these sessions can prove promising. It is indeed promising enough that we can now proceed with the creation of an XE session.

CREATE EVENT SESSION [TrackDBFileChange] ON SERVER 
ADD EVENT sqlserver.databases_data_file_size_changed(
	ACTION(sqlserver.session_id,sqlserver.database_id,sqlserver.client_hostname,
            sqlserver.sql_text)
	)
,ADD EVENT sqlserver.databases_log_file_size_changed(
	ACTION(sqlserver.session_id,sqlserver.database_id,sqlserver.client_hostname,
            sqlserver.sql_text)
	)
ADD TARGET  package0.asynchronous_file_target(
     SET filename='C:\XE\DBFileSizeChange.xel',max_file_size = 5,max_rollover_files = 4
         ,metadatafile='C:\XE\DBFileSizeChange.xem') --if the path does not exist, a nasty error will be thrown
,
ADD TARGET package0.ring_buffer		-- Store events in the ring buffer target
	(SET max_memory = 4096)
WITH (MAX_MEMORY = 4MB, EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS, TRACK_CAUSALITY = ON, MAX_DISPATCH_LATENCY = 1 SECONDS,startup_state = ON)
GO

ALTER EVENT SESSION TrackDBFileChange 
ON SERVER 
STATE = START;
GO

I decided to just focus on the two events that included file_size_changed in the name of the event. You can also see that I chose to send this to two targets. You would be fine to just send this to an asynchronous file target. Just bear in mind that you do need to specify a path for the file_target that does exist or an error will be thrown. I have also specified that this session will restart on server startup and then I turned the session to the started state.

Does it work?

That is a really good question. Now that we have a test database and a session to trap events, all we need is to figure out how to test it. We would want to do a few different things to test—like grow and shrink a file in some way. And that is exactly what we are going to do at this point.

First test, since we have a really small empty database, is to try and force the database to grow by inserting some data. We will do that with this next query.

SELECT TOP 100000
        SomeID       = IDENTITY(INT,1,1),
        SomeInt      = ABS(CHECKSUM(NEWID()))%50000+1,
        SomeLetters2 = CHAR(ABS(CHECKSUM(NEWID()))%26+65)
                     + CHAR(ABS(CHECKSUM(NEWID()))%26+65),
        SomeMoney    = CAST(ABS(CHECKSUM(NEWID()))%10000 /100.0 AS MONEY),
        SomeDate     = CAST(RAND(CHECKSUM(NEWID()))*3653.0+36524.0 AS DATETIME),
        SomeHex12    = RIGHT(NEWID(),12)
		 ,TheBlob		= REPLICATE('Hey, Dracula!! Why do you say blah, blah blah?',10)
   INTO Sandbox2.dbo.millionrowtest
   FROM Master.dbo.SysColumns t1,
        Master.dbo.SysColumns t2
 
--drop table millionrowtest
GO


SELECT DB_NAME(database_id) AS DBName,name AS FileName,size/124.0 AS size_mb
	FROM master.sys.master_files
	WHERE database_id  in (2,DB_ID('Sandbox2'));

I didn’t post the numbers earlier for my files for the Sandbox2 database, but they were small. I had a 2MB data file and a 1mb log file. Now, when I look at the results from that last query that included the file size information, I will see a database that has grown a fair amount.

That should be enough of a change to have triggered something in our extended event session. Let’s take a look at the session. To do that, I am going to pull out a query to help parse the XML and see what has happened. Here is that query:

SELECT
    event_data.value('(event/@name)[1]', 'varchar(50)') AS event_name
	,event_data.value('(event/@timestamp)[1]','varchar(max)') as timestamp
    ,event_data.value('(event/data[@name="count"]/value)[1]', 'bigint') AS DbSizeChangeTo_KB
	,event_data.value('(event/action[@name="sql_text"]/value)[1]', 'varchar(max)') AS sql_text
	,db_name(event_data.value('(event/action[@name="database_id"]/value)[1]', 'int')) AS DBQueryExecutedFrom
	,db_name(event_data.value('(event/data[@name="database_id"]/value)[1]','int')) as AffectedDB
	,event_data.value('(event/action[@name="client_hostname"]/value)[1]', 'varchar(max)') AS ClientHost
	,event_data.value('(event/action[@name="session_id"]/value)[1]', 'varchar(max)') AS session_id
FROM(    
        SELECT CAST(event_data AS xml) AS TargetData
            FROM sys.fn_xe_file_target_read_file('C:\XE\DBFileSizeChange*.xel','C:\XE\DBFileSizeChange*.xem',NULL, NULL)
        
    ) AS evts(event_data)
WHERE event_data.value('(event/@name)[1]', 'varchar(50)') = 'databases_log_file_size_changed'
	or event_data.value('(event/@name)[1]', 'varchar(50)') = 'databases_data_file_size_changed'
	--or event_data.value('(event/@name)[1]', 'varchar(50)') = 'databases_log_growth'
ORDER BY timestamp asc;

Despite having two targets in my session, we will only cover the query that helps parse the data from the asynchronous file target. When I run that query against the files that exist for this session I get a result set of 90 records on my system. That probably seems like a ton of results for such a small increase in the database. As it would happen, I left the growth settings at the default growth increments (don’t do that in a production system) and this means I get a ton of growth activities at very small numbers (1mb for the data file and 10% for the log file). Here is a sample of the output:

You can see how this might be helpful for when a process runs hog wild in the environment. What about going in the other direction though? What if we need to know about when the database is shrunk or when a file is shrunk? Well, let’s do that too. Let’s try the following query.

USE [Sandbox2]
GO
DBCC SHRINKFILE (N'Sandbox2' , 1)
GO

USE [Sandbox2]
GO
DBCC SHRINKFILE (N'Sandbox2_log' , 1)
GO

SELECT DB_NAME(database_id) AS DBName,name AS FileName,size/124.0 AS size_mb
	FROM master.sys.master_files
	WHERE database_id  in (2,DB_ID('Sandbox2'));

GO

Again, I check for a baseline on the files to see if the file sizes changed. In this case, you can run that final query and compare or just trust me on it. Having shrunk both files in the Sandbox2 database, let’s check the XE session data again:

Check that out! We have captured the shrink events too! There are multiple shrink events in this case only because I ran the shrink statements multiple times. This is excellent news for everybody that is trying to keep an eye on these database size changes. You can see in the XE session that I applied the sql_text() action. I have done this so I will be able to see what query caused the growth or shrink event to occur. Even better news is that this event session is perfect for those of you still on SQL 2008.

If you enjoyed this article, check out some of the follow-up articles: Data Growth Audits or Mysterious Growth.

How can there be ghosts in the database?

Why would there be ghosts in the database?

Do they happen because somebody issued a KILL statement?

Let’s address each of those in turn.   A database ghost record is (in a very basic form) one that’s just been deleted in an index on a table . Delete operations don’t actually physically remove records from pages – they only mark them as having been deleted (ghosted). Now why is it done this way?  The answer here is largely performance based.  This is a performance optimization that allows delete operations to complete more quickly. Additionally, it allows the rollback of delete operations to process more quickly.  The rollback processes faster because all that needs to happen is to “flip the flag” for the records as being deleted/ghosted, instead of having to reinsert the deleted records.  That may be a bit over-generalized, but I hope you get the gist.  In short, records are marked as “ghosted” when a delete operation is performed; and to rollback, you simply undo that mark.

Now, what about this KILL statement thing?  The kill statement is pure Halloween fun and does not create ghost records.

Ghost Hunting

Now that we have established the purpose of Ghosts in the database, how do you verify the existence of Ghosts?  In other words, what can we do to prove there really are spectral things in the database?  This is where the fun really begins.  First, we need to get out the equipment and tools (as any good ghost hunter would do) so we can capture these phantasms.  Let’s call the first tool the “trap”.  Here is what you will need for it.

Use master;
Go
IF DB_ID('Sandbox') IS NULL 
BEGIN
	EXECUTE ('CREATE DATABASE Sandbox');
	ALTER DATABASE [Sandbox] SET RECOVERY SIMPLE WITH NO_WAIT
	ALTER DATABASE [Sandbox] MODIFY FILE ( NAME = N'Sandbox', SIZE = 6144000KB , FILEGROWTH = 262144KB )
	ALTER DATABASE [Sandbox] MODIFY FILE ( NAME = N'Sandbox_log', SIZE = 131072KB , FILEGROWTH = 131072KB )
END
GO
Use Sandbox;
Go

SET ANSI_NULLS ON
GO

SET QUOTED_IDENTIFIER ON
GO
IF SCHEMA_ID('Halloween') IS NULL EXECUTE ('CREATE SCHEMA Halloween');
GO

IF OBJECT_ID('Halloween.Ghosts','U') IS NOT NULL
BEGIN
DROP TABLE Halloween.Ghosts
END
GO


DECLARE @BeginDate DATE = '2014-10-01'
		,@EndDate DATE = '2014-10-31'

 SELECT TOP 1000000
        Pinky       = IDENTITY(INT,1,1),
        Blinky		= 'GHOST ITEM ' + CONVERT(VARCHAR(20),ISNULL('', 0)),
        CandyMan	= ABS(CHECKSUM(NEWID()))%50000+1,
        Sadako		= CHAR(ABS(CHECKSUM(NEWID()))%26+65)
                     + CHAR(ABS(CHECKSUM(NEWID()))%26+65),
        Slimer		= DATEADD(DAY,RAND(CHECKSUM(NEWID())) * ( 1 + DATEDIFF(DAY, @EndDate,@BeginDate) ), @EndDate),
        Poltergeist	= RIGHT(NEWID(),12),
        MalcomCrowe	= CAST(ABS(CHECKSUM(NEWID()))%10000 /100.0 AS MONEY),
        TheBlob		= REPLICATE('Hey, Dracula!! Why do you say blah, blah blah?',1000)
   INTO Halloween.Ghosts
   FROM Master.dbo.SysColumns t1,
        Master.dbo.SysColumns t2 --Lack of join criteria makes this a CROSS-JOIN

/* let's get our Blinky updated properly */
Update hg
	Set Blinky = 'GHOST ITEM ' + CONVERT(VARCHAR(20),ISNULL(Pinky, 0))
	From Halloween.Ghosts hg;

--===== A table is not properly formed unless a Primary Key has been assigned
     -- Takes about 1 second to execute.
  ALTER TABLE Halloween.Ghosts
        ADD PRIMARY KEY CLUSTERED (Pinky)

/* Make sure the soylent green blob is BIG */
--UPDATE Halloween.Ghosts
--	SET TheBlob = TheBlob + REPLICATE('Casper is not a real Poltergeist! Frankenstein hates being called Frankenmeanie.',15000);
--GO

/* additional index to show the ghosts */

Create NonClustered Index IX_GhostPinky on Halloween.Ghosts (Pinky)

This trap, err database, can be a bit large.  As currently configured, we will need about 16GB of disk space to support it.  If that is too much, I recommend removing the last column – “TheBlob”.  As you can see, we are setting a rather large trap.  The table we create (Halloween.Ghosts) will receive One Million records.  This is most probably overkill to catch these ghosts, so you can also cut back on the number of records to be affected.

Now, to make sure we have some data and that we can use the table, let’s just run a little test query.

Select top 100 *
	From Halloween.Ghosts

Excellent, we have a good sample of data.

At this point, it is important to note that we have done nothing that will cause database ghost records.  All that has been done is to set the framework so we can see the ghosts.  With the framework in place, let’s try to catch some ghosts.  To do so, we need to try to delete something.  Since we just happen to have had a clerical error in our database, we have 666 prime candidates to try and fix.  We happen to have several records that were supposed to be given a Slimer date of Halloween.  The clerk, being absent minded, thought that Halloween was supposed to be on Oct. 30.  Our business model dictates that the invalid records must be deleted first and then we can try to enter the replacement records.  So, let’s go ahead and try to remove those records.

Before we remove the records though, we need to discuss one important requirement for us to be able to see the ghosts.  Let’s call it spectral vision goggles.  In the database realm, we call it a trace flag.  In order to see the the ghosts on the pages, we need to enable TF 661.  We can do that with the following statement.  There is a serious side effect to this method too – it alters the behavior of the Ecto Containment Unit or automatic ghost cleanup process.  If you enable this, you will need to disable it later and/or manually run a ghost cleanup.

/* tweak the ghost cleanup with an undocumented TF 
Required for DBCC Page results to show which pages have a GHOST entry
Don't try this at home - we are what you call experts (besides this is only a lab box!!) */
DBCC TRACEON (661, -1)
GO

Now that we have the last piece of equipment in place, let’s go ahead and try to delete some records.

Delete top(666)
	From Halloween.Ghosts
	WHERE Slimer = '10/30/2014';

With all of those records deleted (all 666 of them), let’s see what we might have captured.  First, let’s take a look at some index stats.

/* ghost_record_count */
Set transaction Isolation level read uncommitted
SELECT DB_NAME(database_id) AS DBName,schema_name(so.schema_id) + '.' + so.name as ObjName, record_count,index_type_desc
	,ghost_record_count,version_ghost_record_count
FROM sys.dm_db_index_physical_stats(db_id('sandbox'), OBJECT_ID('Halloween.Ghosts'), NULL, NULL , 'DETAILED') ps
	INNER JOIN sys.objects so
		ON ps.object_id = so.object_id
WHERE index_level = 0
	;

If we look at the output of this query, we will see that we did indeed attempt to delete 666 records.  Those records will now display in the ghost_record_count column.  We will also see that, since we had two indexes on the table, there are 666 ghost records marked on each index.

Very cool!  We are definitely on the track to capturing those ghosts.  We have a trail that they exist in the database.  Let’s keep going and see where we can see them.  You should note that there is an additional column in our result set that looks like it might be related to ghost records.  We are going to leave the discovery of version_ghost_record_count as a homework experiment for you to perform.  It is beyond the current scope of this article.

/* Use Fn_dblog to get Pages with Ghost Counts */
Select Operation,Context,[Transaction ID],[Page ID] AS [File:PageIDHex],AllocUnitId,[Slot ID],PartitionId
		,CONVERT(INT,CONVERT(VARBINARY,'0x' + RIGHT([Page ID], 8),1)) AS PageID
	From fn_dblog(null,null) fn
	Where CONTEXT = 'LCX_MARK_AS_GHOST';

Now this is getting exciting.  We have stronger evidence in the log showing that these ghosts are hanging around in the database.  Not only are they hanging around in the database, we can see which pages in the database on which they are trying to hide.

This is really solid information!  fn_dblog is giving us just about everything we need in order to get those ghosts.  It took a little bit of work since the log reports the page number in hex.  Converting that to an integer page number is essential for us to look at the page (besides integer values are easier to interpret for most people).  Now I can take that PageID and pass that number, for any of the records reported by fn_dblog, and pass it into yet another undocumented procedure known as DBCC Page.

When looking to use DBCC page, we can either look at the PFS Page and see more pages that have ghost record counts.  Or we can take the results seen from the fn_dblog output  and then look at the contents of the page and catch those ghosts.  We will take a quick look at the PFS page first.  Then we will take a look at an index page next.  In this database that we have created, the PFS page will show several other pages that have ghost records on them.  Due to the size (over 2 million pages), we only see index pages with ghost records in that result.  If our database were smaller, we would quite possibly see data pages in our first PFS page of the database.  Let’s see a sample from the first PFS in this database.

DBCC PAGE('Sandbox',1,1,3) WITH TABLERESULTS
GO

We can follow that link from this point to page 126.  Page 126 happens to be an index page similar to the following.  There are a couple of indicators that this is an index page.  First being that when we run DBCC Page with a format of 3, we will see two result sets.  The second result set will show statistics and index information.  The second being in the image attached after the query.  We will leave it as an exercise to you to see other ways to demonstrate that this is an index page.

DBCC PAGE('Sandbox',1,126,3) WITH TABLERESULTS
GO

That is great, but we have more ghosts to find.  Let’s look at a ghost on a data page.  Randomly picking a PageID from that list that was output from fn_dblog, let’s see what DBCC Page will provide to us.

DBCC PAGE('Sandbox',1,1522936,3) WITH TABLERESULTS
GO

Conclusion

Well, isn’t that just cool!  We have trapped a bunch of ghosts and were even able to see them.  This has been a fantastic deep dive into the crypts of the database.  This is merely a scratch on the surface though.  We hope this will encourage you to explore a bit and at least try one of the homework assignments we left behind in this article.

With all of that, we have a bit of cleanup to do.  The cleanup comes in one of two methods.  Method one involves manual labor.  Method two involves our friendly little trace flag we already used.  Since most DBAs prefer the automated mechanisms over manual, let’s just discuss method two for now.  It is extremely effortless.

DBCC TRACEOFF (661, -1)
GO

That will put the system back to the way it was when we started (and of course we trust that nobody did this on their prod box).

This has been one of a few articles about ghosts in the database. You can check out some of the others here and here.

The presence of XE in ADS comes via an extension and comes with a few other caveats. I will explore the extension for XE available in ADS in this article and discuss some of the caveats. As you read the article, it might be helpful to go ahead and download ADS if you do not already have it.

History

Roughly 10 years ago Microsoft felt it necessary to introduce a cool tool called Extended Events. Soon after they decided deprecate the features called “Profiler” and “Trace”. Unfortunately the page with the deprecation announcement is no longer available, but some evidence of how long it has been deprecated is available here.

The deprecation announcement remains in effect (and online) for all versions since SQL Server 2012. It just may be difficult to find the 2012 announcement as we roll into newer releases of SQL Server.

Now, we have XE Profiler (or XEvent profiler depending on your release of SSMS – read more here). Profiler is deprecated and now we have some confusion in SSMS as to what is Profiler since we are now using that term with the “XE Profiler” feature.

Now enter ADS. XE is not included with ADS by default. You have to install an extension to gain access to the feature. So, the first thing you will need to do is visit the extensions node and then search for “SQL Server Profiler”. I can hear you right now. It isn’t even using any part of the real feature name anymore – they are just calling it the same exact thing as the deprecated feature.

And yes, my heart breaks a little more every time I see “SQL Server Profiler”. We have been teaching Database Professionals for years to use Extended Events and not SQL Server Profiler. And they have been adopting that change in rather large numbers. This just seems like it will cause so much more confusion. Nevertheless, once you have selected the extension, look to the right hand side and you will see a screen similar to this.

After installing the extension and the reloading ADS we are ready to start using this extension. For the remainder of this article, I will just refer to it as “XE extension”.

Where did it go?

After the reload of ADS, finding the XE extension is not really that easy. If you read the info page where you clicked install, there is some info there on how to access it. In short, on a Windows machine Alt-P will be your friend. Where you use that key combination is not your friend though. If you are in a script for instance and hit that key combo, no connection will be made to your server – even if your script is connected.

In addition to that not-so-obvious message, there is a more obvious message box that pops up in the bottom right corner letting you know a connection could not be established. Unfortunately, the problem can’t be resolved from this screen. Just close the tab and try again from the instance connection as shown here.

While a bit annoying, I can manage with that little caveat – just as long as I remember between uses what I did. After the XE extension is open, you should see a screen similar to the following.

In the preceding picture, I show three indicators of a connection being established. The top right corner in the example is difficult to tell that there is a server name there but it is. In my example I am just using the shorthand notation to connect to my server or “.” (a dot) which connects me to the localhost instance.

The top left indicator is a drop down list of all XE sessions I have on the server.

Beyond that, I don’t find it terribly useful. I can’t edit a session or script it from this tool yet.

Managing a session doesn’t appear to be possible at this point, so let’s try to create a new session and see what happens.

Sweet! The create session does something useful. I can create a new session. Unfortunately, all I can do is use one of three templates similar to the XE Profiler tool in SSMS. The full feature XE GUI tool has a much more complete list of templates and possibilities ever since SQL Server 2012. I documented that in this article.

Unfortunately, I have no use for the three default templates. So, for me, this tool drives me back to needing to use TSQL to create my sessions if I want to use ADS. Here is the big takeaway from that statement. The use of TSQL has been the biggest detractor for most Data Professionals when using XE. They want a full featured GUI. So, you are using ADS and must create an XE session, you will need to pull out your TSQL skills and probably need to pull down some of my helper scripts to get you going. The alternative would be to use SSMS where there is a full featured GUI that is more powerful than you might think.

Conclusion

There surely will continue to be more development around this idea of an XE style profiler. More development generally means that the product will mature and get better over time. This article shows how there is more being added to the feature to try and give you better control over the tool. We love control so the addition of these options is actually a good thing. Is it enough to sway me away from using the already established, more mature, and high performing tools that have been there for several generations? Nope! I will continue to use TSQL and the GUI tools available for XE that predated the XEvent Profiler.

Some say that data professionals really want the “Profiler” tool. In my opinion, that is certainly not the majority and now calling XE by the name “Profiler” is going to cause confusion at the least. Some say that maybe this tool needs to integrate a way to shred XML faster. To that, I say there are methods already available for that such as Powershell, the live data viewer, the Target Data viewer, or even my tools I have provided in the 60 day series.

Knowing just how long an event takes is a common requirement when troubleshooting. Sometimes, figuring out the unit of time is a bit troublesome. Is it milliseconds, seconds or microseconds?

It is really easy to mistakenly use milliseconds when microseconds is required. It is also really easy to forget that one event is measured in seconds while another might be measured in milliseconds. To add to the confusion, what if the unit of measure changes between one version of SQL Server and the next?

While none could think it should be easy and consistent to figure out time, sometimes it just takes a little more effort. The same unit of time just isn’t applicable for every type of event. That is not just true in SQL Server but in life in general. You wouldn’t want to use hours when timing muzzle velocity, but hours could be entirely applicable to a surgery or training seminar.

Where does that leave us SQL Geeks when looking at timing of internal events inside of SQL Server? Well, we either need to do a little digging or we could simply read the rest of this article to find a simple script that can do the bulk of the work for us.

When dealing with events inside of SQL Server, the tool of choice to use is Extended Events (XE). If you are unfamiliar with XE, I really encourage you to take a look at these resources to become more familiar with the tool.

Time

None of us want to translate or interpret time incorrectly. Imagine the CIO looming over your shoulder asking how long before the database is back online. If you miscalculate the time and tell him 30 seconds when it really is 5 hours, the level of frustration and anger probably becomes exponentially worse.

While that may seem like a bit of an exaggeration, it’s not an experience any DBA likes to encounter. On a slightly smaller scale, when a developer asks how long a piece of code is running in production, you do want to be as accurate as possible. If the code takes 5 minutes to execute, the developer needs to know it was 5 minutes and not a miscalculated 3.9 seconds. 3.9 seconds may be entirely within the realm of acceptable for the project and thus be dismissed by the development team.

While trapping the precise time and calculating it may seem trivial, it is important (as previously mentioned) to use the correct time unit. There are many events within XE that provide a time unit of measure. The unit of measure is different through most of the events so a different calculation may be required depending on what you are troubleshooting at the moment.

Thankfully, there is some means to figure out if the time measurement is in seconds, milliseconds, microseconds or something different. We just need to pull it all out from the metadata views. Here is how we get to that data.

DECLARE @ServerMajorVersion DECIMAL(4, 2)
SELECT @ServerMajorVersion = CONVERT(DECIMAL(4, 2), PARSENAME(dt.fqn, 4) + '.'
		+ PARSENAME(dt.fqn, 3))
	FROM ( SELECT CONVERT(VARCHAR(20), SERVERPROPERTY('ProductVersion'))
			) dt ( fqn );

SELECT CASE WHEN @ServerMajorVersion = 15.00 THEN 'SQL Server 2019'
			WHEN @ServerMajorVersion = 14.00 THEN 'SQL Server 2017'
			WHEN @ServerMajorVersion = 13.00 THEN 'SQL Server 2016'
			WHEN @ServerMajorVersion = 12.00 THEN 'SQL Server 2014'
			WHEN @ServerMajorVersion = 11.00 THEN 'SQL Server 2012'
		END AS SQLVer
	, @ServerMajorVersion AS SvrVersion
	, xp.name package_name
	, xo.name event_name
	, xoc.name event_field
	, DurationUnit = 
		CASE WHEN xoc.description LIKE '%milliseconds%' 
			THEN 'MilliSeconds'
			WHEN xoc.description LIKE '%microseconds%' 
			THEN 'MicroSeconds'
			WHEN xoc.description LIKE '%seconds%' 
			THEN 'Seconds'
			ELSE NULL
		END
	, xoc.type_name AS DataType
	, xoc.description AS DurationDesc
	, xo.description AS PayloadDesc
	, ch.Channel
	, ca.map_value AS SearchKeyword
FROM sys.dm_xe_objects xo
		INNER JOIN sys.dm_xe_packages xp
			ON xo.package_guid = xp.guid
		INNER JOIN sys.dm_xe_object_columns xoc
			ON xo.name = xoc.object_name
		OUTER APPLY (SELECT TOP 1 mv.map_value
						FROM sys.dm_xe_object_columns occ
						INNER JOIN sys.dm_xe_map_values mv
							ON occ.type_name = mv.name
							AND occ.column_value = mv.map_key
						WHERE occ.name = 'KEYWORD'
							AND occ.object_name = xoc.object_name) ca
		INNER JOIN (SELECT c.object_name AS EventName,c.object_package_guid AS PkgGuid, v.map_value AS Channel
					FROM sys.dm_xe_object_columns c
						INNER JOIN sys.dm_xe_map_values v
							ON c.type_name = v.name
							AND c.column_value = CAST(v.map_key AS NVARCHAR)
					WHERE c.name = 'channel') ch
			ON ch.EventName = xoc.object_name
			AND ch.PkgGuid = xoc.object_package_guid
WHERE xo.object_type = 'event'
	AND xoc.name = 'duration'
ORDER BY xp.name,xo.name;

Holy crap! That doesn’t look easy. That is a ton more code than you might have expected. True. However, I like to have as much information as possible at my finger tips. In this case, I want to know the SQL Server version, channels, and search terms (keywords) related to the event.

Why have the extra data? When troubleshooting, it is nice to know if there are other events that might be related in nature that could shed a bit more light on the problem from a different angle. The use of keywords (think google search) and channels is perfect for helping me find those other events.

Here is what a snippet of that data might look like.

With this snippet, I can see there are multiple different units of measure but there are also multiple different keywords. These keywords can be essential to the troubleshooting process.

You will also see from that image that there are some events that don’t define the time unit very clearly. In fact it is just a null value for the description. That is a bit of a problem – but significantly less troublesome than not knowing the unit of measure for any of the events.

Conclusion

Figuring out the correct unit of time measurement can mean the difference between accurate troubleshooting or leaping down the wrong path. Figuring out the time units is made easier when you are able to query the metadata efficiently and have all of the pertinent details at your fingertips.

Extended Events is a powerful tool to help in troubleshooting and tuning your environment. I recommend investing a little time in reading the 60 day series about Extended Events. This is not a short series but is designed to provide an array of topics to help learn the tool over time. Don’t forget to go back and read the companion article showing how to audit these events via the default trace.

I recently had the mis-adventure of working through some ISE crashes after bringing some systems up to PoSh 5.1 that were either PoSh 2.0 or 3.0. It’s not a very fun situation to run a WMI update and then run into a crash of any type when testing if it worked. Your first thought is something terrible has happened.

As it stands, the problem is more of a nuisance than a critical failure. That said, it is enough of a problem that anyone who uses the ISE or .Net applications may experience a slight cardiac event.

Fonts

As you work to quickly recover from your missed heart beat, you start digging through logs and then hitting good old trusty google.

Diving through the logs, you might just happen across an error similar to the following:

Problem signature: Problem Event Name: PowerShell NameOfExe: PowerShell_ISE.exe FileVersionOfSystemManagementAutomation: 6.1.7600.16385 InnermostExceptionType: System.Xml.XmlException OutermostExceptionType: System.Reflection.TargetInvocation
DeepestPowerShellFrame: indows.PowerShell.GuiExe.Internal.GPowerShell.Main DeepestFrame: indows.PowerShell.GuiExe.Internal.GPowerShell.Main ThreadName: unknown.

Maybe the first error you encounter might look like this one instead:

System.TypeInitializationException

“FileFormatException: No FontFamily element found in FontFamilyCollection
that matches current OS or greater: Win7SP1”.

Inner exception originates from: CompositeFontParser

Either way, the error shoots us back to the same fundamental problem. The ISE won’t load, you get an error message and you can’t confirm that the WMI patch was applied properly.

As you work your fingers faster and faster through the pages on google, you discover that this problem is caused more explicitly by a patch for the .Net framework and not necessarily the work to upgrade your PoSh version. It only waited to manifest itself after the upgrade.

That’s gravy and all, but how does one fix the problem? For me, the quickest and most reliable fix was to simply jump straight to the root of the problem – fonts. The ISE is a WPF application and it also requires a fallback font (if a character isn’t present in your font set, then the app chooses a substitute from the fallback font – or something like that).

The fix is extremely simple and really underscores why this is merely a nuisance issue and not a critical problem. Thus it shouldn’t cause any sort of sinking internal feelings of any sort. There are a few plausible fixes floating around out there. I recommend just doing a manual font replacement. It is all but three simple steps:

1. Download GlobalUserInterface.CompositeFont
2. XCOPY the font to %windir%\Microsoft.NET\Framework\v4.0.30319\WPF\Fonts
3. XCOPY the font to %windir%\Microsoft.NET\Framework64\v4.0.30319\WPF\Fonts

After you have copied the font to those two directories, then all that is needed to be done is launch the ISE. I ran into the same problem on three or four servers and the fix took no more than 5 minutes on each of the servers.

Conclusion

I previously mentioned that I have been working more and more with PoSh to try and improve my skillset there. This is one of those very low-level trinkets that I ran into as I have been working to hone my skills in that tech. For other, possibly, interesting articles about my experiences with PowerShell, you can check out these articles.

Given this job is tightly related to the system_health black box sessions (sp_server_diagnostics and system_health xe session), I recommend fixing the job. In addition, I also recommend reading the following series about XE and some of those black box recorder sessions – here.

The other day, I shared an article showing how to audit database offline events via the default trace. Today, I will show an easier method to both audit and monitor for offline events. What is the difference between audit and monitor? It largely depends on your implementation, but I generally consider an audit as something you do after the fact. Monitor is a little more proactive.

Hopefully, a database being taken offline is a known event and not a surprise. Occasionally there are gremlins, in the form of users with too many permissions, that tend to do very strange things to databases and database servers.

Having read the previous article, you already know one method to try and find these database offline anomalies. That method may not be the most sleek solution nor most reliable given the possibility that events can quickly roll out of your default trace files. The better more reliable method is use Extended Events (XE) to monitor explicitly for those types of events. If you are unfamiliar with XE, I really encourage you to take a look at these resources to become more familiar with the tool.

Monitor

Beyond the power of XE to be able to better diagnose problems and trace events in your server, there is the ability to monitor for specific events as well. I won’t go into details about how to monitor with XE until a later article, but suffice it say I can monitor for Events to occur and immediately alert necessary parties to get more immediate action. This is quite some power for a built in tool and it is better than event notifications or agent alerts when it comes to ease of use and reliability.

As I look to monitor for these odd unplanned database offline events, I have several events within XE that can provide the requisite information: sqlserver.object_altered, sqlserver.database_started, sqlserver.database_stopped, and sqlserver.errorlog_written. I can hear you asking already “Wait, this seems to be a bit like a drill sergeant – very overbearing!”

Yes, it is probably a bit excessive for this session. However, I prefer to be comprehensive and the ability to link events together so I can better understand if it is a single one-off or if there is a bigger problem with the entire instance. Databases being stopped, started or set to offline should be rare and far between really. With that rarity in mind, the session should be relatively quiet.

IF EXISTS ( SELECT *
				FROM sys.server_event_sessions
				WHERE name = 'DBStateChange' )
	DROP EVENT SESSION DBStateChange 
    ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE';
GO

CREATE EVENT SESSION DBStateChange ON SERVER
ADD EVENT sqlserver.object_altered ( SET collect_database_name = ( 1 )
	ACTION ( sqlserver.sql_text,sqlserver.nt_username,sqlserver.server_principal_name,sqlserver.client_hostname,
	package0.collect_system_time,package0.event_sequence,sqlserver.session_id ) 
	WHERE object_type = 'DATABASE' --16964
		AND (sql_text LIKE '%ONLINE%'
			OR sql_text LIKE '%OFFLINE%')
	),
ADD EVENT sqlserver.database_started(
ACTION ( sqlserver.sql_text,sqlserver.nt_username,sqlserver.server_principal_name,sqlserver.client_hostname,
	package0.collect_system_time,package0.event_sequence,sqlserver.session_id ) 
	--WHERE object_type = 'DATABASE' --16964
	),
ADD EVENT sqlserver.database_stopped(
ACTION ( sqlserver.sql_text,sqlserver.nt_username,sqlserver.server_principal_name,sqlserver.client_hostname,
	package0.collect_system_time,package0.event_sequence,sqlserver.session_id ) 
	--WHERE object_type = 'DATABASE' --16964
	),
ADD EVENT sqlserver.errorlog_written(
	ACTION ( sqlserver.sql_text,sqlserver.nt_username,sqlserver.server_principal_name,sqlserver.client_hostname,
	package0.collect_system_time,package0.event_sequence,sqlserver.session_id ) 
	WHERE (sql_text LIKE '%ONLINE%'
			OR sql_text LIKE '%OFFLINE%'))
ADD TARGET package0.event_file ( SET filename = N'C:\Database\XE\DBStateChange.xel' )
WITH (STARTUP_STATE = ON
	,TRACK_CAUSALITY = ON);

/* start the session */
ALTER EVENT SESSION DBStateChange 
ON SERVER 
STATE = START;
GO

If I have that session running and then take a database online/offline or vise versa, I will see something very similar to this output.

Starting from the bottom and working my way up, I can see that a command was issues to bring the s database ONLINE. The very first thing that occurs is the request is written to the error log. Then I see that the database is in the stopped state. Next a message that the database is starting up (because it was stopped). Then we see two events for object_altered (similar to the default trace) due to the begin and commit phases of that transaction.

After that database was brought ONLINE, you can see that I immediately took s back offline – starting with the errorlog event, then a stopped event and the object_altered begin and commit events.

Capturing each of the events I noted previously, not only gives me a complete picture of the event, it also can help me to identify if something happens in between the various “expected” events. If I use this session in my monitoring setup, then I can be quickly alerted to problems with a database as well as have the archive of the events to go back in time and AUDIT or troubleshoot the event of a database being offline or unable to come online.

With this XE Session running, I can be more confident that I have trapped and correlated the correct events in each of the sources. Using the default trace method, I have to make some highly likely correlations but there is still some “magic” involved. With the XE session, you will be far less likely to see any of those events roll out of the log as well. I can’t underscore the importance of that fact enough. The data will be there when you need it!

Conclusion

We all aspire to having a perfect database environment where nothing surprising or unexpected happens. Unfortunately, that is the desire of dreams and fairy tales. The unexpected will happen. A database can unexpectedly be taken offline. Are you prepared to address the problem fully to the CTO should it happen? This XE session can help you with that.

Extended Events is a powerful tool to help in troubleshooting and tuning your environment. I recommend investing a little time in reading the 60 day series about Extended Events. This is not a short series but is designed to provide an array of topics to help learn the tool over time. Don’t forget to go back and read the companion article showing how to audit these events via the default trace.

Unfortunately, when it comes to SQL Server, the symptoms are not as readily visible as aging lines would be. In SQL Server, the symptoms may be as random and difficult to recognize as a stored procedure just suddenly stops working.

What makes this even more difficult is that the stored procedure (that is no longer working) may appear to be entirely unchanged. These failures can occur when other changes have occurred to the system as well as when no change has occurred. Imagine the joys you could experience while trying to troubleshoot this kind of problem.

If you were keen, you would probably implement any number of Extended Event Sessions to help troubleshoot the issue. If you are unfamiliar, you might want a refresher course on how to setup an XE session which you could read from one of these articles, here and here.

If you are curious, there are more XE articles on my blog – here.

Metadata

There are many possible causes for the metadata to become fouled up inside the SQL modules on the database server. Invariably, according to the interested parties, nothing has changed in months! Right or wrong, there is still a problem to fix. Fortunately, the problem is easy enough to fix.

Sure, all of the modules could be re-deployed from source control. But, given that the modules don’t to appear to have lost any of the code within them, re-deploying code seems to be a bit of overkill. We don’t need to go that far. We can simply run sp_refreshsqlmodule for the stored modules (procs, triggers, functions, views etc). This would be pretty simple with the following script:

USE AdventureWorks2014;
GO

SELECT o.name AS ObjName, sm.definition, o.type_desc
	,'EXECUTE sys.sp_refreshsqlmodule ''['+ SCHEMA_NAME(o.schema_id) + '].['  + o.name + ']'';' AS RefreshStmt
	, CASE	WHEN o.type IN ( 'TR','P', 'FN', 'FT', 'TF', 'PC', 'FS', 'AF', 'IF' )
				THEN 1
				WHEN o.type = 'V' THEN 2
				WHEN o.type IN ( 'F', 'D' ) THEN 3
				WHEN o.type = 'SO' THEN 4
				WHEN o.type = 'UQ' THEN 5
				WHEN o.type = 'PG' THEN 6
			END AS ProcessOrder
		FROM sys.all_sql_modules sm
		INNER JOIN sys.objects o
			ON sm.object_id = o.object_id
	ORDER BY ProcessOrder DESC,o.type_desc, o.name
		;

In this script, I am just going to refresh all modules in the database (Adventureworks2014 in this case). I could modify the script to trim it down to a specific module or set of modules. Instead, I leave that as homework for you.

That said, I do have a slightly different alternative that looks for any tables changed on a specific date. After finding those changed tables, then I update all modules related to the changed table(s). Here is how that would look:

USE AdventureWorks2014;
GO

DECLARE @ChangeDate DATE = '2014-07-17 16:11:39.583';

SELECT DISTINCT o.name AS ObjName, sm.definition, o.type_desc
	,'EXECUTE sys.sp_refreshsqlmodule ''['+ SCHEMA_NAME(o.schema_id) + '].['  + o.name + ']'';' AS RefreshStmt
	, CASE	WHEN o.type IN ( 'TR','P', 'FN', 'FT', 'TF', 'PC', 'FS', 'AF', 'IF' )
				THEN 1
				WHEN o.type = 'V' THEN 2
				WHEN o.type IN ( 'F', 'D' ) THEN 3
				WHEN o.type = 'SO' THEN 4
				WHEN o.type = 'UQ' THEN 5
				WHEN o.type = 'PG' THEN 6
			END AS ProcessOrder
	FROM sys.all_sql_modules sm
		INNER JOIN sys.objects o
			ON sm.object_id = o.object_id
		INNER JOIN sys.sql_expression_dependencies ed
			ON o.object_id = ed.referencing_id
	WHERE ed.referenced_id IN (
		SELECT object_id
			FROM sys.tables t
			WHERE CONVERT(DATE,t.modify_date) = @ChangeDate)
	ORDER BY ProcessOrder DESC,o.type_desc, o.name
		;

From here, suppose you want to check for a range of dates where tables were modified. Again, I will leave that as homework for you.

After executing these scripts, I will have an output similar to the following:

This leaves you with one more step to perform – copy the values from the RefreshStmt column to a new query window and execute the statements.

The Wrap

In this article I showed a very simple solution to a problem that plagues some environments – out of date metadata. Forcing a module refresh will often resolve these types of issues.

With such a beautifully simple solution, I could have grouped this article into my “Back to Basics” series, but I did not. That said, there are some pretty interesting articles in the series including (but not limited to): Backups, backup history and user logins. Check them out!

I recently had a client call me out of the blue because he happened to be getting an error while trying to add a user to a database role. The error he was getting was “Cannot use the special principal ‘dbo’.”

This error has probably cropped up on me more than a few times. And on more than a few occasions, I have forgotten about the previous experiences. Some of that is because the fix is rather easy and after a few times seeing it, muscle memory takes over and you just fix it without thinking about it too much.

Until you get to that muscle memory moment though, you may flounder a bit trying this and that and failing then proceeding on to a level of frustration that has you losing precious hair.

As luck would have it, this is an article about security and principals and is similar in nature to some other articles I wrote about some fundamental misconceptions about permissions here and here.

I do hope that there is something you will be able to learn from this basics article. If you are curious, there are more basics articles on my blog – here.

Meet Prince Apole and Rolle…

Adding a user to the db_datareader database fixed role is a pretty simple task. Most of us can likely do that in our sleep. Even using the GUI is usually pretty reliable to do that. Every now again though, somebody has decided to get tricky on us. Maybe a mistake was made somewhere in a setting on the server and nobody has caught it because nothing was “broken” – until it was.

In the aforementioned case, I was asked to help resolve the issue and I found that there was a problem in how the database owner was set. Not only was it a problem in the current database but in 12 other databases on the same server. The systems admin was at wits end. He was dealing with something that was just not in his knowledge-base yet. I remember being in the same boat – so no big deal there. We talked about some of the things he had tried and how none of it was working. I am going to recreate the same basic scenario along with some of the attempted fixes in this article.

First, we need to create a database (best to break a database designed to be broken instead of an existing one used for something else already).

CREATE DATABASE [DummyDB]
GO
USE [master]
GO
ALTER DATABASE [DummyDB] SET RECOVERY SIMPLE WITH NO_WAIT
GO

USE [master]
GO
CREATE LOGIN [mydomain\svc_dummy] FROM WINDOWS WITH DEFAULT_DATABASE=[master]
GO

That is pretty straight forward – the database will be created with the data files in the default directories on your SQL Server instance. In addition, a login called mydomain\svc_dummy will be created as a windows login.

Now let’s try to set the owner of the database and then add the svc_dummy account to the datareader role.

USE [DummyDB]
GO
/*set db owner to the dummy account */
ALTER AUTHORIZATION ON DATABASE::[DummyDB] TO [saeopssql01\svc_dummy]
GO

USE [DummyDB]
GO
ALTER ROLE [db_datareader] ADD MEMBER [dbo]
GO

There is a point of interest here. I said I was going to add svc_dummy to the datareader role – not dbo. Well, I set the database owner in the preceding step to svc_dummy so it basically became dbo. When I try to perform the role addition in the GUI and then script the change, this is the script that is produced. I will show why in a few moments.

The execution of the second part of the script results in the following:

Msg 15405, Level 16, State 1, Line 18

Cannot use the special principal ‘dbo’.

That is obviously not going to work. Let’s try fixing the script and add the svc_dummy principal instead of dbo.

USE [DummyDB]
GO
ALTER ROLE [db_datareader] ADD MEMBER [mydomain\svc_dummy]
GO

Unfortunately, this results in the following:

Msg 15151, Level 16, State 1, Line 22

Cannot add the principal ‘mydomain\svc_dummy’, because it does not exist or you do not have permission.

Well, maybe the problem is because the user doesn’t exist then? Let’s try to create the user and see what happens.

USE [DummyDB]
GO
CREATE USER [mydomain\svc_dummy] FOR LOGIN [mydomain\svc_dummy]
GO

Now we should see this message:

Msg 15063, Level 16, State 1, Line 32

The login already has an account under a different user name.

Oy vey. We seem to be going in circles. Nothing is working. The user is there but not really there. Let’s try to drop the user and just try to clean things up and start over.

USE [DummyDB]
GO
/* GUI Generated - the user dbo is mapped to the dummy login */
DROP USER [dbo]
GO

I hope you see the problem with this one. Trying to drop dbo. I dunno but we should see an error here – and we do get an error.

Msg 15150, Level 16, State 1, Line 27

Cannot drop the user ‘dbo’.

Let’s fix the user then and try to drop the svc_dummy user instead of dbo.

USE [DummyDB]
GO
/* let's fix it to the correct user */
DROP USER [mydomain\svc_dummy]
GO

Which in turn generates yet another error.

Msg 15151, Level 16, State 1, Line 52

Cannot drop the user ‘mydomain\svc_dummy’, because it does not exist or you do not have permission.

If I can’t resolve the problem by changing the user in the database, maybe I can just blow it out of the water by dropping the server login.

USE [master]
GO
DROP LOGIN [mydomain\svc_dummy]
GO

Yet another failure message will ensue. This time the message is:

Msg 15174, Level 16, State 1, Line 55

Login ‘mydomain\svc_dummy’ owns one or more database(s). Change the owner of the database(s) before dropping the login.

So far we have been able to skirt around the problem and generate six different error messages. The last one kind of gives us the best information on what we could do to resolve the issue. The login owns a database and therefore, we need to undo that ownership. Before we do that, let’s take a look at the database principal ‘dbo’.

We already know that svc_dummy is mapped to a user in the DummyDB database. We also know that we cannot add the svc_dummy user because of that prior mapping. We have also learned that when scripting the permissions change from the gui on the svc_dummy login and then generate the script it scripts out the user ‘dbo’. Due to this, let’s look in the sys.database_principals view at the dbo user and see what it tells us.

select dp.name, dp.type_desc, dp.owning_principal_id, dp.authentication_type_desc
	From sys.database_principals dp
where name = 'dbo';

See how the dbo database principal says it is mapped to a windows account type? With this in mind, let’s join to the sys.server_principals and see what windows account is mapped to the dbo database user.

select dp.name as DBUser, sp.name as ServerLogin, dp.type_desc, dp.owning_principal_id, dp.authentication_type_desc
	From sys.database_principals dp
		inner join sys.server_principals sp
			on dp.sid = sp.sid
where dp.name = 'dbo';

Now we see a bit more clearly. Combined with the error messages and the principal information for both the login and the user, we have a better view of the puzzle now. Changing the database owner indeed mapped the windows account to dbo for us and is now restricting us to certain activities when trying to manage permissions for the windows login in the database. From here, we can easily fix the issue by changing the database owner, creating a user mapped to the windows login and then adding that principal to the datareader role.

--fix
USE [DummyDB]
GO
/*set db owner to the sa account */
ALTER AUTHORIZATION ON DATABASE::[DummyDB] TO [sa]
GO
USE [DummyDB]
GO
CREATE USER [mydomain\svc_dummy] FOR LOGIN [mydomain\svc_dummy]
GO
USE [DummyDB]
GO
ALTER ROLE [db_datareader] ADD MEMBER [mydomain\svc_dummy]
GO

And if we run that script for svc_dummy we will see a successful execution as shown here.

The Wrap

In this article I took a rather long route to a simple fix. It’s easy to try each of the steps I showed in this article thinking it will help. It isn’t illogical to try some of those steps. They just don’t work unfortunately. In the end, getting to know the settings in the database and what the errors are really trying to get at is most helpful. Sometimes, it just takes a few more steps to get to the real meaning of the error.

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

It is not unusual to find a server where some random user created a bunch of jobs to be run by SQL Agent. Sometimes, the user creating the job(s) sets themself as the owner of the job. There are certain cases where this behavior is hard to avoid like when creating a maintenance plan.

And of course, there are times when the user just doesn’t know any better. There is of course, the rare occasion when setting the job owner to be ones self makes the most sense -but that is few and far between in the grand scheme. Usually, you will want a non-expiring account such as a service account or a principal without “logon” permissions to be the owner.

The primary reason being simple – humans have an expiration date for every job they will ever have. When that expiration occurs, you may end up with any number of unwanted side effects. Unwanted side effects is exactly what we try to avoid in our jobs run via SQL Agent.

No Expiration Date

There are two basic means to change the owner of every job on your server. Either you open each job one by one and set the owner to an acceptable principal. This method is rather tedious and you will be fighting off the boredom if you have a few hundred jobs on the server. Or, the alternative, change the job owners group by group (set-based theory). This second method can be far less tedious and far more efficient. The second method is by far my preferred method. Let’s take a look at how to make all of these changes in groups.

USE msdb;
GO

DECLARE @OwnerChangeFrom VARCHAR(256) = 'MyDomain\Gomer.Clown'
	, @OwnerChangeTo VARCHAR(256) = 'sa'
	, @OwnerSidChangeTo VARBINARY(85);

SELECT	@OwnerSidChangeTo = sp.sid
	FROM sys.server_principals sp
	WHERE sp.name = @OwnerChangeTo;

BEGIN TRAN;
	SELECT	j.name AS JobName
			, sc.name AS CategoryName
			, ISNULL(sp.name, SUSER_SNAME(j.owner_sid)) AS OwnerName
			, j.owner_sid
			, j.date_created
		FROM	dbo.sysjobs j
				LEFT OUTER JOIN sys.server_principals sp
					ON j.owner_sid = sp.sid
				INNER JOIN syscategories sc
					ON j.category_id = sc.category_id
		WHERE j.owner_sid = SUSER_SID(@OwnerChangeFrom);

	UPDATE	j
		SET owner_sid = @OwnerSidChangeTo
		FROM	dbo.sysjobs j
				LEFT OUTER JOIN sys.server_principals sp
					ON j.owner_sid = sp.sid
				INNER JOIN syscategories sc
					ON j.category_id = sc.category_id
		WHERE sp.name = @OwnerChangeFrom
			--OR sp.name IS NULL
			OR j.owner_sid = SUSER_SID(@OwnerChangeFrom);

	SELECT	j.name AS JobName
			, sc.name AS CategoryName
			, sp.name AS OwnerName
			, j.owner_sid
			, j.date_created
		FROM	dbo.sysjobs j
				INNER JOIN sys.server_principals sp
					ON j.owner_sid = sp.sid
				INNER JOIN syscategories sc
					ON j.category_id = sc.category_id;

--ROLLBACK TRANSACTION;
--COMMIT TRANSACTION;

There are three basic sections to this script. First I fetch what should be changed, then I make the change, and lastly I verify the change. If the change doesn’t look right, then I can rollback the change. If the change is what I expected, then I can commit the change. Those are the broad strokes.

At a more detailed glimpse, I have setup a few variables to compare what I want to change, what the new job owner should be and then I fetch the sid of that new job owner. In my example, I am setting everything to ‘sa’. Why? Because it is easy for the sake of the example in the article – nothing more!

Since sometimes the owner of the job may only have access to the SQL instance via a Domain Group, I also take advantage of a couple of functions to double check that it is the correct account. These functions I am using are SUSER_SID() and SUSER_SNAME().

When all is done as I am expecting, then I should see something similar to the following.

Since the change is what I expect, then at this point I would proceed with the commit transaction statement.

The Wrap

As you can see, making job ownership changes at group scale instead of one by one is pretty easy. This only takes a matter of seconds to run against hundreds of jobs. That same kind of task done one at a time could easily take more than 40 minutes. I am not sure I want to spend that much time on such an innocuous task. I hope you are now able to use what you have learned to improve your skills and become a rock-star DBA. ENJOY!

If you feel the need to read more about single-user mode, here is an article and another on the topic.

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

One of the tasks I find myself doing on a fairly regular basis is running SSMS as a different Windows User. The two biggest use cases for this are: a) to test an account to prove that it is working (or not) and has the appropriate level of access, and b) to use SSMS to connect to a Domain SQL Server from a computer in a different domain (or not on the domain).

In addition to needing to do these tasks for myself, I find that I need to show somebody else how to do the same thing on a fairly consistent basis. Considering the finite keystrokes we all have (which I referenced here), it is time for me to “document” how to do this task.

I will cover two really easy and quick methods to perform this task. One from a command line and the other from the GUI. Both methods will involve a variation of the runas utility.

RUNAS

Let’s start with the easiest of the two methods. In this case, you will need to test windows account (let’s call it a domain account) from a computer which is on the same domain. This requirement allows us to take advantage of the shortcuts from within the GUI to access the runas utility.

To access the runas from Windows, one will first locate the icon for SSMS from the Start Menu, then right click that icon as shown here.

After right clicking the icon, you will see a menu pop up on the screen. Select “Run as different user” from that menu. Once you have selected the appropriate “run as” option, a login prompt will appear as shown here.

Enter the appropriate credentials at the prompt and then SSMS will launch. In this case, I may want to test the account myidomain\domain.useracc. So, I merely need to enter the domain credentials for that account. A caveat here is that the account you are testing will need to have the necessary permissions to “logon” to the workstation in order to launch the app – unlike the second method.

CMD Line

This second method has a few advantages over the GUI method with the biggest advantage being that you can use this method from any machine on the domain or even a machine not joined to the domain (so long as you have the ability to authenticate to the domain). And of course the additional advantage that the account you are testing does not require “logon” permissions on the machine you are using.

Let’s start with the basic command.

runas.exe /netonly /user:myidomain\domain.useracc "Ssms.exe"

I can run that from a command line, or I can throw that into a desktop shortcut (the shortcut method is much more convenient). After I hit “enter” from the command line, I am prompted for a password for the account to be used for that session. Here’s an example of how that would look.

You won’t be able to see the password being typed (don’t fat finger the password ), but after you enter it successfully and press “enter” then you will see SSMS start to launch. After a successful SSMS launch, you should see something similar to the following:

I have a few things highlighted here of interest. First, in the red box, you will note that the user shown as connected to the server is my “local” test box account instead of the domain account. However, if I verify the authenticated account, I can see that the domain account is indeed accessing the SomeServer SQL Server (as demonstrated by the green box on the right).

The Wrap

Sometimes what may be ridiculously easy for some of us may be mind-blowing to others. Sometimes we may use what we think are common terms only to see eyes start to glaze over and roll to the backs of peoples heads. This just so happens to be one of those cases where launching an app as a different principal may be entirely new to the intended audience. In that vein, it is worthwhile to take a step back and “document” how the task can be accomplished.

Runas should be a very common tool in the toolbox of all IT professionals – not just Data Professionals. Learning how to test different accounts is essential to being an effective and efficient professional that can provide solid results.

If you feel the need to read more about single-user mode, here is an article and another on the topic.

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

In a recent article, I took you on a trip through how to hack (ethically) a SQL Server to regain sysadmin access. In that article, I made quick mention of restarting SQL Server into single-user mode. It only makes sense to show quickly how to get into single-user mode.

Before getting into that, I do hope that there is something you will be able to learn from this basics article. If you are curious, there are more basics articles on my blog – here.

Single-User

So, what exactly is this single-user mode thing? Single-user mode is basically the official back-door into SQL Server for various reasons such as:

• Somebody deleted all of the logins that were in the sysadmin role.
• The sa account is disabled or the password has been forgotten.
• Somebody deleted any Windows groups that were members of the sysadmin role.
• All members of the sysadmin role are no longer with the company.
• You need to restore the master database
• You want to keep SQL Server all to yourself because you are greedy!

These are some pretty solid reasons to need to be able to use the back door. But how exactly do we get to the back door?

Two Paths

As luck would have it, there are two ways to enable single-user mode. You can either get there by making some changes for the SQL Server service in Configuration Manager, or you can utilize a command prompt. I won’t cover the gui path beyond the gentle reminder that you must remember to undo your change when using that method.

My preferred method is through the command line. Using my SQL Server 2017 as the experiment, I would navigate to the Binn directory for that instance. In this case, as shown in the next image.

Before getting too far ahead of myself, I am going to stop my SQL Server.

Notice, I also queried to find all of my services related to SQL before stopping the MSSQLServer service via the net stop mssqlserver command. We will come back to some net start and net stop commands later.

With the service successfully stopped, I can now restart the service in single-user mode.

And then the validation that we are indeed starting in single-user mode…

But wait, did you notice that bit of trickery on the startup command?

sqlservr.exe -m"SQLCMD"

This is a pro-tip for when you must use single-user mode. Inevitably, somebody will steal the single-user connection and you will be locked out of the session. By using an app name after the single-user switch, you are telling SQL Server to only accept connections for that specific application. Since most apps will not be using sqlcmd, you will have far less contention to gain that connection and you will be able to complete your task much easier.

You could also pass something like this instead…

sqlservr.exe -m"Microsoft SQL Server Management Studio - Query"

In this case, I would be limiting the connections to a query from SSMS (and not object explorer).

Now that I have a single-user connection, I can add a sysadmin or restore the master database or just sit on it and play devious. It all depends on what your objective for the single-user session happens to be.

More Command Line

Remember that reference to the NET commands? Well, it turns out we can also start SQL Server in single-user via net start. Let’s check it out.

The command is pretty simple:

net start mssqlserver -m"SQLCMD"

The effect here is the same as navigating to the Binn directory and starting SQL Server with the sqlservr.exe executable. The big difference is considerably less typing and less verbose output of the service startup.

When using the net start method, you do need to know the service name of the SQL Server instance. To get that, I do recommend the following powershell script.

get-service -name *sql* |format-table -auto

This will produce results similar to the following.

From the results, I can pick the SQL Server service and then pass that to the net start command fairly easily.

The Wrap

Starting SQL Server in single-user mode should be a tool every data professional holds in the bag. This is an essential tool that can be used in multiple scenarios and ensure you are able to fully maintain and control your server. I have shown how to get to single-user mode via two command line methods and mentioned a GUI method. The nice thing about the command line methods is that you don’t have to remember to undo the startup switch like you do with the GUI method.

If you feel the need to read more about single-user mode, here is an article and another on the topic.

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

As a data professional can you recall the last time you needed to support a SQL Server instance for which you had no access? What if you used to have access and then that access magically disappeared?

I know I run into this dilemma more than I would probably like to. It is rather annoying to be under a crunch to rapidly provide support only to discover you are stuck and have to wait on somebody else who hopefully has access.

It’s one thing to not have access in the first place. This is usually an easy fix in most cases. The really unpleasant access issue is the one when you have confirmed prior access to the instance and then to be completely locked out. More succinctly, you have lost that SQL access!

Whoa is Me!

All hope is now lost right? OK, that isn’t entirely true. Or is it? What if everybody else from the team is also locked out and there is no known sysadmin account. In essence everybody is locked out from managing the instance and now you have a real crisis, right? Well, not so fast. You can still get back in to the instance with sysadmin access. It should be no real secret that you could always restart the SQL instance in single-user mode. Then again, that probably means bigger problems if the server is a production server and is still servicing application requests just fine.

What to do? What to do?

Restart Prohibited

If you really cannot cause a service disruption to bounce the server into single-user mode, my friend Argenis Fernandez (b | t) has this pretty nifty trick that could help you. Truth be told, I have tested that method (even on SQLExpress) several times and it is a real gem. Is this the only alternative?

Let’s back it up just a step or two first. Not having access to SQL Server is in no way the same thing as not having access to the server. Many sysadmins have access to the windows server. Many DBAs also have access to the Windows server or can at least work with the sysadmins to get access to the Windows server in cases like this. If you have admin access to windows – then not much is really going to stop you from gaining access to SQL on that same box. It is a matter of how you approach the issue. Even to restart SQL Server in single-user mode, you need to have access to the Windows server. So, please keep that in mind as you read the article by Argenis as well as the following.

Beyond the requirement of having local access to the server, one of the things that may cause heartburn for some is the method of editing the registry as suggested by Argenis. Modifying the registry (in this case) is not actually terribly complex but it is another one of those changes  that must be put back the way it was. What if there was another way?

As luck would have it, there is an alternative (else there wouldn’t be this article). It just so happens, this alternative is slightly less involved (in my opinion). Let’s start with a server where I don’t have SQL access (beyond public) but I do have Windows access.

We can see on this SQLExpress instance on the TF server that my “Jason” does not exist. Since I don’t have access, I can’t add my own account either. Time to fix that. In order to fix it, I am going to create Scheduled task in Windows that will run a SQLCMD script from my C:\Database folder. The folder can be anywhere, but I generally have one with scripts and such somewhere on each server that I can quickly access.

From here, you will want to click on the “Change User or Group” button to change it to an account that does have access to SQL Server. The account that I use is not a “user” account but rather it is a “system” account called “NT AUTHORITY\SYSTEM” that is present all the way through SQL Server 2017.

To locate the “NT AUTHORITY\SYSTEM” account, just type “SYSTEM” into the new window and click “Check Names”. The account will resolve and then you can click OK out of the “Select User or Group” window.

With the account selected that will run this task, we can now focus our attention on the guts of the task. We will now go to the “Actions” tab.

Click the new button, and here we will configure what will be done.

I do recommend putting the full path to SQLCMD into “Program/Script” box. Once entered, you will add the following to the parameter box.

-S YourServer\yourinstance -i c:\database\myscript.sql

If you do not have an instance, then just the server name will suffice after the -S parameter. The -i parameter specifies the path to the SQL script file that will be created and placed in the C:\database directory (or whichever directory you have chosen).

That is it for the setup of the task. Now let’s look at the guts of the script file.

create login [mydomain\jason] from windows
exec sp_addsrvrolemember 'mydomain\jason','sysadmin';
go

Save that into a script document named myscript.sql in the aforementioned directory and then execute the windows task. After executing the Windows task, it is time to verify if it worked or not.

Boom! From no access to a sysadmin in a matter of seconds. Here is that quick verify script – generalized.

SELECT	spr.name AS SvrRole
	, sp.name AS SvrPrincipal
FROM	sys.server_principals sp
		INNER JOIN sys.server_role_members rm
			ON sp.principal_id = rm.member_principal_id
		INNER JOIN sys.server_principals spr
			ON rm.role_principal_id = spr.principal_id
WHERE spr.name = 'sysadmin';

The Wrap

Losing access to a SQL instance is never a desirable situation – for the DBA. When the people that are supposed to have access, lose that access, all hope is not lost. There are plenty of methods available to regain the requisite access to manage the server. Today, I shared one such method that I view as being extremely easy. If you lose access, I would recommend taking the steps shown in this article to regain that access.

While not in the back to basics series, I do recommend checking out my other posts in that series. Some topics in the series include (but are not limited to): Backups, backup history and user logins. I would also recommend reading this audit article. If you are able to elevate your permissions, then obviously anybody with server access can elevate their permissions too. For that reason, you should regularly audit the permissions and principals in SQL Server.

One of my most favorite things in the world is the opportunity to deal with extremely varying database and environment requirements. Many vendors and databases seem to have a wide swath of different requirements. Some of the reasons for these requirements are absurd and some are not. That is a discussion for a different day.

When dealing with vendors, sometimes you get good documentation and requirements for the app. If you happen across one of these opportunities, you should consider buying a lottery ticket. Most of the time, the requirements and documentation are poorly assembled and then suffer from linguistic shortcomings.

What do you do when you run into poor documentation from a vendor? The typical answer would be to either call them or make a best guess (even if you call them, you are likely stuck with a best guess anyway). Then what do you do when you find that your best guess was completely wrong? Now it is time to back pedal and fix it, right?

When that mistake involves the server collation setting, the solution is simple – right? All you need to do is uninstall and reinstall SQL Server. That is the common solution and is frankly a horrific waste of time. This article will show some basics around fixing that problem quickly without a full reinstall.

I do hope that there is something you will be able to learn from this basics article. If you are curious, there are more basics articles on my blog – here.

Reinstall Prohibited

I am not a huge fan of wasting time doing something, especially if there is a more efficient way of achieving the same end result. I am not talking about cutting corners. If you cut corners, you likely just end up with having more work to do to fix the problems your sloppiness will have caused. That to me is not the same end result.

Having run into a bit of a problem with a vendor recently (with lacking requirements), I found myself with a server that was installed with the wrong collation instead of what the vendor wanted (never-mind they said nothing of it until a month after the server was setup and ready for them to use). The vendor needed the collation fixed immediately (basically it needed to be fixed yesterday). I really did not want to do a reinstall of the server and the sysadmins were just about to click through the uninstall and redo the install.

Oy Vey! Everybody hold up just a second here! First things first – verify with certainty there is good reason to need to change the server collation. It is perfectly legit to give the vendor the third degree here. Make sure they understand why they need the change. If they can answer the questions satisfactorily, then proceed with the change.

Next, just because the vendor says you have to uninstall/reinstall (or reboot) the server to make a certain change, does not mean they know what they are talking about. I have run into too many cases where the vendor thinks you must reboot the server to change the max memory setting in SQL Server (not true for sure).

Sure, common myth would say that you must reinstall SQL Server in order to change the default server collation. That is not entirely accurate. Reinstall is just one option that exists.

In the case of this vendor, they required that the SQL_Latin1_General_CP850_CS_AS collation be used. The server was set for SQL_Latin1_General_CP1_CI_AS. So, let’s see how we can change the collation without a reinstall.

The first thing to do is to confirm the collation we have set.

We can see from these results that indeed the collation is wrong and we need to change it in order to comply with the request from the vendor. Next we will need to stop the SQL Server services.

I think that is pretty clear there what to do. As a reminder, the preferred method to stop and start SQL Server services is via the SQL Server Configuration Manager. We won’t do every start/stop from here for this article for good reason.

Once the services are stopped, then we need to open an administrative command prompt and navigate to the SQL Server binn directory as shown here.

This is for a default instance on SQL Server 2017. If you have a named instance or a different version of SQL Server, you will need to navigate the instance folder structure for your instance.

Next is where the magic happens. We enter a command similar to this:

sqlservr -m -T4022 -T3659 -q"SQL_Latin1_General_CP850_CS_AS"

Here is a quick summary of those flags in this command:

[-m] single user admin mode
[-T] trace flag turned on at startup
[-q] new collation to be applied

There are more such as -s available in books online for your perusal.

If you are curious what is up with those Trace Flags, pretty simple. TF4022 is to bypass startup procs. TF3659 on the other hand is supposed to write errors to the error log (at least in theory).

When the script starts, you will see something like the next two screens:

In the first, you can see that it says it is attempting to change the collation. In the second, just before the completion message, it states that the default collation was successfully changed. Let’s close this command prompt window and then go start SQL Server and validate the change.

And that is a successful change. See how easy that is? This effort takes all of maybe 5 minutes to complete (validation, starting, stopping and so on). I would take this over a reinstall on most days.

Now that we have changed the collation, all I need to do is repeat the process to set the collation back to what it was originally (in my test lab) and make sure to bookmark the process so I can easily look it up the next time.

There is a bit of a caveat to this. On each change of the collation, I ran into permissions issues with my default logging directory (where the sql error logs are written). I just needed to reapply the permissions and it was fine after that (SQL Server would not start). That said, the permissions issue was not seen on the box related to the change for the vendor. So just be mindful of the permissions just in case.

The Wrap

Every now and again we have to deal with a sudden requirements change. When that happens, we sometimes just need to take a step back and evaluate the entire situation to ensure we are proceeding down the right path. It is better to be pensive about the course of action rather than to knee jerk into the course of action. Do you want to spend 5 minutes on the solution or 30-40 minutes doing the same thing? Changing collation can be an easy change, or it can be one met with a bit of pain doing a reinstall (more painful if more user databases are present). Keep that in mind and keep cool when you need to make a sudden change.

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

This topic is not new to me. You may recall a previous article I wrote entailing one type of failure and how to resolve that failure. That article can be found here.

I recently ran into a variant of the problem outline in that previous article that requires just a bit of a different approach. The errors turn out to be similar on the surface but really are different upon closer inspection.

Phantom Health Records

If you are unfamiliar with the topic, I recommend reading the previous article. Then after reading that article, maybe brush up a little bit on the SQL Agent. The failures we will be looking at are within the SQL Agent and come from the job called: syspolicy_purge_history.

For this latest bout of failure, I will start basically where I left off in the the last article. The job fails on a server and I have access to other servers of the same SQL version where the job works.

Before diving any further into this problem, let’s look at what the error is:

A job step received an error at line 1 in a PowerShell script.
The corresponding line is ‘set-executionpolicy RemoteSigned -scope process -Force’.
Correct the script and reschedule the job. The error information returned by PowerShell is:
‘Security error. ‘. Process Exit Code -1. The step failed.

Having the error in hand, and knowing that the job works elsewhere, my next logical step (again based on experience from the last article with this job) is to script the job from another server and use it to replace the job on the server where it fails. In principle this is an AWESOME idea.

Sadly, that idea was met with initial failure. As it turns out, the error remained exactly the same. This is good and unfortunate at the same time. Good in that I was able to confirm that the job was correctly configured with the following script in the job:

if ('$(ESCAPE_SQUOTE(INST))' -eq 'MSSQLSERVER') {$a = '\DEFAULT'} ELSE {$a = ''};
(Get-Item SQLSERVER:\SQLPolicy\$(ESCAPE_NONE(SRVR))$a).EraseSystemHealthPhantomRecords()

Since the step fails from SQL Server let’s see what else we can do to make it run. Let’s take that code and try it from a powershell ise. So, for giggles, let’s cram that script into powershell and see what blows up!

SRVR : The term 'SRVR' is not recognized as the name of a cmdlet, function, 
script file, or operable program. Check the spelling of the name, or if a path 
was included, verify that the path is correct and try again.
At line:2 char:46

Now isn’t that a charming result! This result should be somewhat expected since the code I just threw into the ISE is not entirely powershell. If you look closer at the code, you will notice that it is using sqlcmd like conventions to execute a parameterized powershell script. Now, that makes perfect sense, right? So let’s clean it up to look like a standard PoSH script. We need to replace some parameters and then try again.

(Get-Item sqlserver:\SQLPolicy\MyServer\Default).EraseSystemHealthPhantomRecords()

This will result in the following (resume reading after you scratch your head for a moment):

Get-Item : Cannot find drive. A drive with the name 'sqlserver' does not exist.
At line:1 char:2
+ (Get-Item 
sqlserver:\SQLPolicy\SPMS1\Default).EraseSystemHealthPhantomRecords()

The key in this failure happens to be in the sqlserver. PoSH thinks we are trying to pass a drive letter when we are just trying to access the SQLServer stuff. Depending on your version of server, SQL Server, and PoSH you may need to do one of a couple different things. For this particular client/issue, this is what I had to try to get the script to work.

Import-Module “sqlps” -DisableNameChecking;

(Get-Item sqlserver:\SQLPolicy\myserver\Default).EraseSystemHealthPhantomRecords()

If you read the previous article, you may notice this command looks very much like the command that was causing the problems detailed in that previous article. Yes, we have just concluded our 180 return to where we started a few years back. Suffice it to say, this is expected to be a temporary fix until we are able to update the system to PoSH 5 and are able to install the updated sqlserver module.

As is, this script is not quite enough to make the job succeed now. To finish that off, I created a ps1 file to house this script. Then from a new step (defined as a sqlcmd step type) in the syspolicy purge job, I execute that powershell script as follows:

powershell.exe "c:\dba\syshealth.ps1"

Tada, nuisance job failure alert is resolved and the system is functioning again.

Conclusion

I dare say the quickest resolution to this job is to probably just disable it. I have seen numerous servers with this job disabled entirely for the simple reason that it fails frequently and just creates noise alerts when it fails. Too many fixes abound for this particular job and too few resolve the failures permanently.

I would generally err on the side of fixing the job. Worst case, you learn 1000 ways of what not to do to fix it.

Given this job is tightly related to the system_health black box sessions (sp_server_diagnostics and system_health xe session), I recommend fixing the job. In addition, I also recommend reading the following series about XE and some of those black box recorder sessions – here.

It may come as no surprise to many that Microsoft has hastened the SQL Server development cycle. Furthermore, it may be no surprise to many that Microsoft has also hastened the patch cycle for SQL Server.

If you were unaware of this, consider this as your notice that Microsoft has indeed hastened the patch cycle. Not only has the patch cycle become more rapid, the idea of Service Packs is more or less a notion of history at this point. Critical Updates (or CUs) is the new norm. This is a pretty good thing due in large part to the rapid improvements that can be made to the product due to Azure.

With all of this considered now, there is some really awesome news. A hint to this awesome news is in the preceding image and title of this post. In a recent CU for SQL Server 2017, Availability Groups and Extended Events both have seen massive upgrades. The upgrades are so big in fact that it is mind blowing. These upgrades were no small feat by any means and it took some major investment and cooperation from the likes of some well known competitors.

Upgrade the first: Availability Groups have now been extended to be able to include nodes from MySQL, PostGres and MariaDB. Frankly, I don’t understand the MariaDB move there but it’s all good. I am 100% on board with the MySQL addition and may have to work really hard to find a use case to include PostGres.

Imagine the realm of possibility this change brings!! First we got SQL Server on Linux and now we can include a predominantly Linux flavored DBMS in a SQL Server High Availability solution. LAMP engineers have got to be losing their gourds right about now over this. Microsoft is taking away every anti-MS premise that has been used in recent years and turning the world on its ears to become more global and reachable in the architecture and DBMS world.

Upgrade the second: In order to help support and troubleshoot AGs on these other platforms, we need some tools. The tools of choice happen to be in the form of Extended Events. While there is nothing quite yet in place on these other platforms to properly monitor an AG, XE is able to capture some MySQL, PostGres and MariaDB information as transmitted across the wire when these platforms are added to an AG. How COOL is that?

If you are really chomping at the bit, I recommend procuring the latest CU that was recently released. You can find that CU from this Microsoft site here.

The Wrap

I am in full support of this new direction from Microsoft. Partnering with other large platforms to provide a supremely improved overall product is very next level type of stuff and frankly quite unheard of in this ultra competitive world. It is so unheard of in fact that this was a nicely crafted April Fools joke. Happy April Fools Day!

Having mentioned Extended Events, if you are interested, I do recommend a serious read from any number of articles posted in 60 day series.

It should come as no surprise that I write a lot of articles about Extended Events (XE). This happens to be another article on Extended Events. Truth be told, this article is hopefully something that is more of an edge case scenario. Well, I sure hope that is the case and that it is not a common problem.

One of the recommended methods to trap payload data in an XE session is via the use of the event_file target. Sending data to a file has numerous benefits such as being able to take the trace and evaluate the trace file from a different machine (locally to that machine).

Every once in a blue moon you just may run into various issues with the event_file such as explained here or here. Though slightly different, the net effect is quite similar and should be treated with roughly the same kind of troubleshooting steps.

Configuration Error

As luck would have it, I ran into one of these rare opportunities to troubleshoot an error occurring on a client server. Truth be told, I was unfamiliar with the actual error at first. Here is that error.

Error: 25602, Severity: 17, State: 22.

The preceding error was scraped out of the SQL Server error log. Obviously a little more detail was needed because this error is far from useful without that detail. Looking a little deeper, I found some errors like this.

Msg 25602, Level 17, State 22, Line 43
The target, “5B2DA06D-898A-43C8-9309-39BBBE93EBBD.package0.event_file”,
encountered a configuration error during initialization. Object cannot be added to the event session.

Some very good clues are actually contained in that particular message. Some of these clues include the following: a) the term “target”, b) the term “event_file”, and c) the phrase “event session.” Ok, I get it at this point. One of my Extended Event Sessions I had put on the server and used previously was broken. But, since it had been working and I know I had fetched data from it, I found myself puzzled as to why it might be busted.

The next logical thing to do at this point was to test the various sessions that are stopped and try to figure out which one is causing the problem and see if the error is reproduced. Finally upon finding the session that is failing, I ran into the complete message.

Msg 25602, Level 17, State 22, Line 43
The target, “5B2DA06D-898A-43C8-9309-39BBBE93EBBD.package0.event_file”,
encountered a configuration error during initialization. Object cannot be added to the event session.
The operating system returned error 3: ‘The system cannot find the path specified.
‘ while creating the file ‘C:\Database\XEDROPME\SVRLoginAudit_0_131650006658030000.xel’.

The additional info that I needed is in bold text in the previous text. So, for some reason, there is a problem with the path for the XE trace file output. Going out to the file system to check it out, I found that the client in this case decided to delete the entire folder. How does that happen? Well, it does! When it happens, the XE traces will start to fail and you will no longer capture the intended trace data. Let’s take a look at a simulated reproduction of this issue.

First, I will create a session and then start the session, then validate the session exists and then stop the session.

USE master;
GO
-- Create the Event Session
IF EXISTS ( SELECT *
				FROM sys.server_event_sessions
				WHERE name = 'SVRLoginAudit' )
	DROP EVENT SESSION SVRLoginAudit 
    ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XEDROPME';
GO

CREATE EVENT SESSION SVRLoginAudit ON SERVER
ADD EVENT sqlserver.login ( SET collect_database_name = ( 1 ), collect_options_text = (1)
	ACTION ( sqlserver.sql_text,sqlserver.nt_username,sqlserver.server_principal_name,sqlserver.client_hostname,
	package0.collect_system_time,package0.event_sequence,
	sqlserver.database_id, sqlserver.database_name,
	sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id,
	sqlserver.context_info, sqlserver.client_connection_id ) 
	)
ADD TARGET package0.event_file ( SET filename = N'C:\Database\XEDROPME\SVRLoginAudit.xel' , max_file_size = ( 5120 )
									, max_rollover_files = ( 4 ) )
WITH (STARTUP_STATE = ON
	,TRACK_CAUSALITY = ON);

/* start the session */
ALTER EVENT SESSION SVRLoginAudit 
ON SERVER 
STATE = START;
GO

DECLARE @SessionName VARCHAR(128) = NULL --'BackupThroughput' --'sp_server_diagnostics session' --NULL for all
;

SELECT ISNULL(ses.name,xse.name) AS SessionName
		, CASE
			WHEN ISNULL(ses.name,'') = ''
			THEN 'Private'
			ELSE 'Public'
			END AS SessionVisibility
		, CASE
			WHEN ISNULL(xse.name,'') = ''
			THEN 'NO'
			ELSE 'YES'
			END AS SessionRunning
		, CASE
			WHEN ISNULL(xse.name,'') = ''
				AND ISNULL(ses.name,'') = ''
			THEN 'NO'
			ELSE 'YES'
			END AS IsDeployed
	FROM sys.server_event_sessions ses
	FULL OUTER JOIN sys.dm_xe_sessions xse
		ON xse.name =ses.name
	WHERE COALESCE(@SessionName, ses.name, xse.name) = ISNULL(ses.name, xse.name)
	ORDER BY ses.event_session_id;
GO

/* stop the session */
ALTER EVENT SESSION SVRLoginAudit 
ON SERVER 
STATE = STOP;
GO

And here is what I see on my test server when I validate the session exists.

Perfect so far. Now, let’s make that trace output directory disappear. For this demo, you might note that I had created the directory as “C:\Database\XEDROPME”. My intent in the name was obviously to notify the world that the folder was to be dropped.

xp_cmdshell 'rd C:\Database\XEDROPME /Q /s'
GO

That statement is easy enough and is performed from my test environment for those getting weary of the use of xp_cmdshell. Now, let’s try to start that session that we knew was previously running.

I re-formatted the output of the error for ease of readability. Otherwise, the output in the preceding image is what will happen when the output directory is dropped. The fix is rather simple at this point – put the folder structure back into place. To read an introductory post about checking if a session exists or not on your server, check this out (a more advanced post is coming soon).

Conclusion

From time to time we will run into various problems supporting Extended Events. This is bound to happen more frequently as we support more varied environments with more hands in the kitchen (so to speak). We need to learn that even small changes can have a ripple effect to other things that may be running on the server. It is worthwhile to perform a little due diligence and clean things up as we make changes – or at minimum to observe the system for a time to ensure no unintended consequences have occurred.

Extended Events is a powerful tool to help in troubleshooting and tuning your environment. I recommend investing a little time in reading the 60 day series about Extended Events. This is not a short series but is designed to provide an array of topics to help learn the tool over time.

With that there is a little more about the feature that the aforementioned book did not cover. In fact, this information has pretty much gone ignored and mostly stays hidden under the covers.

Settings

As of SSMS 17.4 we have been given the ability to control XEvents Profiler just a tiny bit more. For what it is worth, we as Database Professionals love to be able to control our database environment. So this teeny tiny bit of new control ability is potentially a huge win, right?

If you are the controlling type, or maybe just the curious type, you will be pleased to know that under “Options” from the Tools menu in SSMS, Microsoft has tucked some new control options to help you configure XEvents Profiler – to a degree. If you open options, you will see this new node.

If you expand the “XEvent Profiler” node (circled in red), you will discover the “options” node. If you click on this “options” node and do a quick comparison (in SSMS 17.4 and SSMS 17.5) you will also find that you don’t need t expand the “XEvent Profiler” node at all because the options are listed in the right hand pane for both nodes and they are exactly the same. So, choose one or the other and you will end up at the same place.

The options that you currently have are:

• Stop Session on Viewer Closed
• Toolbar commands stop and restart

You can either set these options to True or False. I recommend you play with them a bit to discover which you really prefer. That said, I do prefer to have the “Stop Session on Viewer Closed” set to true. There is “profiler” in the name of the feature afterall. And if you have read the “book” I wrote about this feature, you would know that the filtering offered by the default sessions of this feature basically turn on the fire hose effect and can have a negative impact on your server. Are you sure you want a profiler style fire hose running on your production server?

Conclusion

There surely will continue to be more development around this idea of an XE style profiler. More development generally means that the product will mature and get better over time. This article shows how there is more being added to the feature to try and give you better control over the tool. We love control so the addition of these options is actually a good thing. Is it enough to sway me away from using the already established, more mature, and high performing tools that have been there for several generations? Nope! I will continue to use TSQL and the GUI tools available for XE that predated the XEvent Profiler.

Some say this is a way of bridging the gap. In my opinion, that gap was already bridged with the GUI that has been available for several years. Some say that maybe this tool needs to integrate a way to shred XML faster. To that, I say there are methods already available for that such as Powershell, the live data viewer, the Target Data viewer, or even my tools I have provided in the 60 day series.

I would challenge those that are still unfamiliar with the XE GUI (out for nearly 6 years now) to go and read some of my articles or articles by Jonathan Kehayias about the power that is in XE as well as some of the power in the GUI.

One of my all-time favorite things in SQL Server is security. No matter what, it always seems that there is a new way to abuse permissions. When people abuse their access level or abuse the way permissions should be set in a SQL Server environment, we get the pleasure of both fixing it and then trying to educate them on why what they did was wrong and how to do it the right way.

In similar fashion, I previously wrote about some fundamental misconceptions about permissions here and here. I have to bring those specific articles up because this latest experience involves the basics discussed in those articles along with a different twist.

I do hope that there is something you will be able to learn from this basics article. If you are curious, there are more basics articles on my blog – here.

Gimme Gimme Gimme…

It is not uncommon to need to create a login and grant that login access to a database (or associate that login to a database user. In fact, that is probably a fairly routine process. It is so routine, that I have a demo script for it right here.

USE [master]
GO
IF EXISTS	(SELECT 1/0 FROM sys.databases WHERE name = 'GimmeSA')
BEGIN
DROP DATABASE GimmeSA;
END

CREATE DATABASE [GimmeSA];

IF NOT EXISTS (SELECT 1/0 FROM sys.server_principals WHERE name = 'IMustHaveSA')
BEGIN
SELECT 'IMustHaveSA Does NOT Exist - Create the login'
CREATE LOGIN [IMustHaveSA] WITH PASSWORD=N'weakpassword', DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF;
END


USE [GimmeSA]
GO
CREATE USER [IMustHaveSA] FOR LOGIN [IMustHaveSA]
GO
USE [GimmeSA]
GO
ALTER ROLE [db_accessadmin] ADD MEMBER [IMustHaveSA]
GO
USE [GimmeSA]
GO
ALTER ROLE [db_backupoperator] ADD MEMBER [IMustHaveSA]
GO
USE [GimmeSA]
GO
ALTER ROLE [db_datareader] ADD MEMBER [IMustHaveSA]
GO
USE [GimmeSA]
GO
ALTER ROLE [db_datawriter] ADD MEMBER [IMustHaveSA]
GO
USE [GimmeSA]
GO
ALTER ROLE [db_ddladmin] ADD MEMBER [IMustHaveSA]
GO
USE [GimmeSA]
GO
ALTER ROLE [db_owner] ADD MEMBER [IMustHaveSA]
GO
USE [GimmeSA]
GO
ALTER ROLE [db_securityadmin] ADD MEMBER [IMustHaveSA]
GO

I even went as far as to include some of the very routine mistakes I see happening on a frequent basis (as referenced by a prior post here).

To this point, we only have a mild abuse of how to set permissions for a principal. Now it is time for that twist I mentioned. This user account needs to be created on a secondary server that is participating in either a mirror or an Availability Group. Most people will take that user account that was just created on the first server and then use the same script to add the account to the secondary server. Let’s see how that might look.

For this example, I will not go to the extent of creating the mirror or AG. Rather, I will pretend I am just moving the database to a new server. So I have taken a backup and then I will restore the database to the new server.

USE [master]
RESTORE DATABASE [GimmeSA] FROM  DISK = N'C:\Database\GimmeSA.bak' 
WITH  FILE = 1
,  MOVE N'GimmeSA' TO N'D:\SQL16\Data\GimmeSA.mdf'
,  MOVE N'GimmeSA_log' TO N'D:\SQL16\Log\GimmeSA_log.ldf',  NOUNLOAD,  STATS = 5, replace

GO

Next, let’s go ahead and recreate the login we created on the previous server.

IF NOT EXISTS (SELECT 1/0 FROM sys.server_principals WHERE name = 'IMustHaveSA')
BEGIN
SELECT 'IMustHaveSA Does NOT Exist - Create the login'
CREATE LOGIN [IMustHaveSA] WITH PASSWORD=N'weakpassword', DEFAULT_DATABASE=[master], CHECK_EXPIRATION=OFF, CHECK_POLICY=OFF;
END

You see here that I am only going to create the login if it does not exist already. Running the script produces the following for me.

Now, let’s deviate a bit and grant permissions for the login just like so many administrators will do.

It seems pretty apparent that my login that I just created does not have access to the GimmeSA database, right? Let’s go ahead and add permissions to the GimmeSA database and see what happens.

Well, that did not work according to plan right? Enter twist the second.

What I am seeing more and more of, is people at this point will just grant that login (that was just created) sysadmin rights. You can pick up your jaw now. Indeed! People are just granting the user SA permissions and calling it good. This practice will certainly work – or appear to work. The fact is, the problem is not fixed. This practice has only camouflaged the problem and it will come back at some future date. That date may be when somebody like me comes along and starts working on stripping non-essential sysadmins from the system.

There are two legitimate fixes for this particular problem (and no granting sysadmin is definitely not one of them). First you can run an orphan fix with a script such as this one by Ted Krueger. That will map the user that already exists in the database to the login principal (thus the reason for the error we saw). Or, you can prep your environment better by using the SID syntax with the create login as follows.

CREATE LOGIN TestLogin WITH PASSWORD = 'SuperSecret52&&'
, SID = 0x241C11948AEEB749B0D22646DB1A19F2;

The trick here is to go and lookup the SID for the login on the old server first and then use that sid to create the login on the new server. This will preserve the user to login mappings and prevent the orphan user issue we just saw. It will also prevent the band-aid need of adding the login to the sysadmin server role.

The Wrap

In this article I have introduced you to some basics in regards to creating and synchronizing principals across different servers. Sometimes we try to shortcut the basics and apply band-aids that make absolutely no sense from either a practical point of view or a security point of view. Adhering to better practices will ease your administration burden along with improving your overall security presence.

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

Calmly, you settle in and check the server and eventually find your way to the error logs to see the following:

Msg 823, Level 24, State 2, Line 1

The operating system returned error 1(Incorrect function.) to SQL Server during a read at offset 0x0000104c05e000 in file ‘E:\Database\myproddb.mdf’. Additional messages in the SQL Server error log and system event log may provide more detail. This is a severe system-level error condition that threatens database integrity and must be corrected immediately. Complete a full database consistency check (DBCC CHECKDB). This error can be caused by many factors; for more information, see SQL Server Books Online.

Suddenly you understand and feel the collective fear and paranoia. What do you do now that the world has seemingly come to an end for your database?

Corruption

What exactly does this error message mean? Well, typically, an 823 error is a very strong indicator that there is some sort of problem with the storage system, hardware or driver that is in the path of the I/O request.

Great! That fear is getting a little heavier knowing what the error represents. This doesn’t bode well for the database. Let’s go ahead and crack out the list of what we can do or check when a problem like this hits:

1. Check msdb.dbo.suspect_pages
2. Run a consistency check for all databases on the same volume
3. Check Logs (SQL, Windows, Storage system) to see if there may be additional info (via different errors/warnings) in close proximity to the 823 error.
4. Check your drivers
5. Restore the database

This is where your experience, training, and preparedness come in handy. An experienced data professional will be prepared with database backups (including log backups). So you are not concerned here because all of your backups are reporting successful. As you prep to pull the backups (for the past couple of days just in case) you notice that there are no available backups in your repository. Looking closer at your backup jobs you discover that the backups completed in mere seconds where they normally take hours for this database.

Now that your heart is racing, forehead is beading up with sweat, gut is sinking and the fear is very palpable – what do you do? Time to step through the rest of the steps and pull out your lucky charms, right?

Querying against suspect_pages, you find the table to be completely empty. You know that checkdb runs regularly but maybe it didn’t run last night. That is easy enough to check with a little query from here. Since a consistency check does not seem to have run (as confirmed by the script) and is the second item on the checklist, let’s go ahead and run it now.

Msg 0, Level 11, State 0, Line 0

A severe error occurred on the current command.  The results, if any, should be discarded.

Msg 0, Level 20, State 0, Line 0

A severe error occurred on the current command.  The results, if any, should be discarded.

Crud. Blood pressure and nerves are getting a little more frazzled now. Maybe we can cycle through the database and find which table is causing the problem. Let’s try a checktable of every table in the database. Before doing the checktable, one more check against suspect_pages still shows no rows to be found.

Running the checktable, every table is coming up clean except one. That one table produces the same sort of error as the checkdb and just so happens to be the largest and most critical table to the database. Blood pressure is closing in on critical now. We have a corruption issue that is severe enough that checktable cannot complete, we know we have 823 errors and some sort of IO issue and do not have a backup.

Wait…backup. Let’s try to force a backup and see what happens. We can tell the backup to continue after error so let’s see what happens. Maybe that will allow you to move the database to a different server or different spindles to try and just recover the data.

Msg 3202, Level 16, State 2, Line 1

Write on “E:\SQLBackups\myproddb.bak” failed: 1(Incorrect function.)

Msg 3013, Level 16, State 1, Line 1

BACKUP DATABASE is terminating abnormally.

The situation just does not want to get any better at this point. Time for drastic measures – shut down the SQL Server services and try to xcopy the data and log files to a different server and try to re-attach from there. Anything is worth a shot, right?

Error 1: Incorrect Function

Glad to know it is not just SQL Server throwing the errors – sorta. The corruption is ever present and there is nothing more that can be done, right? All hope is lost. Time to fill out the resume and move on to something else, right? Give it one more shot. A spark of insanity hits and you wonder if a mere query against the clustered index will work, if not then maybe something against any of the indexes to try and spare any data at all.

You rub your luck horseshoe and query the table (yes the table that checktable aborts because of corruption). Lo and behold you get results from this very simple query. How is that possible. On a whim, you drop all the Non-Clustered Indexes and try a fresh backup.

Hallelujah! The backup completes without error. Time to take this backup and restore the database to a completely different server. Then on the new server run a consistency check to determine if it is all clear. To your liking, there is absolutely no corruption at this point so the non-clustered indexes can be recreated (easy to do because you have a script with the index definitions handy).

Wrap

This journey from fire and brimstone and the world ending to the epiphany and then the sweet euphoric feelings of success is not a normal resolution for these types of errors. This sort of thing happened for a client that called trying to get around the 823 errors. We had absolutely no indication whatsoever of where the corruption was beyond knowing we had failing disks at the time. We got lucky in that the non-clustered indexes in this case ended up being stored on the bad sectors and dropping those indexes allowed us to recover the database and make it usable.

When everything is breaking, the unconventional thought (especially without having a checkdb complete to tell you which index or which pages are corrupt) of dropping indexes may just save your bacon. It saved my clients bacon!

As a follow-up item, it is important to regularly check the dbccLastKnownGood for each database. That can be done by following the script in this article.

One thing that SQL Server does very well is come pre-configured in a lot of ways. These pre-configured settings would be called defaults. Having default settings is not a bad thing nor is it necessarily a good thing.

For me, the defaults lie somewhere in the middle ground and they are kind of just there. You see, having defaults can be good for a horde of people. On the other hand, the default settings can be far from optimal for your specific conditions.

The real key with default settings is to understand what they are and how to get to them. This article is going to go through some of the basics around one group of these defaults. That group of settings will be accessible via the sp_configure system stored procedure. You may already know some of these basics, and that is ok.

I do hope that there is something you will be able to learn from this basics article. If you are curious, there are more basics articles on my blog – here.

Some Assembly Required…

Three dreaded words we all love to despise but have learned to deal with over the past several years – some assembly required. More and more we find ourselves needing to assemble our own furniture, bookcases, barbecue grills, and bathroom sinks. We do occasionally want some form of set and forget it.

The problem with set it and forget it type of settings (or defaults) is as I mentioned – they don’t always work for every environment. We do occasionally need to manually adjust settings for what is optimal for that database, server, and/or environment.

When we fail to reconfigure the defaults, we could end up with a constant firefight that we just don’t ever seem to be able to win.

So how do we find some of these settings that can help us customize our environment for the better (or worse)? Let’s start taking a crack at this cool procedure called sp_configure! Ok, so maybe I oversold that a bit – but there is some coolness to it.

Looking at msdn about sp_configure I can see that it is a procedure to display or change global configuration settings for the current server.

If I run sp_configure without any parameters, I will get a complete result set of the configurable options via this procedure. Let’s look at how easy that is:

Ok, so that was exceptionally easy. I can see that the procedure returns the configurable settings, the max value for the setting, configured value, and the running value for each setting. That is basic information, right? If I want a little more detailed information, guess what? I can query a catalog view to learn even more about the configurations – sys.configurations.

SELECT *
	FROM sys.configurations c
	ORDER BY c.name;

That query will also show me (in addition to what I already know from sp_configure) a description for each setting, if the setting is a dynamic setting and whether or not the setting is an advanced configuration (and thus requires “show advanced options” to be enabled). Pro-tip: The procedure just queries the catalog view anyway. Here is a snippet from the proc text.

-- If no option name is given, the procedure will just print out all the
	--  options and their values.
	if @configname is NULL
	begin
		select name,
			convert(int, minimum) as minimum,
			convert(int, maximum) as maximum,
			convert(int, isnull(value, value_in_use)) as config_value,
			convert(int, value_in_use) as run_value
		from  sys.configurations
		where (is_advanced = 0 or @show_advance = 1)
		order by lower(name)

		return (0)
	end

Seeing that we have some configurations that are advanced and there is this option called “show advanced options”, let’s play a little bit with how to enable or disable that setting.

sp_configure 'show advanced options',1

With the result (on my system) being:

We can see there that the configuration had no effect because I already had the setting enabled. Nonetheless, the attempt to change still succeeded. Let’s try it a different way.

I ran a whole bunch of variations there for giggles. Notice how I continue to try different words or length of words until it finally errors? All of them have the same net effect (except the attempt that failed) they will change the configuration “show advanced options”. This is because all that is required (as portrayed in the failure message) is that the term provided is enough to make it unique. The uniqueness requirement (shortcut) is illustrated by this code block from sp_configure.

See the use of the wildcards and the “like” term? This is allowing us to shortcut the configuration name – as long as we use a unique term. If I select a term that is not unique, then the proc will output every configuration option that matches the term I used. From the example I used, I would get this output as duplicates to the term I used.

Ah, I can see the option I need! I can now just copy and paste that option (for the sake of simplicity) into my query and just proceed along my merry way. This is a great shortcut if you can’t remember the exact full config name or if you happen to be really bad at spelling.

The Wrap

In this article I have introduced you to some basics in regards to default server settings and how to quickly see or change those settings. Not every environment is able to rely on set-it and forget-it type of defaults. Adopting the mentality that “some assembly is required” with your environments is a good approach. It will help keep you on top of your configurations at the bare minimum. This article will help serve a decent foundation for some near future articles. Stay tuned!

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

Once in a while I come across something that sounds fun or interesting and decide to dive a little deeper into it. That happened to me recently and caused me to preempt my scheduled post and work on writing up something entirely different. Why? Because this seemed like fun and useful.

So what is it I am yammering on about that was fun?

I think we can probably concede that there are some best practices flying around in regards to the configuration of tempdb. One of those best practices is in regards to two trace flags within SQL Server. These trace flags are 1117 and 1118. Here is a little bit of background on the trace flags and what they do.

A caveat I have now for the use of trace flags is that I err on the same side as Kendra (author of the article just mentioned). I don’t generally like to enable trace flags unless it is very warranted for a very specific condition. As Kendra mentions, TF 1117 will impact more than just the tempdb data files. So use that one with caution.

Ancient Artifacts

With the release of SQL Server 2016, these trace flags were rumored to be a thing of the past and hence completely unnecessary. That is partially true. The trace flag is unneeded and SQL 2016 does have some different behaviors, but does that mean you have to do nothing to get the benefits of these Trace Flags as implemented in 2016?

As it turns out, these trace flags no longer do what they did in previous editions. SQL Server now pretty much has it baked into the product. Buuuuut, do you have to do anything slightly different to make it work? This was something I came across while reading this post and wanted to double check everything. After all, I was also under the belief that it was automatically enabled. So let’s create a script that checks these things for me.

/*
http://blog.waynesheffield.com/wayne/archive/2017/09/registry-sql-server-startup-parameters/
https://docs.microsoft.com/en-us/sql/t-sql/statements/alter-database-transact-sql-file-and-filegroup-options
https://docs.microsoft.com/en-us/sql/t-sql/statements/alter-database-transact-sql-set-options

*/
USE tempdb;
GO

DECLARE @ServerMajorVersion DECIMAL(4, 2)
	, @TSQL VARCHAR(2048);

IF ( SELECT OBJECT_ID('tempdb..#dbsettings')) IS NOT NULL
	BEGIN
		DROP TABLE #dbsettings;
	END;
IF ( SELECT OBJECT_ID('tempdb..#tfstatus')) IS NOT NULL
	BEGIN
		DROP TABLE #tfstatus;
	END;
CREATE TABLE #tfstatus
	(
		TraceFlag INT
	, Status TINYINT
	, Global TINYINT
	, Session TINYINT
	);
CREATE TABLE #dbsettings
	(
		DBName VARCHAR(128)
	, DataFileName VARCHAR(128)
	, physical_name VARCHAR(1024)
	, FileGroupName VARCHAR(256)
	, is_mixed_page_allocation_on TINYINT
	, is_autogrow_all_files TINYINT
	, is_persistent_log_buffer TINYINT
	);

SELECT	@ServerMajorVersion = CONVERT(
								DECIMAL(4, 2)
								, PARSENAME(dt.fqn, 4) + '.'
								+ PARSENAME(dt.fqn, 3))
FROM	( SELECT	CONVERT(VARCHAR(20), SERVERPROPERTY('ProductVersion'))) dt(fqn);

SELECT	@ServerMajorVersion;

INSERT INTO #tfstatus ( TraceFlag
						, Status
						, Global
						, Session )
	EXECUTE ( 'DBCC TRACESTATUS' );

IF @ServerMajorVersion < 13.00
	BEGIN

		SET @TSQL = 'SELECT	d.name AS DBName
			, df.name AS DataFileName
			, df.physical_name
			, fg.name as FileGroupName
			, NULL AS is_mixed_page_allocation_on
			, NULL AS is_autogrow_all_files
			, NULL AS is_persistent_log_buffer
		FROM	sys.databases d
				INNER JOIN sys.master_files mf
					ON d.database_id = mf.database_id
				INNER JOIN sys.database_files df
					ON mf.file_id = df.file_id
					AND mf.name = df.name COLLATE Latin1_General_CI_AI
				LEFT OUTER JOIN sys.filegroups fg
					ON df.data_space_id = fg.data_space_id;';

		INSERT INTO #dbsettings ( DBName
								, DataFileName
								, physical_name
								, FileGroupName
								, is_mixed_page_allocation_on
								, is_autogrow_all_files
								, is_persistent_log_buffer )
		EXECUTE ( @TSQL );

		SELECT	db.DBName
			, db.DataFileName
			, db.physical_name
			, CASE WHEN db.is_mixed_page_allocation_on IS NULL
						AND oa.TraceFlag IS NULL THEN
						'Mixed Page Alloc Not Enabled'
				END AS Feature
			, CASE WHEN db.is_mixed_page_allocation_on IS NULL
						AND oa.TraceFlag IS NULL THEN
						'DBCC TRACEON (1118, -1);'
				END AS EnableStatement
		FROM	#dbsettings db
				OUTER APPLY ( SELECT	TraceFlag
								FROM	#tfstatus tf
								WHERE tf.TraceFlag = 1118 ) oa
		WHERE is_mixed_page_allocation_on IS NULL
			AND db.FileGroupName IS NOT NULL
		UNION
		SELECT	db.DBName
			, db.DataFileName
			, db.physical_name
			, CASE WHEN db.is_autogrow_all_files IS NULL
						AND oa.TraceFlag IS NULL THEN
						'Even Data File Growth Not Enabled'
				END AS Feature
			, CASE WHEN db.is_autogrow_all_files IS NULL
						AND oa.TraceFlag IS NULL THEN
						'DBCC TRACEON (1117, -1);'
				END AS EnableStatement
		FROM	#dbsettings db
				OUTER APPLY ( SELECT	TraceFlag
								FROM	#tfstatus tf
								WHERE tf.TraceFlag = 1117 ) oa
		WHERE db.is_autogrow_all_files IS NULL
			AND db.FileGroupName IS NOT NULL;

		SELECT	d.DBName
			, d.DataFileName
			, d.physical_name
			, d.is_mixed_page_allocation_on
			, d.is_autogrow_all_files
			, d.is_persistent_log_buffer
		FROM	#dbsettings d;
	END;

IF @ServerMajorVersion >= 13.00
	BEGIN
		SET @TSQL = 'SELECT	d.name AS DBName
			, df.name AS DataFileName
			, df.physical_name
			, fg.name as FileGroupName
			, d.is_mixed_page_allocation_on
			, fg.is_autogrow_all_files
			, mf.is_persistent_log_buffer
		FROM	sys.databases d
				INNER JOIN sys.master_files mf
					ON d.database_id = mf.database_id
				INNER JOIN sys.database_files df
					ON mf.file_id = df.file_id
					AND mf.name = df.name COLLATE Latin1_General_CI_AI
				LEFT OUTER JOIN sys.filegroups fg
					ON df.data_space_id = fg.data_space_id;';

		INSERT INTO #dbsettings ( DBName
								, DataFileName
								, physical_name
								, FileGroupName
								, is_mixed_page_allocation_on
								, is_autogrow_all_files
								, is_persistent_log_buffer )
		EXECUTE ( @TSQL );
		
		--just to check for giggles
		--SELECT	*
		--FROM	#tfstatus;

		SELECT	db.DBName
			, db.DataFileName
			, db.physical_name
			, CASE WHEN db.is_mixed_page_allocation_on = 1 THEN
						'Mixed Page Alloc Is Enabled'
				END AS Feature
			, CASE WHEN db.is_mixed_page_allocation_on = 1 THEN
						'ALTER DATABASE [' + db.DBName
						+ '] SET MIXED_PAGE_ALLOCATIONS OFF;'
				END AS EnableStatement
		FROM	#dbsettings db
				OUTER APPLY ( SELECT	TraceFlag
								FROM	#tfstatus tf
								WHERE tf.TraceFlag = 1118 ) oa
		WHERE is_mixed_page_allocation_on <> 0
			AND db.FileGroupName IS NOT NULL
		UNION
		SELECT	db.DBName
			, db.DataFileName
			, db.physical_name
			, CASE WHEN db.is_autogrow_all_files IS NULL
						AND oa.TraceFlag IS NULL THEN
						'Even Data File Growth Not Enabled'
				END AS Feature
			, CASE WHEN db.is_autogrow_all_files = 0 THEN
						'ALTER DATABASE [' + db.DBName
						+ '] MODIFY FILEGROUP [' + FileGroupName
						+ '] AUTOGROW_ALL_FILES;'
				END AS EnableStatement
		FROM	#dbsettings db
				OUTER APPLY ( SELECT	TraceFlag
								FROM	#tfstatus tf
								WHERE tf.TraceFlag = 1117 ) oa
		WHERE db.is_autogrow_all_files <> 1
			AND db.FileGroupName IS NOT NULL;

		SELECT	d.DBName
			, d.DataFileName
			, d.physical_name
			, d.FileGroupName
			, d.is_mixed_page_allocation_on
			, d.is_autogrow_all_files
			, d.is_persistent_log_buffer
		FROM	#dbsettings d;

		IF EXISTS ( SELECT		*
					FROM	#tfstatus
					WHERE TraceFlag IN ( 1117, 1118 ))
			BEGIN
				SELECT	'One or Both Trace Flags (1117 & 1118) is/are enabled and has NO EFFECT. Please Disable' AS Warning
					, 'DBCC TRACEOFF (1117, 1118, -1);' AS DisableStatement;
			END;

	END;

Holy cannoli batman – that is more than a simple script, right? Well, it may be a bit of overkill. I wanted it to work for version before and after and including SQL Server 2016 (when these sweeping changes went into effect). You see, I am checking for versions where the TF was required to make the change and also for versions after the change where the TF has no effect. In 2016 and later, these settings are database scoped and the TF is unnecessary.

The database scoped settings can actually be queried in 2016 more specifically with the following query.

SELECT	d.name AS DBName
			, df.name AS DataFileName
			, df.physical_name
			, fg.name as FileGroupName
			, d.is_mixed_page_allocation_on
			, fg.is_autogrow_all_files
			, mf.is_persistent_log_buffer
		FROM	sys.databases d
				INNER JOIN sys.master_files mf
					ON d.database_id = mf.database_id
				INNER JOIN sys.database_files df
					ON mf.file_id = df.file_id
					AND mf.name = df.name COLLATE Latin1_General_CI_AI
				LEFT OUTER JOIN sys.filegroups fg
					ON df.data_space_id = fg.data_space_id;

In this query, I am able to determine if mixed_page_allocations and if is_autogrow_all_files are enabled. These settings can be retrieved from sys.databases and sys.filegroups respectively. If I run this query on a server where the defaults were accepted during the install, I would see something like the following.

You can see here, the default settings on my install show something different than the reported behavior. While autogrow all files is enabled, mixed_page_allocations is disabled. This matches what we expect to see by enabling the Trace Flags 1117 and 1118 – for the tempdb database at least. If I look at a user database, I will find that mixed pages is disabled by default still but that autogrow_all_files is also disabled.

In this case, you may or may not want a user database to have all data files grow at the same time. That is a great change to have implemented in SQL Server with SQL 2016. Should you choose to enable it, you can do so on a database by database basis.

As for the trace flags? My query checks to see if maybe you enabled them on your instance or if you don’t have them enabled for the older versions of SQL Server. Then the script generates the appropriate action scripts and allows you to determine if you want to run the generated script or not. And since we are changing trace flags (potentially) I recommend that you also look at this article of mine that discusses how to audit the changing of trace flags. And since that is an XEvent based article, I recommend freshening up on XEvents with this series too!

The Wrap

In this article I have introduced you to some basics in regards to default behaviors and settings in tempdb along with some best practices. It is advisable to investigate from time to time some of these recommendations and confirm what we are really being told so we can avoid confusion and mis-interpretation.

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

Every now and then I find a topic that seems to fit perfectly into the mold of the theme of “Back to Basics”. A couple of years ago, there was a challenge to write a series of posts about basic concepts. Some of my articles in that series can be found here.

Today, my topic to discuss is in regards to altering the default logs directory location. Some may say this is no big deal and you can just use the default location used during install. Fair enough, there may not be massive need to change that location.

Maybe, just maybe, there is an overarching need to change this default. Maybe you have multiple versions of SQL Server in the enterprise and just want a consistent folder to access across all servers so you don’t have to think too much. Or possibly, you want to copy the logs from multiple servers to a common location on a central server and don’t want to have to code for a different directory on each server.

The list of reasons can go on and I am certain I would not be able to list all of the really good reasons to change this particular default. Suffice it to say, there are some really good requirements out there (and probably some really bad ones too) that mandate the changing of the default logs directory to a new standardized location.

Changes

The logs that I am referring to are not the transaction logs for the databases – oh no no no! Rather, I am referring to the error logs, the mini dumps, and the many other logs that may fall into the traditional “logs” folder during the SQL Server install. Let’s take a peek at a default log directory after the install is complete.

I picked a demo server that has a crap load of stuff available (and yeah not so fresh after install) but where the installation placed the logs by default. You can see I have traces, default XE files, some SQL logs, and some dump files. There is plenty going on with this server. A very fresh install would have similar files but not quite as many.

If I want to change the Log directory, it is a pretty easy change but it does require a service restart.

In SQL Server Configuration Manager, navigate to services then to “SQL Server Service”. Right click that service and select properties. From properties, you will need to select the “Startup Parameters” tab. Select the parameter with the “-e” and errorlog in the path. Then you can modify the path to something more appropriate for your needs and then simply click the update button. After doing that, click the ok button and bounce the SQL Service.

After you successfully bounce the service, you can confirm that the error logs have been migrated to the correct folder with a simple check. Note that this change impacts the errorlogs, the default Extended Events logging directory, the default trace directory, the dumps directory and many other things.

See how easy that was? Did that move everything over for us? As it turns out, it did not. The old directory will continue to have the SQL Agent logs. We can see this with a check from the Agent log properties like the following.

To change this, I can execute a simple stored procedure in the msdb database and then bounce the sql agent service.

USE msdb;
GO
EXECUTE dbo.sp_set_sqlagent_properties 
	@errorlog_file=N'D:\Logging\SQLAGENT.OUT'
GO

With the agent logs now writing to the directory verified after agent service restart as shown here.

At this point, all that will be left in the previous folder will be the files that were written prior to the folder changes and the service restarts.

The Wrap

In this article I have introduced you to an easy method to move the logs for SQL Server and the SQL Server Agent to a custom directory that better suits your enterprise needs. This concept is a basic building block for some upcoming articles – stay tuned!

This has been another post in the back to basics series. Other topics in the series include (but are not limited to): Backups, backup history and user logins.

What is this newfangled feature? Well, that is kind of the problem. The name has already changed in the second release – be it ever so slightly. The feature is XE Profiler or XEvents Profiler and in theory it is supposed to give us something new in management studio.

Fact or Fiction

This Extended Events Profiler is a brand spanking new feature (ok, so the bouncing baby is a few months old now and no longer a newborn). Despite it being new, it seems there are a handful of “facts” already published about it that may or may not be accurate. Due to that, I want to play a little game of fact or fiction with it in this article.

Here is the short list from which we will work:

1. Sessions (XE Profiler Sessions) are Customizable
2. The default standard session displays all Extended Events
3. It works with instances of SQL 2012 or greater only
4. XE Profiler Adds ability to quickly start/stop sessions
5. XE Profiler provides a new live view of the Event Session
6. XE Profiler provides a GUI for Extended Events
7. Templates were not available in XE until the release of XE Profiler
8. Similarly, XE Profiler also brought the Profiler templates with it as part of the feature release
9. A DBA can now do something that was not previously possible to do (quickly start a session)

That is a rather wealthy list of “facts” to be checked for validity if you ask me. Let’s go ahead and start diving into each of these. I will not be following the provided list order. Rather, I will be looking at items that seem appropriate to validate in my own special order. Rest assured tho, we will get through the list.

GUI

I don’t know how this is still a misconception about Extended Events, but it is certainly still out there. Suddenly XE Profiler comes along and people suddenly think that this tool has provided the GUI that everybody has been clamoring about since Extended Events was released in 2008 (yes almost 10 yrs ago).

I am not going to waste much time on this because it is a complete and total piece of fiction. The GUI has been readily and easily available since SQL Server 2012. In addition to that, there was a GUI that could have been installed for the older versions that was created by Jonathan Kehayias.

From SSMS 2014, here is a sample screenshot of the GUI wizard followed by a standard GUI screen.

As you can see, there is clearly a GUI prior to the release of SSMS 17.3.

Result: Fiction

Templates

This is probably the next easiest notion to prove or disprove. I have heard from more than a single source that templates are brand new with SSMS 17.3 (which is when XE Profiler was released).

I don’t really need to go very far or work very hard to prove this notion as a complete work of fiction. I have SSMS 2014, 2016, 17.0, 17.3, and 17.4 readily available to me. I am going to start with SSMS 2014.

If I  once again open the GUI for XEvents in SSMS 2014, I can easily see the available templates in that version.

It looks pretty obvious there that templates were introduced well before SSMS 17.3.

I can also see all of the templates available by browsing out to the file system.

And for good measure, here is one from a 2012 instance.

The same templates (minus possible changes within the template definitions) are available in SSMS 2012, 2014 and 2016. We will see those same templates available in SSMS 17 and above as well – in addition to a few that have been added.

I want to also add here that these are just the default templates. Just like in Profiler, you can create and save your own templates for future use.

Result: Fiction

Profiler Templates

We just saw how there are templates available prior to the release of SSMS 17.3. We can also see that the profiler templates are not available in those prior versions of SSMS. So does that also mean that the profiler templates were only released as a part of SSMS 17.3? We best be careful with that slippery slope of assumption. Let’s take a closer look at SSMS 17.0.

And just so we can confirm that this is indeed a version of SSMS that pre-dates 17.3…

As you can see there, this is RC3 of SSMS 17.0 so it is definitely a version that precedes 17.3

And similarly from a peek inside the folder on disk:

Seems to me we have ample evidence that the profiler templates were made available to the XE GUI well in advance of SSMS 17.3

Result: Fiction

Event Session Live View

One of the most widely used feature of profiler is the ability to watch the data as it streams through the session. According to many, this ability is missing in Extended Events and the XE Profiler feature has brought this ability to the forefront with XE now.

Let’s start by taking a look at an XE Session in SSMS 2014 connected to a 2014 database instance to check the validity of this concern.

All I need to do is browse the nodes in SSMS down to the session I want to view. Right click the session and then select the option to “Watch Live Data”.

After clicking watch live data, the event session live stream will appear on the right and start streaming the data to your SSMS workspace as shown in the following image.

For this session demo, I selected a session that should be readily available to everybody – the system_health default session. That session comes installed and running for you already.

Don’t like the layout of the screen? Well, you can customize the view to your liking and it can be customized differently for each and every different running session on your server.

As it turns out, the ability to watch livestream data has been available to you for Extended Events since 2012 when the GUI was made available. If you are curious about learning more about the GUI and customizing the view, you can read various articles I have written in my 60 day series you can find here.

Result: Fiction

That brings us to the halfway point. To recap we are now sitting at a score of Fact 0 and Fiction 4. Let’s see if we can turn the tide in these last 5.

Quick Start a Session

To be quite frank here, this has been a long time argument against using Extended Events. It just isn’t possible to start a session quickly and easily. I have two methods that I have shown in my training sessions concerning that concern. Let’s take a look at the first. I will demonstrate this from SSMS 2014.

The first thing that needs to be done is browse the SSMS tree until a session is found that you want to start. Here is an ancient cheat code – I leave many sessions on the server in the stopped state until I need to use them. Pro-tip: create your XE sessions on all of your servers and leave them stopped until you think you need to use it.

For this example, I have selected my AG_LeaseTimeout sessions which is in the stopped state (denoted by the little red “stop” icon next to the session name).

Then right click the session name and select “Start Session” from the context menu. To help clarify, I have circled the session name in green and the menu item in blue. This an extremely easy method to start a session. Once started, and if you really feel you must watch data stream onto the screen, then you can also follow the steps shown for watching the live stream data in the previous section.

The second method to quick start a session is only a touch more difficult. The steps are just the same as shown in the “template” section. You select a template that matches your desires and give the session a name. Here are some broad strokes shown in SSMS 17.0 (same steps apply for SSMS 2014 minus the ability to select a profiler template):

Right click the “Session” node and selection “New Session” from the context menu.

Select the “Standard” template from the dropdown box. This is a profiler equivalent (and also happens to be one of the two default sessions in “XE Profiler” so makes sense for a good comparison. Note that the window has a warning that the Session name cannot be empty, so let’s give the session a name.

With that warning cleared, we can now move on to the next configuration. Notice the two red arrows in the preceding image. Enabling these two options will start the session immediately and also give you that profiler warm and fuzzy by allowing you to immediately start watching the session without needing to lift another finger.

From here, click the OK button ( I know I normally teach people that they should script it, but in this case we are going for quick and easy). After clicking “ok”, be prepared for something like the following.

After just a few seconds on a busy server, one should expect thousands of events to fire in a very profiler type manner. I want to reiterate here, that this was the same session template that feeds one of the two available sessions in XE Profiler.

Once you have determined that you have enough data, you can easily stop the session by right clicking the session name and then selecting the “Stop Session” menu item.

And that is all. It is very easy to start a session in two different manners without XE Profiler. One of the options even allows you to create the exact same session as XE profiler but without that feature even being made available on the version of SSMS being used. So if you really want that profiler type of session, you can use a template to quickly create it even if your SSMS version pre-dates 17.3 and even if you don’t have the profiler templates by default (remember you can simply add the profiler templates should you choose).

In the end, I have to call this one a piece of fiction as well. The XE Profiler does not add any new ability to quickly start a session that wasn’t already a viable option. That said, the XE Profiler does add a third alternative to start a session – so long as it is one of the two default sessions.

Result: Fiction

SQL 2012 Required

This is one of those areas that I wish were fiction. However, this has actually been a consistent theme ever since the GUI was released for XE. The GUI components only work on 2012 instances and later. Let’s take a look at a connection to a 2014 Server and a 2008R2 Server in the same SSMS window from SSMS 17.0.

We can see here that the standard Extended Events GUI is just not available for the 2008R2 instance. The 2014 instance is denoted with a Green box and arrow to help distinguish between that and the red marks for the 2008R2 instance.

Now, let’s take a look at it with SSMS 17.3. With a similar setup and similar notation, we see the following in SSMS 17.3 when connected to both a 2014 instance and a 2008R2 instance.

Once again, I will admit this is something I really wish were not the case. It would definitely be helpful if the XE GUI were to be made available to SQL Server Instances that are 2008 and 2008R2. Especially given that the tool is an “SSMS feature”. That said, it really behooves everybody to get their SQL Servers updated to current technologies (2008 is nearly 10 years old afterall).

Result: Fact

The default “Standard” session displays all Extended Events

So far, we have been easing ourselves into this XE Profiler very gently. It is time to finally get into the deeper end and actually look at some very specific XE Profiler things. When I first heard this one, I was admittedly flabbergasted. One session displays all extended events? Really? Let’s actually establish some baselines about the default sessions for XE Profiler first.

1. There are only two default sessions for XE Profiler
2. The session names are “Standard” and “TSQL”
3. The default sessions are based on templates (discussed previously)
4. The template for the “Standard” session is the xe_Profiler_Standard xml template file
5. The template for the “TSQL” session is xe_Profiler_TSQL

If I peruse the template file located in the directory on disk (C:\Program Files (x86)\Microsoft SQL Server\140\Tools\Templates\sql\xevent) for the standard session, I will be able to see that it contains the following events:

<event package="sqlserver" name="attention">
<event package="sqlserver" name="sql_batch_starting">
<event package="sqlserver" name="sql_batch_completed">
<event package="sqlserver" name="rpc_completed">
<event package="sqlserver" name="existing_connection">
<event package="sqlserver" name="logout">
<event package="sqlserver" name="login">

Now, that is not the entire template file. I shorted it to only include the events and none of the other details like session configuration or the actions attached to each event. As you can see, there is a grand total of seven events in this particular session.

If I run the following statement against a SQL Server 2014 instance, I will get a few more events than just seven.

SELECT SERVERPROPERTY('ProductMajorVersion') AS SvrVersion,COUNT(*) AS EventCount
	FROM sys.dm_xe_objects xo
	WHERE xo.object_type = 'event';

And the results:

And just for the sake of clarity, the results on a 2017 instance:

Obviously the math does not add up. There are multiple things wrong with the notion that a session from XE Profiler will contain all events. First, there is no sense at all in adding every event to a single event session. The second problem is that the session is supposed to mimic a profiler default template session. If it is supposed to mimic a profiler style session, you can only use 180 events max anyway. Profiler only had 180 events and XE is obviously way beyond that with so much more feature support than Profiler could even muster back in its glory days.

In case, you were wondering, here is how you access the XE Profiler. It is not under the Extended Events Node, it is a brand new node.

As you can see, there really are only the two options for XE Profiler and if you recall, both of those options are derived from the templates that were made available in the traditional XE GUI and are very easy to create a quick session via the same templates as these two options.

Whether using the XE Profiler or the previous method to create these sessions, there is just no way that these sessions could contain every event in Extended Events. It’s just a bunch of hot air with no foundation to the claim.

Result: Fiction

XE Profiler adds ability to start/stop session

Let’s start back with the last image from the previous section and then go from there. We need to access what is available to us via the XE Profiler node so we can test this theory. I will start by right clicking one of the sesisons (right clicking on the “XE Profiler object itself will not yield anything very promising fwiw).

As you can see here, we are given some pretty basic options to either “Stop Session” or to “Launch Session”. It also seems very obvious what they want you to do here because the option is bolded – “Launch Session”.

This menu is the same regardless of the session state. What I mean by that is even if you do not have the session created, this menu is the same as if you have the session created and running. None of the options dim and all are clickable on this menu. This holds true for SSMS 17.4 as well. Let’s take a look at a server where none of these “profiler” sessions are running.

I want to make it abundantly clear. This is the way it appears after a completely fresh install of SSMS 17.3. None of these sessions have ever run on this particular instance of SQL Server 2017 and this is even a completely fresh install of SQL Server 2017. That said, I have done nothing yet to break it – buahaha.

Knowing that no sessions have ever been “launched” for XE Profiler, it is clear that it is time to go ahead and “launch” a session. In this new feature “launch” is the new equivalent of “start”. Once you launch a session you will get the same net effect as what we saw earlier when I created a session from the profiler template and checked the box to watch live data and the box to start the session. When I click launch, I will see something like the following:

A new XE session was created under the Extended Events node and it is now called “QuickSessionStandard”. Once again, this is created from that profiler template I have mentioned a few times.

And then after a bit, I may want to try and stop the session.

You can see here that I was able to capture some quick data into the live stream view (which was a feature that was already there prior to 17.3). And after clicking the “stop session” menu item, I will see the following.

Note that the “QuickSessionStandard” event session now has a “Stop” icon instead of a “play” icon. So yay – I was able to start and stop a session quickly from XE Profiler. In addition, XE profiler launched a live stream view just like we saw earlier when creating the same session from the template in the XE GUI. Now for a pro-tip – start and stop an XE Session from the old GUI – “right click any event session and select start/stop from the context menu.”

Yes it is that easy. In addition, only the available option for that session will be enabled (e.g. you cannot stop a session that is already stopped – or not running). In fact you can even start/stop from TSQL. If you try to stop a session that is not running in TSQL, you will get a warning message that it is already stopped. Far more intuitive than XE Profiler. You will get no indication that the session is unable to be stopped if it is already stopped. Let’s take a look at the old way.

And after the session is started, I can also quickly stop it.

You can also see from each of those images, that I have the option to watch live data for the session. Another easy access feature from the old XE GUI.

While XE Profiler has added an additional means to be able to start/stop sessions quickly, the fact of the matter is that it is not a new feature brought to the table by XE Profiler. All it did was add a different way of doing it. I will also add that this new method can actually be somewhat confusing as well. You can only determine (visually) if the session is running or stopped by expanding the “Extended Events” node. Stopping a session from the XE Profiler node does not close the livestream viewer so you could easily presume it is still running. The XE Profiler node gives you no insight into the actual state of the session.

And for giggles, this is how easy it is to start or stop a session via TSQL.

ALTER EVENT SESSION QuickSessionStandard on SERVER
STATE = START;
GO

ALTER EVENT SESSION QuickSessionStandard on SERVER
STATE = STOP;
GO

If I run the statement to stop the session when it is already stopped, I will get the following:

Msg 25704, Level 16, State 1, Line 9
The event session has already been stopped.

I have previously covered this specific topic (start/stop sessions) in my 60 Day series which can be found here.

Result: Fiction

Customizable XE Profiler Sessions

Let’s take a gander at the previous section where I showed the context menu available in XE Profiler and not the extent of that menu. The profiler iteself does not offer any sort of customization. You have only two default sessions. You cannot change either session within that particular feature.

Since I cannot customize anything for these sessions via the XE Profiler, let’s use the XE GUI that is tried and true. I will effectively perform the following via the XE Session Properties window:

ALTER EVENT SESSION [QuickSessionStandard] ON SERVER 
DROP EVENT sqlserver.attention, DROP EVENT sqlserver.logout
GO

And after I am done, that session will look like the following:

CREATE EVENT SESSION [QuickSessionStandard] ON SERVER 
ADD EVENT sqlserver.existing_connection(SET collect_options_text=(1)
    ACTION(package0.event_sequence,sqlserver.client_app_name,sqlserver.client_pid,sqlserver.nt_username,sqlserver.server_principal_name,sqlserver.session_id)),
ADD EVENT sqlserver.login(SET collect_options_text=(1)
    ACTION(package0.event_sequence,sqlserver.client_app_name,sqlserver.client_pid,sqlserver.nt_username,sqlserver.server_principal_name,sqlserver.session_id)),
ADD EVENT sqlserver.rpc_completed(
    ACTION(package0.event_sequence,sqlserver.client_app_name,sqlserver.client_pid,sqlserver.database_id,sqlserver.nt_username,sqlserver.query_hash,sqlserver.server_principal_name,sqlserver.session_id)
    WHERE ([package0].[equal_boolean]([sqlserver].[is_system],(0)))),
ADD EVENT sqlserver.sql_batch_completed(
    ACTION(package0.event_sequence,sqlserver.client_app_name,sqlserver.client_pid,sqlserver.database_id,sqlserver.nt_username,sqlserver.query_hash,sqlserver.server_principal_name,sqlserver.session_id)
    WHERE ([package0].[equal_boolean]([sqlserver].[is_system],(0)))),
ADD EVENT sqlserver.sql_batch_starting(
    ACTION(package0.event_sequence,sqlserver.client_app_name,sqlserver.client_pid,sqlserver.database_id,sqlserver.nt_username,sqlserver.query_hash,sqlserver.server_principal_name,sqlserver.session_id)
    WHERE ([package0].[equal_boolean]([sqlserver].[is_system],(0))))
WITH (MAX_MEMORY=8192 KB,EVENT_RETENTION_MODE=ALLOW_SINGLE_EVENT_LOSS,MAX_DISPATCH_LATENCY=5 SECONDS,MAX_EVENT_SIZE=0 KB,MEMORY_PARTITION_MODE=PER_CPU,TRACK_CAUSALITY=ON,STARTUP_STATE=OFF)
GO

And visually, from the GUI, it will look like this:

You can see that the aforementioned events are definitely gone from this session. From here, I can absolutely start that session from the XE GUI and it will run just fine. I can hit “watch live data” from the context menu and it will open up the previous view of the live data that was used (so the exact same configuration of that window because the XE GUI has always done that). If I double check my session after it is started, I will be able to confirm that the altered session is indeed still tact just the same as I customized it.

Let’s go ahead and stop that and then “launch” it from the XE Profiler now. After the “launch” I will re-script the session to verify that it is either the same or altered from what I had configured. In this case, since the session was already in place, I will find that the XE Profiler did not alter my session in any way and merely started the session. That is a win for XE Profiler in that it was able to determine that the session did indeed exist. However, I have to use the XE GUI and not XE Profiler in order to get a decent config on those default sessions.

Result: Fiction

Bonus time

XE Profiler only has SQL Profiler Events

This is actually an interesting statement. While the XE Profiler relies on templates based on SQL Profiler and the events in those templates are events that are available in SQL Profiler, they are not exclusive to SQL Profiler. These events can be used in Extended Events and have been a part of the XE Engine since inception. So, I would call this one fiction as well. I think an obvious litmus test for that is that the events are actually running through an XE Session so they must obviously be XE events as well.

Result: Fiction

Up to this point I have not addressed any changes in the Extended Events Profiler that were made for SSMS 17.4. Here is where it seems we compound a bit of confusion for the masses. Not only do we have this Extended Events tool that we have decided to call “Profiler” which has already had many people asking me about it because the name confused them (is it profiler or is it XE??), but it seems that the identity crisis for this feature is exacerbated slightly with a sudden name change.

Using yet another very clean install of SSMS on a different instance of SQL Server to ensure that none of the XE Profiler components had been used or any of the default sessions created and launched, I have the following:

Yes, the name change is very innocuous but it can lead to some confusion for various parties involved. If a set of step by step instructions says to do something and the learner cannot find that explicit node, they will become confused.

For anybody that has been working with XE for some time, they will know that XE is synonymous with XEvent and will quickly figure it out.

Conclusion

So, in wrapping up this very long article, I want to recap the score of the Fact v. Fiction items. The final tally is Fact 1 and Fiction 9 (9 original list items plus 1 bonus entry). This means that there is a whole lot of stuff going on around out there about this new feature that is just not accurate. While some may be able to derive some small use from the XE Profiler, it really does not add anything that you could not already do with either TSQL or the old XE GUI.

Some say this is a way of bridging the gap. In my opinion, that gap was already bridged with the GUI that has been available for several years. Some say that maybe this tool needs to integrate a way to shred XML faster. To that, I say there are methods already available for that such as Powershell, the live data viewer, the Target Data viewer, or even my tools I have provided in the 60 day series.

I would challenge those that are still unfamiliar with the XE GUI (out for nearly 6 years now) to go and read some of my articles or articles by Jonathan Kehayias about the power that is in XE as well as some of the power in the GUI.

The undead of Halloween are now upon us. Among the hordes of data zombies roaming the streets in lab-coats and fishnet stockings, few of us are still scrambling to remove the hexes we have looming over our data.

As chance would have it, these hexidecimals, err hexes, have a more profound effect on us than we first thought. Many may have yet to even recognize the impact of the hexes placed along with the monsters that now lurk in the data after having been summoned via those hexes.

DB and Fun Related

Seeing as I am a really big fan of this holiday I have a few Halloween posts over the years. If you are interested in the previous Halloween posts, here is a list of a few of them:

All Halloween posts

That list is my Halloween treat this year. Now for a bit of a trick with a very strong warning. Because of this warning, I am not posting any code showing how to perform the trick.

Warning

The contents of this post are for the intent of HUMOR!

Freddy Kreuger

I can hear you clamoring from half a world away right now – “There is nothing called ‘Freddy Kreuger’ in SQL Server. What in the world are you talking about?”

You would be very accurate in your exclamation there. But this is not necessarily a strict exercise in feature names within SQL Server. I want you to think a little further outside the norms for a while.

Do you currently or have you ever needed to shred XML? XML shredding via TSQL can be a monstrously bloody killer to your database performance. As it turns out, Mr. Kreuger was also a monstrously bloody shredder.

Jason Voorheese

Yet another beast that is not truly in SQL Server, or is it? A not so new but new feature in SQL Server is called JSON. This feature does actually perform better than XML in some regards. That said, we do have a very common problem between the two of these features – blobs.

If you are not familiar with what that means -here you go. A blob is an overly large item being stored in the database. If you wish, you could correlate that to the other well known Halloween beast – “The Blob”.

Over time, this blob acts like sludge and just slows down your database queries. In addition, like the creature, the blob in your database tends to continue to grow in size and is seemingly never able to be put in check.

Skeletons

When I find skeletons, I have to be honest, I don’t find them terribly frightening. When talking about skeletons in your database, I am even less frightened.

Then again, when I run into the situation as described recently, in this post, I may get a bit of a startle and get just a wee bit concerned.

Overall though, I am rarely startled or frightened by any skeletons in the database. These are really just the supporting structures of a nice secure database and are called “schemas”. See, not really all that frightening here if we think about it just a bit.

This next one however, might be a little harder and should be nearly enough to cause some heart pain.

Warlocks

Surely there are no wizarding type of people in the database, right? Warlocks? I know for absolute certainty that there is no such feature or anything remotely close to a warlock, witch or wizard within the database. That is unless my database is about mystical creatures and people.

Alas, I urge you again to expand the box of perception a little bit and become just a tiny bit imaginative. This one, truth be told, does require a fair amount of explanation and imagination though.

The problem comes in part from some magical data issues that can occur due to this particular feature. In addition, this also comes from the wonderful grammatical errors from various blog posts and forums out there mis-spelling “which” as “witch”. Since “manwich” is really close to “man-witch”, I am calling it a warlock.

Now, since I am calling it a warlock, that leads us to the next strong hint about the feature. “Lock” in this case is the key. Now which magical, imaginative feature might there be that is related to “lock”? That would be the “nolock” directive and all of the data quality issues that it presents. Here is a really really good recap (by Aaron Bertrand) on this feature along with reference to it being “magic” – at this site.

Pirates

This is probably the easiest of the day by far. For all the data loving geeks out there, SQL Server has this pirate flavored way for you to get your drool on. This feature is called “R”. Yup – just like what a pirate says matey.

R is a tool to be used by data scientists or data geeks in general to try and throw together many different flavors of statistical analysis about your data.

Split Brain

Finally, (at least for this Halloween) we have this condition that is real within SQL Server. While treatable with long hours and heavy medication, it is something to be feared.

This condition is something rare but it is very real. The split brain syndrome is pretty much a multiple identity personality disorder in your database. If you have multiple nodes in a cluster, mirror or availability group, it is possible for more than one of those nodes to believe it is the master node and then for different transactions to become hardened in each of those nodes.

When this happens, you will not be able to use bleach to clean up the mess. Instead, you will be required to spend a grundle of time with your database cuddling it and nursing it back to data consistency and good mental health.

Last but not least, HAPPY HALLOWEEN!

Back in late December of 2015, a challenge of sorts was issued by Tim Ford (twitter) to write a blog post each month on a SQL Server Basic. Some have hash-tagged this as #backtobasics. Here is the link to that challenge sent via tweet.

While the challenge may have been for the year 2016 and I haven’t contributed in several months, it is still a worthwhile effort. In that vain, I am adding this article to that series.

With this being another installment in a monthly series, here is a link to review the other posts in the series – back to basics. Reviewing that link, you can probably tell I am a bit behind in the monthly series.

Endpoints

You may have heard the term endpoints thrown around in technical discussion and wondered “what the heck is an endpoint?” Well, that is a good question. An endpoint in the simplest form is a connection or point of entry into SQL server.

Another way of thinking about an endpoint is to look at the ends of a line. Generally speaking, there is a stop point at each end of the line. At this stopping point, the line may connect to something else and therefore be a point of entry into that “something else”.

When we deal with SQL Server there is a handful of default endpoints and then a handful of other types of endpoints. To figure out what the default endpoints are, it is pretty easy. The following query will expose the default endpoints.

USE master;
GO
SELECT e.name, e.protocol_desc, e.type_desc
	FROM sys.endpoints e
	WHERE e.endpoint_id < 65536;

Executing that query will produce results similar to the following:

Some of those might look pretty straight forward and then you get to that VIA endpoint. Don’t worry about that endpoint. The VIA endpoint is on the deprecation list. All of these default endpoints will be owned by sa and don’t require you to do anything with them per se other than to understand they exist. You may have noticed that I filtered my results by looking only at endpoints with an id of less than 65536. Any endpoint id lower than this number is a default endpoint. All others are user defined endpoints.

Endpoint Owners

So far so good. This is pretty straight forward to this point. If you have implemented anything like mirroring, Availability Groups, or Service Broker, then you may be interested to know that you have also created additional endpoints in the Instance. When you create an endpoint, did you know that you become the owner of that endpoint by default? It is very similar to when you restore a database or create a database – the default owner of that database will be the person that restored/created it.

Do you know who owns your endpoints? Did you create all of the endpoints? Do you know if you have any additional endpoints beyond the default endpoints? If you cannot answer yes to all of these questions, then you will probably want to figure out what the endpoints are and who owns those endpoints. Let’s try that with a slight modification to the previous query.

USE master;
GO
SELECT ISNULL(e.name, 'Unused PayloadType') AS EndpointName
		, e.endpoint_id
		, sp.name AS EndpointOwner
		, CASE	WHEN e.protocol IS NULL
						AND oa.typeid = 1 THEN 1
				WHEN e.protocol IS NULL
						AND oa.typeid = 3 THEN 2
				WHEN e.protocol IS NULL
						AND oa.typeid = 4 THEN 2
				ELSE e.protocol
			END AS protocolid
		, CASE WHEN e.protocol IS NULL
						AND oa.typeid = 1 THEN  'HTTP'
			WHEN e.protocol IS NULL
						AND oa.typeid = 3 THEN 'TCP'
			WHEN e.protocol IS NULL
						AND oa.typeid = 4 THEN 'TCP'
			ELSE e.protocol_desc
			END AS protocol_desc
		, oa.typeid
		, oa.PayloadType
		, e.state
		, e.state_desc
		, e.is_admin_endpoint
	FROM sys.endpoints e
		INNER JOIN sys.server_principals sp
			ON e.principal_id = sp.principal_id
		RIGHT OUTER JOIN ( VALUES ( 1, 'SOAP'), ( 2, 'TSQL'),
				( 3, 'SERVICE_BROKER'), ( 4, 'DATABASE_MIRRORING') ) AS oa ( typeid, PayloadType )
			ON oa.typeid = e.type;

This will yield results similar to the following:

I took the script a step further than necessary. I wanted to illustrate potential endpoint types that may not be in use. This script covers both in use endpoint types and those not used. In my results you may note that I have a DATABASE_MIRRORING endpoint. As reality would actually have it, it is an Availability Group endpoint but those are presented as DATABASE_MIRRORING endpoints.

Note that my mirroring endpoint (aka Hadr_endpoint) is owned by “YourDomain\DBAdmin”. What if the owner of that particular endpoint was no longer present in the organization and I wanted to change it to something more sustainable? Well, I could do the following:

USE master;
ALTER AUTHORIZATION ON ENDPOINT::Hadr_endpoint TO sa;

In this case, the default endpoints are owned by sa and it does make enough sense to assign the owner to be sa for the mirroring endpoint. Take notice that the name of the endpoint is required in order to reassign the owner. The name of the endpoint follows the :: in the script. So, whatever your endpoint name happens to be, just place “Hadr_endpoint” that follows the :: in my script.

Recap

Endpoints are a fundamental piece of the puzzle with SQL Server.  Getting to know your endpoints and the owners of those endpoints is an essential component of knowing your environment. Who knows, it may come to pass that the owner of an endpoint may no longer exist in your environment or possibly lose permissions along the way. Knowing who owns the endpoint may just save three or four grey hairs when that day comes.

Over the past couple of articles I have illustrated some of the fun that can be had when dealing with the scheduling capabilities of Reporting Services (SSRS). The first article covered how to create more advanced schedules (from the SSRS point of view). In another I article, I showed how to retrieve scheduling information from the ReportServer database. In that last article, I also promised a follow-up article for more in-depth scheduling details.

SSRS provides the capability to review the scheduled reports (subscriptions) in a far moare detailed fashion than shown in that previous article. That ability is held within the ReportServer database. This article will dive into the source of this scheduling information within the ReportServer database.

This dive will be a bit more detailed than the first time I dove into SSRS scheduling – here. That particular dive was missing an important set of data.

Deeper Dive

My first dive into building a report of the report schedules was pretty comprehensive and I used it quite frequently. Many others also used it regularly for their environments as well. So the first attempt wasn’t bad by any stretch. The main problem (at least for now) is that the script does not account for any of the custom schedules that can be built. I have to be honest in that I hadn’t really considered that feasibility. Times and experience change that perspective. When that perspective changes, it is time to dive back in and add coverage for the shortcoming in the script.

When I dove back in to fetch the custom scheduling information, I realized there was a discrepancy even in the old report in that I was not gathering Job information pertinent to the schedule. Recall that SSRS subscriptions are performed via the SQL Agent. With that in mind, it is reasonable that the Agent job information is pertinent and germane to the report subscription and probably should be included in a report. Couple that with the methods on creating custom schedules for the SSRS reports, and we have a resounding need to ensure that data is trapped properly.

Due to this epiphany, I have now a more complete script to include both the data from SQL Agent as well as the data from the ReportServer database in regards to subscriptions and schedules of reports.

Script

In pulling the data together from the two sources, I opted to return two result sets. Not just two disparate result sets, but rather two result sets that each pertained to both the agent job information as well as the ReportServer scheduling data. For instance, I took all of the subscriptions in the ReportServer and joined that data to the job system to glean information from there into one result set. And I did the reverse as well. You will see when looking at the query and data. One of the reasons for doing it this way was to make this easier to assimilate into an SSRS style report.

/* union the ssrs schedule query with the sql agent schedule query to get all 
report schedules
*/

IF OBJECT_ID('tempdb..#morepower') IS NOT NULL DROP TABLE #morepower;
IF OBJECT_ID('tempdb..#ReportJobScheds') IS NOT NULL DROP TABLE #ReportJobScheds;
IF OBJECT_ID('tempdb..#SSRSScheds') IS NOT NULL DROP TABLE #SSRSScheds;

--DROP TABLE #morepower;
--DROP TABLE #ReportJobScheds;
--DROP TABLE #SSRSScheds;

DECLARE @ReportName VARCHAR(100)
SET @ReportName = NULL;

CREATE TABLE #morepower (MonthDate BIGINT,N BIGINT,PowerN BIGINT PRIMARY KEY CLUSTERED
							,NameofMonth VARCHAR(25),WkDay VARCHAR(25))
;

WITH powers(powerN, n) AS (
	SELECT POWER(2,number), number 
		FROM master.dbo.spt_values 
		WHERE type = 'P' AND number < 31)

INSERT INTO #morepower ( MonthDate ,N,PowerN ,NameofMonth ,WkDay)
	SELECT ROW_NUMBER() OVER (ORDER BY N) AS MonthDate,N,PowerN
			,CASE WHEN N BETWEEN 0 AND 11 
				THEN DateName(month,DATEADD(month,N+1,0)-1)
				ELSE NULL
				END AS NameofMonth
			,CASE WHEN N BETWEEN 0 AND 6
				THEN DATENAME(weekday,DATEADD(DAY,n+1,0)-2)
				ELSE NULL
				END AS WkDay
		FROM powers

SELECT DISTINCT 
		ReportScheduleID = s.ScheduleID
		,Ca.Path as ReportManagerPath,Ca.Name as ReportName
		,U.UserName as SubscriptionCreator
		,PrimaryScheduleSource = 'SSRS Catalog'
		,Su.Description as SubscriptionDescription
		,SSRSScheduleStartDate = S.StartDate
		,Su.LastRunTime
		,CASE 
				WHEN s.RecurrenceType = 1 THEN 'Once'
				WHEN s.RecurrenceType = 2 THEN 'Hourly'
				WHEN s.RecurrenceType = 3 THEN 'Daily'
				WHEN s.RecurrenceType = 4 THEN 'Weekly'
				WHEN s.RecurrenceType = 5 THEN 'Monthly' 
				WHEN s.RecurrenceType = 6 THEN 'Week of Month' 
			END AS RecurrenceType
		, [Sched Enabled] = CASE su.InactiveFlags
								WHEN 0 THEN 'Yes'
								ELSE 'No'
							END
		,s.EventType
		,ISNULL(REPLACE(REPLACE(STUFF(
					(Select ', ['+CONVERT(VARCHAR(20),MonthDate)+']' AS [text()] 
						FROM #morepower m1 
						WHERE m1.powerN < s.DaysofMonth+1 
							AND s.DaysofMonth & m1.powerN >0 
						ORDER BY N FOR XML PATH(''), TYPE).value('.','VARCHAR(MAX)')
			   , 1, 2, ''),'[',''),']','')
			,'N/A') AS DaysofMonth
		,ISNULL(c1.NameOfMonth,'N/A') AS MonthString
		,ISNULL(c2.WkDays,'N/A') AS DaysofWeek
		,CASE MonthlyWeek
				WHEN 1 THEN 'First'
				WHEN 2 THEN 'Second'
				WHEN 3 THEN 'Third'
				WHEN 4 THEN 'Fourth'
				WHEN 5 THEN 'Last'
				ELSE 'N/A'
			END AS MonthlyWeek
		,ISNULL(CONVERT(VARCHAR(10),s.DaysInterval),'N/A') AS DaysInterval
		,ISNULL(CONVERT(VARCHAR(10),s.MinutesInterval),'N/A') AS MinutesInterval
		,ISNULL(CONVERT(VARCHAR(10),s.WeeksInterval),'N/A') AS WeeksInterval
	into #SSRSScheds
	FROM #morepower mp, dbo.Schedule s
		INNER JOIN ReportSchedule RS
			ON S.ScheduleID = RS.ScheduleID
		INNER JOIN Catalog Ca
			ON Ca.ItemID = RS.ReportID
		INNER JOIN Subscriptions Su
			ON Su.SubscriptionID = RS.SubscriptionID
		INNER JOIN Users U
			ON U.UserID = S.CreatedById
			OR U.UserID = Su.OwnerID
	CROSS APPLY (Select s.ScheduleID,REPLACE(REPLACE(STUFF(
							(SELECT ', ['+ NameofMonth + ']' AS [text()] 
								FROM #morepower m1 ,dbo.Schedule s1
								WHERE m1.NameofMonth IS NOT NULL 
									AND m1.powerN & s1.Month >0 
									AND s1.ScheduleID = s.ScheduleID
								ORDER BY N FOR XML PATH(''), TYPE).value('.','VARCHAR(MAX)')
							, 1, 2, ''),'[',''),']','') AS NameOfMonth)c1
	CROSS APPLY (SELECT s.ScheduleID,REPLACE(REPLACE(STUFF(
							(SELECT ', [' + WkDay + ']' AS [text()] 
								FROM #morepower m1 ,dbo.Schedule s2
								WHERE m1.WkDay IS NOT NULL 
									AND DaysOfWeek & m1.powerN >0
									AND  s2.ScheduleID = s.ScheduleID
								ORDER BY N FOR XML PATH(''), TYPE).value('.','VARCHAR(MAX)')
							, 1, 2, ''),'[',''),']','') AS WkDays) c2
	WHERE Ca.Name = ISNULL(@ReportName,Ca.Name);



SELECT [Job ID] = job.job_id
		, [Job Name] = job.name
		, [Job Enabled] = CASE job.enabled
							WHEN 1 THEN 'Yes'
							WHEN 0 THEN 'No'
							END
		, [JobSchedID] = sched.schedule_id
		, PrimaryScheduleSource = 'SQL Agent Job Catalog'
		, [RecurrenceType] = CASE sched.freq_type
								WHEN 1 THEN 'Once'
								WHEN 4 THEN 'Daily'
								WHEN 8 THEN 'Weekly'
								WHEN 16 THEN 'Monthly'
								WHEN 32 THEN 'Monthly relative'
								WHEN 64 THEN 'When SQLServer Agent starts'
								END
		, [Sched Enabled] = CASE sched.enabled
								WHEN 1 THEN 'Yes'
								WHEN 0 THEN 'No'
							END
		, EventType = 'SharedSchedule'
		, CASE sched.freq_type 
			WHEN 1 THEN ''
			WHEN 4 THEN convert(VARCHAR,sched.freq_interval) + ' days'
			WHEN 8 THEN c2.WkDays
			WHEN 16 THEN 'Day of Month'
			WHEN 32 THEN case sched.freq_interval
								WHEN 1 THEN 'Sunday'
								WHEN 2 THEN 'Monday'
								WHEN 3 THEN 'Tuesday'
								WHEN 4 THEN 'Wednesday'
								WHEN 5 THEN 'Thursday'
								WHEN 6 THEN 'Friday'
								WHEN 7 THEN 'Saturday'
								WHEN 8 THEN 'Day'
								WHEN 9 THEN 'Weekday'
								WHEN 10 THEN 'Weekend Day'
								END
			WHEN 64 THEN ''
			WHEN 128 THEN ''
			END as [Frequency Interval]
		, [Relative Interval] = CASE sched.freq_relative_interval
									WHEN 1 THEN 'First'
									WHEN 2 THEN 'Second'
									WHEN 4 THEN 'Third'
									WHEN 8 THEN 'Fourth'
									WHEN 16 THEN 'Last'
									ELSE NULL
								END
		, sched.freq_recurrence_factor
		, [Next Run Date] = convert(datetime,CASE next_run_date
								WHEN 0 THEN NULL
								ELSE SUBSTRING(CONVERT(VARCHAR(15), next_run_date),
												1, 4) + '/'
										+ SUBSTRING(CONVERT(VARCHAR(15), next_run_date),
													5, 2) + '/'
										+ SUBSTRING(CONVERT(VARCHAR(15), next_run_date),
													7, 2)
							END + ' ' + CASE LEN(next_run_time)
								WHEN 1
								THEN CAST('00:00:0' + RIGHT(next_run_time, 2) AS CHAR(8))
								WHEN 2
								THEN CAST('00:00:' + RIGHT(next_run_time, 2) AS CHAR(8))
								WHEN 3
								THEN CAST('00:0' + LEFT(RIGHT(next_run_time, 3),
														1) + ':'
										+ RIGHT(next_run_time, 2) AS CHAR(8))
								WHEN 4
								THEN CAST('00:' + LEFT(RIGHT(next_run_time, 4),
														2) + ':'
										+ RIGHT(next_run_time, 2) AS CHAR(8))
								WHEN 5
								THEN CAST('0' + LEFT(RIGHT(next_run_time, 5), 1)
										+ ':' + LEFT(RIGHT(next_run_time, 4), 2)
										+ ':' + RIGHT(next_run_time, 2) AS CHAR(8))
								WHEN 6
								THEN CAST(LEFT(RIGHT(next_run_time, 6), 2) + ':'
										+ LEFT(RIGHT(next_run_time, 4), 2) + ':'
										+ RIGHT(next_run_time, 2) AS CHAR(8))
							END)
		, [Max Duration] = CASE LEN(run_duration)
								WHEN 1
								THEN CAST('00:00:0' + CAST(run_duration AS CHAR) AS CHAR(8))
								WHEN 2
								THEN CAST('00:00:' + CAST(run_duration AS CHAR) AS CHAR(8))
								WHEN 3
								THEN CAST('00:0' + LEFT(RIGHT(run_duration, 3),
														1) + ':'
										+ RIGHT(run_duration, 2) AS CHAR(8))
								WHEN 4
								THEN CAST('00:' + LEFT(RIGHT(run_duration, 4), 2)
										+ ':' + RIGHT(run_duration, 2) AS CHAR(8))
								WHEN 5
								THEN CAST('0' + LEFT(RIGHT(run_duration, 5), 1)
										+ ':' + LEFT(RIGHT(run_duration, 4), 2)
										+ ':' + RIGHT(run_duration, 2) AS CHAR(8))
								WHEN 6
								THEN CAST(LEFT(RIGHT(run_duration, 6), 2) + ':'
										+ LEFT(RIGHT(run_duration, 4), 2) + ':'
										+ RIGHT(run_duration, 2) AS CHAR(8))
							END
		, [Subday Frequency] = CASE ( sched.freq_subday_interval )
									WHEN 0 THEN 'Once'
									ELSE CAST('Every '
											+ RIGHT(sched.freq_subday_interval,
													2) + ' '
											+ CASE ( sched.freq_subday_type )
												WHEN 1 THEN 'Once'
												WHEN 4 THEN 'Minutes'
												WHEN 8 THEN 'Hours'
												END AS CHAR(16))
								END
		, [Sched End Date] = sched.active_end_date
		, [Sched End Time] = sched.active_end_time
		, [AgentScheduleStartDate] = convert(datetime,CASE sched.active_start_date
								WHEN 0 THEN NULL
								ELSE SUBSTRING(CONVERT(VARCHAR(15), sched.active_start_date),
												1, 4) + '/'
										+ SUBSTRING(CONVERT(VARCHAR(15), sched.active_start_date),
													5, 2) + '/'
										+ SUBSTRING(CONVERT(VARCHAR(15), sched.active_start_date),
													7, 2)
							END + ' ' + CASE LEN(sched.active_start_time)
								WHEN 1
								THEN CAST('00:00:0' + RIGHT(sched.active_start_time, 2) AS CHAR(8))
								WHEN 2
								THEN CAST('00:00:' + RIGHT(sched.active_start_time, 2) AS CHAR(8))
								WHEN 3
								THEN CAST('00:0' + LEFT(RIGHT(sched.active_start_time, 3),
														1) + ':'
										+ RIGHT(sched.active_start_time, 2) AS CHAR(8))
								WHEN 4
								THEN CAST('00:' + LEFT(RIGHT(sched.active_start_time, 4),
														2) + ':'
										+ RIGHT(sched.active_start_time, 2) AS CHAR(8))
								WHEN 5
								THEN CAST('0' + LEFT(RIGHT(sched.active_start_time, 5), 1)
										+ ':' + LEFT(RIGHT(sched.active_start_time, 4), 2)
										+ ':' + RIGHT(sched.active_start_time, 2) AS CHAR(8))
								WHEN 6
								THEN CAST(LEFT(RIGHT(sched.active_start_time, 6), 2) + ':'
										+ LEFT(RIGHT(sched.active_start_time, 4), 2) + ':'
										+ RIGHT(sched.active_start_time, 2) AS CHAR(8))
							END)
		, LastRunTime = Q1.LastRun
		, ca.ReportScheduleID
	into #ReportJobScheds
	FROM msdb.dbo.sysjobs job
		CROSS APPLY ( SELECT js.job_id, SUBSTRING(js.command, CHARINDEX('@EventData=', js.command) + 12, 36) as ReportScheduleID
							FROM msdb.dbo.sysjobsteps js
							WHERE js.command LIKE '%AddEvent @EventType=%'
					) ca
		CROSS APPLY ( SELECT CASE	WHEN TRY_CONVERT(UNIQUEIDENTIFIER, j.name) IS NULL
									THEN j.name
									ELSE NULL
								END AS jobname
							FROM msdb.dbo.sysjobs j
							WHERE j.job_id = job.job_id
					) caj
		LEFT JOIN ( SELECT job_schd.job_id
							, sys_schd.enabled
							, sys_schd.schedule_id
							, sys_schd.freq_type
							, sys_schd.freq_interval
							, sys_schd.freq_relative_interval
							, sys_schd.freq_recurrence_factor
							, sys_schd.freq_subday_type
							, sys_schd.freq_subday_interval
							, next_run_date = CASE	WHEN job_schd.next_run_date = 0
													THEN sys_schd.active_start_date
													ELSE job_schd.next_run_date
												END
							, next_run_time = CASE	WHEN job_schd.next_run_date = 0
													THEN sys_schd.active_start_time
													ELSE job_schd.next_run_time
												END
							, active_end_date = NULLIF(sys_schd.active_end_date,
														'99991231')
							, active_end_time = NULLIF(sys_schd.active_end_time,
														'235959')
							, sys_schd.active_start_date
							, sys_schd.active_start_time
						FROM msdb.dbo.sysjobschedules job_schd
							LEFT JOIN msdb.dbo.sysschedules sys_schd
								ON job_schd.schedule_id = sys_schd.schedule_id
					) sched
			ON job.job_id = sched.job_id
		LEFT OUTER JOIN ( SELECT job_id
									, MAX(job_his.run_duration) AS run_duration
									,MAX(CAST(
STUFF(STUFF(CAST(job_his.run_date as varchar),7,0,'-'),5,0,'-') + ' ' + 
STUFF(STUFF(REPLACE(STR(job_his.run_time,6,0),' ','0'),5,0,':'),3,0,':') as datetime)) AS [LastRun]
								FROM msdb.dbo.sysjobhistory job_his
								GROUP BY job_id
						) Q1
			ON job.job_id = Q1.job_id
		CROSS APPLY (SELECT sched.schedule_id,REPLACE(REPLACE(STUFF(
						(SELECT ', [' + WkDay + ']' AS [text()] 
							FROM #morepower m1 ,msdb.dbo.sysschedules s2
							WHERE m1.WkDay IS NOT NULL 
								AND freq_interval & m1.powerN >0
								AND  s2.schedule_id = sched.schedule_id
							ORDER BY N FOR XML PATH(''), TYPE).value('.','VARCHAR(MAX)')
						, 1, 2, ''),'[',''),']','') AS WkDays) c2
	WHERE job.job_id = ca.job_id
		AND caj.jobname IS NOT NULL;


SELECT sub.ReportScheduleID
	 , rjs.[Job Name]
	 , sub.ReportManagerPath
	 , sub.ReportName
	 , sub.SubscriptionCreator
	 , rjs.PrimaryScheduleSource
	 , sub.SubscriptionDescription
	 , sub.SSRSScheduleStartDate
	 , sub.LastRunTime
	 , sub.RecurrenceType
	 , sub.[Sched Enabled]
	 , sub.EventType
	 , sub.DaysofMonth
	 , sub.MonthString
	 , sub.DaysofWeek
	 , sub.MonthlyWeek
	 , sub.DaysInterval
	 , sub.MinutesInterval
	 , sub.WeeksInterval
	FROM #ReportJobScheds rjs
		INNER JOIN (SELECT DISTINCT 
		ReportScheduleID = s.ScheduleID
		,Ca.Path as ReportManagerPath
		,Ca.Name as ReportName
		,U.UserName as SubscriptionCreator
		--,PrimaryScheduleSource = 'SSRS Catalog'
		,Su.Description as SubscriptionDescription
		,SSRSScheduleStartDate = S.StartDate
		,Su.LastRunTime
		,CASE 
				WHEN s.RecurrenceType = 1 THEN 'Once'
				WHEN s.RecurrenceType = 2 THEN 'Hourly'
				WHEN s.RecurrenceType = 3 THEN 'Daily'
				WHEN s.RecurrenceType = 4 THEN 'Weekly'
				WHEN s.RecurrenceType = 5 THEN 'Monthly' 
				WHEN s.RecurrenceType = 6 THEN 'Week of Month' 
			END AS RecurrenceType
		, [Sched Enabled] = CASE su.InactiveFlags
								WHEN 0 THEN 'Yes'
								ELSE 'No'
							END
		,s.EventType
		,ISNULL(REPLACE(REPLACE(STUFF(
					(Select ', ['+CONVERT(VARCHAR(20),MonthDate)+']' AS [text()] 
						FROM #morepower m1 
						WHERE m1.powerN < s.DaysofMonth+1 
							AND s.DaysofMonth & m1.powerN >0 
						ORDER BY N FOR XML PATH(''), TYPE).value('.','VARCHAR(MAX)')
			   , 1, 2, ''),'[',''),']','')
			,'N/A') AS DaysofMonth
		,ISNULL(c1.NameOfMonth,'N/A') AS MonthString
		,ISNULL(c2.WkDays,'N/A') AS DaysofWeek
		,CASE MonthlyWeek
				WHEN 1 THEN 'First'
				WHEN 2 THEN 'Second'
				WHEN 3 THEN 'Third'
				WHEN 4 THEN 'Fourth'
				WHEN 5 THEN 'Last'
				ELSE 'N/A'
			END AS MonthlyWeek
		,ISNULL(CONVERT(VARCHAR(10),s.DaysInterval),'N/A') AS DaysInterval
		,ISNULL(CONVERT(VARCHAR(10),s.MinutesInterval),'N/A') AS MinutesInterval
		,ISNULL(CONVERT(VARCHAR(10),s.WeeksInterval),'N/A') AS WeeksInterval
	FROM #morepower mp, dbo.Schedule s
		INNER JOIN ReportSchedule RS
			ON S.ScheduleID = RS.ScheduleID
		INNER JOIN Catalog Ca
			ON Ca.ItemID = RS.ReportID
		INNER JOIN Subscriptions Su
			ON Su.SubscriptionID = RS.SubscriptionID
		INNER JOIN Users U
			ON U.UserID = S.CreatedById
			OR U.UserID = Su.OwnerID
		CROSS APPLY (Select s.ScheduleID,REPLACE(REPLACE(STUFF(
								(SELECT ', ['+ NameofMonth + ']' AS [text()] 
									FROM #morepower m1 ,dbo.Schedule s1
									WHERE m1.NameofMonth IS NOT NULL 
										AND m1.powerN & s1.Month >0 
										AND s1.ScheduleID = s.ScheduleID
									ORDER BY N FOR XML PATH(''), TYPE).value('.','VARCHAR(MAX)')
								, 1, 2, ''),'[',''),']','') AS NameOfMonth)c1
		CROSS APPLY (SELECT s.ScheduleID,REPLACE(REPLACE(STUFF(
								(SELECT ', [' + WkDay + ']' AS [text()] 
									FROM #morepower m1 ,dbo.Schedule s2
									WHERE m1.WkDay IS NOT NULL 
										AND DaysOfWeek & m1.powerN >0
										AND  s2.ScheduleID = s.ScheduleID
									ORDER BY N FOR XML PATH(''), TYPE).value('.','VARCHAR(MAX)')
								, 1, 2, ''),'[',''),']','') AS WkDays) c2
	WHERE Ca.Name = ISNULL(@ReportName,Ca.Name)) sub
	ON sub.ReportScheduleID = rjs.ReportScheduleID
UNION ALL
SELECT ss.ReportScheduleID
	 , [Job Name] = CONVERT(VARCHAR(64),ss.ReportScheduleID)
	 , ss.ReportManagerPath
	 , ss.ReportName
	 , ss.SubscriptionCreator
	 , ss.PrimaryScheduleSource
	 , ss.SubscriptionDescription
	 , ss.SSRSScheduleStartDate
	 , ss.LastRunTime
	 , ss.RecurrenceType
	 , ss.[Sched Enabled]
	 , ss.EventType
	 , ss.DaysofMonth
	 , ss.MonthString
	 , ss.DaysofWeek
	 , ss.MonthlyWeek
	 , ss.DaysInterval
	 , ss.MinutesInterval
	 , ss.WeeksInterval
	From #SSRSScheds ss;

SELECT sub.[Job ID]
	 , sub.[Job Name]
	 , sub.[Job Enabled]
	 , sub.JobSchedID
	 , rjs.PrimaryScheduleSource
	 , sub.RecurrenceType
	 , sub.[Sched Enabled]
	 , sub.EventType
	 , sub.[Frequency Interval]
	 , sub.[Relative Interval]
	 , sub.freq_recurrence_factor
	 , sub.[Next Run Date]
	 , sub.[Max Duration]
	 , sub.[Subday Frequency]
	 , sub.[Sched End Date]
	 , sub.[Sched End Time]
	 , sub.AgentScheduleStartDate
	 , sub.LastRunTime
	 , rjs.ReportScheduleID
	 , InStepScheduleID = sub.ReportScheduleID -- sometimes the scheduleid in the step is different from the actual scheduleid
	FROM #SSRSScheds rjs
		LEFT OUTER JOIN (SELECT [Job ID] = job.job_id
		, [Job Name] = caj.jobname
		, [Job Enabled] = CASE job.enabled
							WHEN 1 THEN 'Yes'
							WHEN 0 THEN 'No'
							END
		, [JobSchedID] = sched.schedule_id
		, PrimaryScheduleSource = 'SQL Agent Job Catalog'
		, [RecurrenceType] = CASE sched.freq_type
								WHEN 1 THEN 'Once'
								WHEN 4 THEN 'Daily'
								WHEN 8 THEN 'Weekly'
								WHEN 16 THEN 'Monthly'
								WHEN 32 THEN 'Monthly relative'
								WHEN 64 THEN 'When SQLServer Agent starts'
								END
		, [Sched Enabled] = CASE sched.enabled
								WHEN 1 THEN 'Yes'
								WHEN 0 THEN 'No'
							END
		, EventType = 'SharedSchedule'
		, CASE sched.freq_type 
			WHEN 1 THEN ''
			WHEN 4 THEN convert(VARCHAR,sched.freq_interval) + ' days'
			WHEN 8 THEN c2.WkDays
			WHEN 16 THEN 'Day of Month'
			WHEN 32 THEN case sched.freq_interval
								WHEN 1 THEN 'Sunday'
								WHEN 2 THEN 'Monday'
								WHEN 3 THEN 'Tuesday'
								WHEN 4 THEN 'Wednesday'
								WHEN 5 THEN 'Thursday'
								WHEN 6 THEN 'Friday'
								WHEN 7 THEN 'Saturday'
								WHEN 8 THEN 'Day'
								WHEN 9 THEN 'Weekday'
								WHEN 10 THEN 'Weekend Day'
								END
			WHEN 64 THEN ''
			WHEN 128 THEN ''
			END as [Frequency Interval]
		, [Relative Interval] = CASE sched.freq_relative_interval
									WHEN 1 THEN 'First'
									WHEN 2 THEN 'Second'
									WHEN 4 THEN 'Third'
									WHEN 8 THEN 'Fourth'
									WHEN 16 THEN 'Last'
									ELSE NULL
								END
		, sched.freq_recurrence_factor
		, [Next Run Date] = convert(datetime,CASE next_run_date
								WHEN 0 THEN NULL
								ELSE SUBSTRING(CONVERT(VARCHAR(15), next_run_date),
												1, 4) + '/'
										+ SUBSTRING(CONVERT(VARCHAR(15), next_run_date),
													5, 2) + '/'
										+ SUBSTRING(CONVERT(VARCHAR(15), next_run_date),
													7, 2)
							END + ' ' + CASE LEN(next_run_time)
								WHEN 1
								THEN CAST('00:00:0' + RIGHT(next_run_time, 2) AS CHAR(8))
								WHEN 2
								THEN CAST('00:00:' + RIGHT(next_run_time, 2) AS CHAR(8))
								WHEN 3
								THEN CAST('00:0' + LEFT(RIGHT(next_run_time, 3),
														1) + ':'
										+ RIGHT(next_run_time, 2) AS CHAR(8))
								WHEN 4
								THEN CAST('00:' + LEFT(RIGHT(next_run_time, 4),
														2) + ':'
										+ RIGHT(next_run_time, 2) AS CHAR(8))
								WHEN 5
								THEN CAST('0' + LEFT(RIGHT(next_run_time, 5), 1)
										+ ':' + LEFT(RIGHT(next_run_time, 4), 2)
										+ ':' + RIGHT(next_run_time, 2) AS CHAR(8))
								WHEN 6
								THEN CAST(LEFT(RIGHT(next_run_time, 6), 2) + ':'
										+ LEFT(RIGHT(next_run_time, 4), 2) + ':'
										+ RIGHT(next_run_time, 2) AS CHAR(8))
							END)
		, [Max Duration] = CASE LEN(run_duration)
								WHEN 1
								THEN CAST('00:00:0' + CAST(run_duration AS CHAR) AS CHAR(8))
								WHEN 2
								THEN CAST('00:00:' + CAST(run_duration AS CHAR) AS CHAR(8))
								WHEN 3
								THEN CAST('00:0' + LEFT(RIGHT(run_duration, 3),
														1) + ':'
										+ RIGHT(run_duration, 2) AS CHAR(8))
								WHEN 4
								THEN CAST('00:' + LEFT(RIGHT(run_duration, 4), 2)
										+ ':' + RIGHT(run_duration, 2) AS CHAR(8))
								WHEN 5
								THEN CAST('0' + LEFT(RIGHT(run_duration, 5), 1)
										+ ':' + LEFT(RIGHT(run_duration, 4), 2)
										+ ':' + RIGHT(run_duration, 2) AS CHAR(8))
								WHEN 6
								THEN CAST(LEFT(RIGHT(run_duration, 6), 2) + ':'
										+ LEFT(RIGHT(run_duration, 4), 2) + ':'
										+ RIGHT(run_duration, 2) AS CHAR(8))
							END
		, [Subday Frequency] = CASE ( sched.freq_subday_interval )
									WHEN 0 THEN 'Once'
									ELSE CAST('Every '
											+ RIGHT(sched.freq_subday_interval,
													2) + ' '
											+ CASE ( sched.freq_subday_type )
												WHEN 1 THEN 'Once'
												WHEN 4 THEN 'Minutes'
												WHEN 8 THEN 'Hours'
												END AS CHAR(16))
								END
		, [Sched End Date] = sched.active_end_date
		, [Sched End Time] = sched.active_end_time
		, [AgentScheduleStartDate] = convert(datetime,CASE sched.active_start_date
								WHEN 0 THEN NULL
								ELSE SUBSTRING(CONVERT(VARCHAR(15), sched.active_start_date),
												1, 4) + '/'
										+ SUBSTRING(CONVERT(VARCHAR(15), sched.active_start_date),
													5, 2) + '/'
										+ SUBSTRING(CONVERT(VARCHAR(15), sched.active_start_date),
													7, 2)
							END + ' ' + CASE LEN(sched.active_start_time)
								WHEN 1
								THEN CAST('00:00:0' + RIGHT(sched.active_start_time, 2) AS CHAR(8))
								WHEN 2
								THEN CAST('00:00:' + RIGHT(sched.active_start_time, 2) AS CHAR(8))
								WHEN 3
								THEN CAST('00:0' + LEFT(RIGHT(sched.active_start_time, 3),
														1) + ':'
										+ RIGHT(sched.active_start_time, 2) AS CHAR(8))
								WHEN 4
								THEN CAST('00:' + LEFT(RIGHT(sched.active_start_time, 4),
														2) + ':'
										+ RIGHT(sched.active_start_time, 2) AS CHAR(8))
								WHEN 5
								THEN CAST('0' + LEFT(RIGHT(sched.active_start_time, 5), 1)
										+ ':' + LEFT(RIGHT(sched.active_start_time, 4), 2)
										+ ':' + RIGHT(sched.active_start_time, 2) AS CHAR(8))
								WHEN 6
								THEN CAST(LEFT(RIGHT(sched.active_start_time, 6), 2) + ':'
										+ LEFT(RIGHT(sched.active_start_time, 4), 2) + ':'
										+ RIGHT(sched.active_start_time, 2) AS CHAR(8))
							END)
		, LastRunTime = Q1.LastRun
		, ca.ReportScheduleID
	FROM msdb.dbo.sysjobs job
		CROSS APPLY ( SELECT js.job_id, SUBSTRING(js.command, CHARINDEX('@EventData=', js.command) + 12, 36) as ReportScheduleID
							FROM msdb.dbo.sysjobsteps js
							WHERE js.command LIKE '%AddEvent @EventType=%'
					) ca
		CROSS APPLY ( SELECT CASE	WHEN TRY_CONVERT(UNIQUEIDENTIFIER, j.name) IS NULL
									THEN NULL
									ELSE j.name
								END AS jobname
							FROM msdb.dbo.sysjobs j
							WHERE j.job_id = job.job_id
					) caj
		LEFT JOIN ( SELECT job_schd.job_id
							, sys_schd.enabled
							, sys_schd.schedule_id
							, sys_schd.freq_type
							, sys_schd.freq_interval
							, sys_schd.freq_relative_interval
							, sys_schd.freq_recurrence_factor
							, sys_schd.freq_subday_type
							, sys_schd.freq_subday_interval
							, next_run_date = CASE	WHEN job_schd.next_run_date = 0
													THEN sys_schd.active_start_date
													ELSE job_schd.next_run_date
												END
							, next_run_time = CASE	WHEN job_schd.next_run_date = 0
													THEN sys_schd.active_start_time
													ELSE job_schd.next_run_time
												END
							, active_end_date = NULLIF(sys_schd.active_end_date,
														'99991231')
							, active_end_time = NULLIF(sys_schd.active_end_time,
														'235959')
							, sys_schd.active_start_date
							, sys_schd.active_start_time
						FROM msdb.dbo.sysjobschedules job_schd
							LEFT JOIN msdb.dbo.sysschedules sys_schd
								ON job_schd.schedule_id = sys_schd.schedule_id
					) sched
			ON job.job_id = sched.job_id
		LEFT OUTER JOIN ( SELECT job_id
								, MAX(job_his.run_duration) AS run_duration
								,MAX(CAST(
								STUFF(STUFF(CAST(job_his.run_date AS VARCHAR),7,0,'-'),5,0,'-') + ' ' + 
								STUFF(STUFF(REPLACE(STR(job_his.run_time,6,0),' ','0'),5,0,':'),3,0,':') AS DATETIME)) AS [LastRun]
							FROM msdb.dbo.sysjobhistory job_his
							GROUP BY job_id
						) Q1
			ON job.job_id = Q1.job_id
		CROSS APPLY (SELECT sched.schedule_id,REPLACE(REPLACE(STUFF(
						(SELECT ', [' + WkDay + ']' AS [text()] 
							FROM #morepower m1 ,msdb.dbo.sysschedules s2
							WHERE m1.WkDay IS NOT NULL 
								AND freq_interval & m1.powerN >0
								AND  s2.schedule_id = sched.schedule_id
							ORDER BY N FOR XML PATH(''), TYPE).value('.','VARCHAR(MAX)')
						, 1, 2, ''),'[',''),']','') AS WkDays) c2
	WHERE job.job_id = ca.job_id
		) sub
	ON sub.[Job Name] = rjs.ReportScheduleID
UNION ALL
	SELECT rjs.[Job ID]
			, rjs.[Job Name]
			, rjs.[Job Enabled]
			, rjs.JobSchedID
			, rjs.PrimaryScheduleSource
			, rjs.RecurrenceType
			, rjs.[Sched Enabled]
			, rjs.EventType
			, rjs.[Frequency Interval]
			, rjs.[Relative Interval]
			, rjs.freq_recurrence_factor
			, rjs.[Next Run Date]
			, rjs.[Max Duration]
			, rjs.[Subday Frequency]
			, rjs.[Sched End Date]
			, rjs.[Sched End Time]
			, rjs.AgentScheduleStartDate
			, rjs.LastRunTime
			, rjs.ReportScheduleID
			, InStepScheduleID = rjs.ReportScheduleID
		FROM #ReportJobScheds rjs;

As you can see, it is not a short script. By fair measure, it is also considerably more complex than the XML version that was recently posted (and mentioned earlier in this article). That said, it is many times more flexible and complete than the XML version as well. I do continue to use the bit math for figuring the schedules as I did in the first version of the script. When done this way, I can handle the custom schedules as well as get extensive details about the schedule from both the msdb and ReportServer databases.

Recap

SSRS provides built-in mechanisms to help report on the scheduled reports that have been deployed. This version of the report will help you retrieve the data from both a job perspective and from the report scheduler perspective. Through this series of articles, you should be confident in being able to now create custom schedules as well as accurately report on any reports that have specific subscriptions/schedules.

Reporting Services (SSRS) is a tool that permits you to create and deliver feature rich reports. The reports can be delivered in various formats and can even be scheduled to be delivered at various times. I even recently wrote about creating more advanced custom schedules.

With the ability to create content delivery schedules, or to subscribe to report content delivery, this imposes a requirement to also know when the various reports are scheduled to be delivered.

SSRS provides the means to be able to review the scheduled reports (subscriptions). That means is held within the ReportServer database. This article will help to uncover one of the sources of this scheduling information within the ReportServer database.

Review Schedules

When looking into the database for SSRS, I can see there are different means to be able to review the report schedules. This article is going to cover just one of those methods. And if I am going to be entirely up front about this method, I don’t like it and I recommend that it not be use.

I can hear the moans now. “If you don’t like it, then why show it to us?” Well, that is a very good question and there is a very good reason for this decision. A lesson I learned a long time ago is sometimes you need to learn the hard way, or less desirable way, to do various things. One of my favorite Calculus teachers from years ago drilled this into my head over and over again. Why? Well, there are three good reasons that come to mind: a) it makes the more desirable method seem much easier, b) it helps you to appreciate the more desirable method all that much more, and c) because if all else fails, you will have another method to fall back to just in case.

Less Disérables

The least desirable method (at least of the methods I will share) is to parse XML from a field stored in the ReportServer database. If I look into the Schedule table within the RepotServer database, I will find this column called MatchData. Up front, this field is not very intuitively named. I would not think this field actually represented the schedule, but it actually does.

Before we start diving into parsing XML, we need an example of what this XML may look like. The following will provide that very example that we need.

<ScheduleDefinition xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <StartDateTime xmlns="http://schemas.microsoft.com/sqlserver/reporting/2010/03/01/ReportServer">2017-03-27T02:00:00.000-06:00</StartDateTime>
  <MonthlyRecurrence xmlns="http://schemas.microsoft.com/sqlserver/reporting/2010/03/01/ReportServer">
    <Days>1</Days>
    <MonthsOfYear>
      <January>true</January>
      <February>true</February>
      <March>true</March>
      <April>true</April>
      <May>true</May>
      <June>true</June>
      <July>true</July>
      <August>true</August>
      <September>true</September>
      <October>true</October>
      <November>true</November>
      <December>true</December>
    </MonthsOfYear>
  </MonthlyRecurrence>
	<MinuteRecurrence xmlns="http://schemas.microsoft.com/sqlserver/reporting/2010/03/01/ReportServer">
		<MinutesInterval>75</MinutesInterval>
	</MinuteRecurrence>
	<WeeklyRecurrence xmlns="http://schemas.microsoft.com/sqlserver/reporting/2010/03/01/ReportServer">
		<WeeksInterval>1</WeeksInterval>
		<DaysOfWeek>
			<Sunday>true</Sunday>
			<Tuesday>true</Tuesday>
			<Thursday>true</Thursday>
			<Friday>true</Friday>
		</DaysOfWeek>
	</WeeklyRecurrence>
</ScheduleDefinition>

Are your gears grinding yet? The XML is not terribly difficult to follow. I am sure you have realized the problem from this format at this point. If I query this to make it human readable in a tabular format (you know DBA format), I will end up with a really wide table that is pretty ugly to look at (unless I get super creative to combine fields etc).

Let’s take a look at the query to parse something like the preceding XML example.

WITH XMLNAMESPACES('http://schemas.microsoft.com/sqlserver/reporting/2010/03/01/ReportServer' AS rs)
, getxml AS (
	SELECT SUB.SubscriptionID
			, ReportName = CAT.[Name]
			, ExtensionSettings = CAST(SUB.ExtensionSettings AS XML)
			, ReportSchedule = CAST(SUB.MatchData AS XML)
		FROM dbo.Subscriptions AS SUB
			INNER JOIN dbo.Users AS USR
				ON SUB.OwnerID = USR.UserID
			INNER JOIN dbo.[Catalog] AS CAT
				ON SUB.Report_OID = CAT.ItemID
			INNER JOIN dbo.ReportSchedule AS RS
				ON SUB.Report_OID = RS.ReportID
					AND SUB.SubscriptionID = RS.SubscriptionID
			INNER JOIN dbo.Schedule AS SCH
				ON RS.ScheduleID = SCH.ScheduleID 
  --WHERE CONVERT(CHAR(8), SUB.LastRunTime, 112  ) > '20170101' 
)
SELECT SubscriptionID
		, ReportName
		, [StartHour] = SUBSTRING(( c.nd.value('(rs:StartDateTime/text())[1]',
												'VARCHAR(500)') ), 12, 2)
		, [StartMin] = SUBSTRING(( c.nd.value('(rs:StartDateTime/text())[1]',
												'VARCHAR(500)') ), 15, 2)
		, [Days] = c.nd.value('(rs:MonthlyRecurrence/rs:Days/text())[1]', 'INT')
		, [January] = c.nd.value('(rs:MonthlyRecurrence/rs:MonthsOfYear/rs:January/text())[1]',
									'BIT')
		, [February] = c.nd.value('(rs:MonthlyRecurrence/rs:MonthsOfYear/rs:February/text())[1]',
									'BIT')
		, [March] = c.nd.value('(rs:MonthlyRecurrence/rs:MonthsOfYear/rs:March/text())[1]',
								'BIT')
		, [April] = c.nd.value('(rs:MonthlyRecurrence/rs:MonthsOfYear/rs:April/text())[1]',
								'BIT')
		, [May] = c.nd.value('(rs:MonthlyRecurrence/rs:MonthsOfYear/rs:May/text())[1]',
								'BIT')
		, [June] = c.nd.value('(rs:MonthlyRecurrence/rs:MonthsOfYear/rs:June/text())[1]',
								'BIT')
		, [July] = c.nd.value('(rs:MonthlyRecurrence/rs:MonthsOfYear/rs:July/text())[1]',
								'BIT')
		, [August] = c.nd.value('(rs:MonthlyRecurrence/rs:MonthsOfYear/rs:August/text())[1]',
								'BIT')
		, [September] = c.nd.value('(rs:MonthlyRecurrence/rs:MonthsOfYear/rs:September/text())[1]',
									'BIT')
		, [October] = c.nd.value('(rs:MonthlyRecurrence/rs:MonthsOfYear/rs:October/text())[1]',
									'BIT')
		, [November] = c.nd.value('(rs:MonthlyRecurrence/rs:MonthsOfYear/rs:November/text())[1]',
									'BIT')
		, [December] = c.nd.value('(rs:MonthlyRecurrence/rs:MonthsOfYear/rs:December/text())[1]',
									'BIT')
		, [Sunday] = c.nd.value('(rs:WeeklyRecurrence/rs:DaysOfWeek/rs:Sunday/text())[1]',
								'BIT')
		, [Monday] = c.nd.value('(rs:WeeklyRecurrence/rs:DaysOfWeek/rs:Monday/text())[1]',
								'BIT')
		, [Tuesday] = c.nd.value('(rs:WeeklyRecurrence/rs:DaysOfWeek/rs:Tuesday/text())[1]',
									'BIT')
		, [Wednesday] = c.nd.value('(rs:WeeklyRecurrence/rs:DaysOfWeek/rs:Wednesday/text())[1]',
									'BIT')
		, [Thursday] = c.nd.value('(rs:WeeklyRecurrence/rs:DaysOfWeek/rs:Thursday/text())[1]',
									'BIT')
		, [Friday] = c.nd.value('(rs:WeeklyRecurrence/rs:DaysOfWeek/rs:Friday/text())[1]',
								'BIT')
		, [Saturday] = c.nd.value('(rs:WeeklyRecurrence/rs:DaysOfWeek/rs:Saturday/text())[1]',
									'BIT')
		, [WeeksInterval] = c.nd.value('(rs:WeeklyRecurrence/rs:WeeksInterval/text())[1]',
										'INT')
		, [MinuteRecurrence] = c.nd.value('(rs:MinuteRecurrence/rs:MinutesInterval/text())[1]',
											'INT')
	FROM getxml
		CROSS APPLY ReportSchedule.nodes('//ScheduleDefinition') c ( nd )
	ORDER BY SubscriptionID
		, ReportName;

And there we have that ugly query to produce a really wide ugly result set. The query is not difficult to write. It’s just extremely repetitive. In similar fashion, the results are very repetitive. This makes, in my eyes, this particular method less desirable.

I haven’t even gotten to the part about the shortcoming in scheduling reports through SSRS that I wrote about recently – here. In that article I discussed a workaround to overcome the SSRS scheduling options. If you employ methods such as I discussed there, then this query will never fully cover the scheduling related to your reports. Because of that, I will be discussing the better solution in the next article.

Recap

SSRS provides built-in mechanisms to help report on the scheduled reports that have been deployed. While parsing the XML is less desirable than what I will be sharing in the near future, it is better than doing nothing at all. I recommend you start looking into the various report schedules you may already have in your environment. Also, stay tuned for the next article that will better show these schedules.

Reporting Services is a pretty feature rich tool for delivering reports to various consumers. There is plenty of power within Reporting Services (SSRS) giving one the ability to perform visualizations, render reports in various formats and even schedule reports to be delivered in different formats or on different schedules.

Unfortunately, the scheduling capability within SSRS is fairly weak. While it is true that one can accomplish a varied array of different schedules, the scheduling of SSRS is far below the power of SQL Agent (for instance).

I will explore the deficiency of the scheduling tool within SSRS in a very specific case. You may even be familiar with this deficiency already. Many organizations have the need to produce end of month reports that need to run on the last day of the month. If you are familiar with the scheduling tool within SSRS, you already know that this is not possible (at least through SSRS 2014). This article will help step you through how to schedule a report subscription to run on the last day of the month.

Schedule Options

Before diving into the custom schedules, let’s take a closer look at the options available for scheduling through SSRS.

While the tool does provide various options, the granularity certainly is not diverse enough to cover many legitimate scheduling needs – especially the “last day of month” requirement for many month end type of reports.

As you may be aware, SSRS subscriptions are actually run through the SQL Agent despite being set through SSRS. If I take a look at some of the scheduling options in SQL Agent, I can see the following.

Note here that there are various built-in options to schedule on a monthly basis, including the “last” option. If I wanted to look even closer at the available options I would see the following.

As you can see here, I can schedule for multiple different types of “last” options relative to the month. One of these options happens to be the last day. This helps to illustrate just how much more powerful and versatile the scheduling within SQL Agent is than what we get with the SSRS scheduler. All of this despite the fact that SSRS subscriptions are actually executed by the SQL Agent. Doesn’t that seem a bit odd?

Custom Schedule

Now that we better understand the limitations of the SSRS scheduler and given that SSRS subscriptions are executed through the SQL Agent, let’s move on to bigger, better and much more useful means of scheduling the SSRS reports.

The very first thing that you should do is to create a share schedule. This should be a shared schedule that is created as a run-once schedule. Let the schedule run that one time and then proceed on to the following steps. If you need help in creating a shared schedule, here is an msdn article. When you create the shared schedule, I recommend using a descriptive name that you can remember. This name will be useful in the next step. For the purposes of this article, my schedule is named “EndOfMonth”.

Once the schedule is created, the next thing to do is to query the ReportServer database. Make sure you know the name of your database. Some people have changed the ReportServer database name from the default. This is an important piece of information to remember. The query against the ReportServer database will be predominantly just to get the schedule id of the newly created schedule.

SELECT sch.scheduleid
		, sub.Description
		, sch.Name
	FROM subscriptions sub
		LEFT JOIN reportschedule rs
			ON sub.subscriptionid = rs.subscriptionid
		LEFT JOIN schedule sch
			ON sch.scheduleid = rs.scheduleid
	WHERE sch.Name = 'EndofMonth'
	ORDER BY sub.modifieddate DESC;

When I run that query, I receive the following results.

I now want to take that scheduleid and then use it to determine what SQL Agent job is actually related to that schedule so I can fetch some info from the job. I could skip this entirely and go to the subsequent step but this helps to understand what needs to be done in that subsequent step. So, from here let’s query the msdb database in SQL Server to fetch some info from the job system.

SELECT j.name
		, j.job_id
		, js.step_name
		, js.command
		, sp.name AS JobOwner
	FROM msdb.dbo.sysjobs j
		INNER JOIN msdb.dbo.sysjobsteps js
			ON j.job_id = js.job_id
		INNER JOIN sys.server_principals sp
			ON j.owner_sid = sp.sid
	WHERE js.command LIKE '%219542cc-c1e8-4b42-889f-682d7276a07a%'
		AND j.description like 'This job is owned by a report server process. %';

The scheduleid is used within a command within a jobstep. By passing the scheduleid into this query and then comparing against the existing job steps, I am able to retrieve the job that is related to the SSRS shared schedule (subscription). When I run the preceding query, I receive results illustrated in the following image.

Take note of the items I highlighted in red. Within the job name, step name and command text for the step, I can find the scheduleid for that shared schedule that I created. Yes, I could easily have changed the query I used to compare to the job name and that would have worked just fine in this case. By querying the command, I can confirm that the schedule is actually being used. The most important piece of information in this result is the entire command text for the job. I will need to take this command text and use it to populate a brand new SQL Agent job. This is how I will get my custom schedule for the SSRS subscription.

exec [ReportServer2012].dbo.AddEvent @EventType='SharedSchedule', @EventData='219542cc-c1e8-4b42-889f-682d7276a07a'

From here, I just need to create a SQL Agent job that uses the options for a monthly schedule indicating last day from the two drop down menus illustrated previously in this article. Then all that is left is a sigh of relief and a boom bada bing.

Recap

SSRS does not have the built-in capability for some of the more complex and often times regularly required report schedules to meet various business requirements. By following the steps outlined in this article, you can circumvent that short-coming and achieve the needed business requirements while looking like a hero to those that need the more advanced report schedules.

More and more I am seeing clients requiring assistance with the Microsoft Dynamics Suite. Each of the products in the suite comes with a different set of performance issues and gotchas. In this article I will only be discussing the AX product and an easy tweak to make troubleshooting that product much easier from the perspective of the database administrator. This tweak is to enable the context info from within the administration console.

Enable Context Info

Some may call this a critical setting that must be activated on every transaction heavy Dynamics AX AOS server. One of the most common reasons is that DAX user sessions frequently block one another. Occasionally the blocking may be uncomfortably long.

In  order to enable this setting on each DAX AOS server, the following steps should be followed:

• Navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\DynamicsServer\6.01\Original (installed configuration). The last key, Original (installed configuration), is the key name for the current server configuration. If your system uses a different configuration than the original installed configuration, navigate to the currently active configuration.
• Create a string registry value called ‘connectioncontext’ and set the value to 1.
• Restart the AOS.

The steps to implement this change is not terribly difficult. The risk is very low as well. There is very minimal cost for sending this additional info. That said, even on an extremely busy system for one client, we have yet to see a negative impact with this setting enabled.

Easy Troubleshooting for the DBA

Now that the context info is enabled within the application, this is where the pains of troubleshooting AX performance issues within the database becomes somewhat easier for the database administrator. Just enabling the setting doesn’t bring you to the promised land though. You still need to do a bit of work.

Once the setting is enabled, what actually happens is the AX application starts to send an extra chunk of data along with each connection to the database server. This chunk of data is the context info. The context info that AX decides to send along is not straight-forward to read however. The AX context info is sent to SQL Server as a varbinary.

What does it mean to be in varbinary format for you as the database administrator? This means that you still have a bit of work to do. Do you need to perform that extra work every time you look at the data? Well, the short answer is “it depends”! If you are smart about your tool-set (e.g. set of administration scripts) then you will save this extra work there. If you do not yet have a tool-set and rewrite your queries every time – you obviously fall at the far opposite end of the spectrum and will have much more work to do.

Whichever end of the spectrum you fall within, here is script to integrate into your scripts to help make your AX DBA work just a tad bit easier.

SELECT SUBSTRING(REPLACE(LTRIM(CAST(s.context_info AS VARCHAR(256))), ' ',
							','),
					CHARINDEX(',',
								REPLACE(LTRIM(CAST(s.context_info AS VARCHAR(256))),
										' ', ',')) + 1,
					CHARINDEX(',',
								REPLACE(LTRIM(CAST(s.context_info AS VARCHAR(256))),
										' ', ','),
								CHARINDEX(',',
											REPLACE(LTRIM(CAST(s.context_info AS VARCHAR(256))),
													' ', ',')) + 1)
					- ( CHARINDEX(',',
									REPLACE(LTRIM(CAST(s.context_info AS VARCHAR(256))),
											' ', ',')) )
					- CASE	WHEN CAST(s.context_info AS VARCHAR(256)) = ''
							THEN 0
							ELSE 1
						END) AS DAXSessionID
		, SUBSTRING(REPLACE(LTRIM(CAST(s.context_info AS VARCHAR(256))), ' ',
							','), 1,
					CHARINDEX(',',
								REPLACE(LTRIM(CAST(s.context_info AS VARCHAR(256))),
										' ', ','))
					- CASE	WHEN CAST(s.context_info AS VARCHAR(256)) = ''
							THEN 0
							ELSE 1
						END) AS DAXUser
		, s.context_info
	FROM sys.dm_exec_sessions s
	WHERE ISNULL(CAST(s.context_info AS VARCHAR(256)),'') <> '';

So What does it DO?

So what value does this query actually bring you? I have talked about it making life easier by enabling the context info from within AX, but I didn’t dive into any details on what it will provide.

Looking at the query I just provided, one can surmise that the context info will provide two significant pieces of information. The first bit is the Session ID. This is not the spid within SQL Server. Rather this is the session id that is a different value within AX. The second piece of information that is highly valuable is the User that is tied to that AX Session. The AX application will show as the service account for the application on all spids within SQL Server. The spid and spid user are fairly useless when trying to figure out who is causing what level of pain since all users in SQL Server for AX will appear to be the same user. The SQL spid will be useless for the DAX admins because the spid will not match the DAX session id. Both of these factors will lead to an extra amount of frustration between the DBA and DAX Admin if in the middle of a performance slowdown.

Being able to extract the DAX User and Session ID from the context info will significantly reduce troubleshooting time when in the trenches trying to figure out who is running what from within the application. This reduces the chances of taking a guess and gives good solid evidence that can be taken back to the business users and try to improve their processes and the overall performance of the system.

Back in late December of 2015, a challenge of sorts was issued by Tim Ford (twitter) to write a blog post each month on a SQL Server Basic. Some have hash-tagged this as #backtobasics. Here is the link to that challenge sent via tweet.

I did not officially accept the challenge. Was an official acceptance required? I don’t know. I do know that I think it is a good challenge and that I intend to participate in the challenge. I hope I can meet the requirements and keep the posts to “basics”. Let’s hope this post holds up to the intent of the challenge.

With this being another installment in a monthly series, here is a link to review the other posts in the series – back to basics. Reviewing that link, you can probably tell I am a bit behind in the monthly series.

Logins and Users

It seems appropriate to re-introduce the concept of principals (aka Logins and Users). Rather than go into depth about principals here though, I will refer you to a recent article on the topic. The article in question was another “basics” article and can be found here.

With that out of the way, it should be conceded that creating principals is a common practice and possibly a frequent requirement of the data professional. While creating those principals, there is a good chance that one will run into an absurd error ever now and then. Today, I want to discuss one absurd error. The fix for the error may seem just as absurd as the error, but would be really easy to implement.

Invalid Characters

Here is the error message that is quite possible to encounter while creating principals.

Msg 15006, Level 16, State 1, Line 6
‘SomeDOmain\jason’ is not a valid name because it contains invalid characters.

At first look, this error makes absolutely no sense. The error states there is an invalid character somewhere in the string “SomeDomain\jason”, yet every character in that string is supported and normal for the collation. This can be a head-scratcher for sure.

To better understand this error, let’s try to reproduce the error. First, we need to create a login.

CREATE LOGIN [SomeDomain\jason] FROM WINDOWS WITH DEFAULT_DATABASE=[master]
GO

Here, I have used “SomeDomain” in lieu of my actual domain or local workstation name. This statement will complete successfully given the user exists within the domain or on the Windows workstation. Great so far!

The next step is to create a database user within the AdminDB (you can pick a database that exists in your environment) and map this user to the Login created in the previous step. This can be done with the following script:

USE AdminDB
GO
CREATE USER [SomeDOmain\jason] FOR LOGIN [SomeD0main\jason]
GO

Bam! Executing the script produces:

Msg 15006, Level 16, State 1, Line 6
‘SomeDOmain\jason’ is not a valid name because it contains invalid characters.

This is where a close inspection of the script is required. Due to a fabulous fat finger, a 0 (zero) instead of O (capital o) was typed in the second occurrence of “SomeDOmain”. This is easy enough to reproduce with a typo of any portion of the windows login that already exists in SQL as a login principal.

The Fix

The fix is insanely easy once you figure out that invalid character actually means you mis-typed the Login portion of the Create User statement. The fix is to type the login correctly. Knowing is half the battle! Running into this error in the wild could cause you a few minutes trying to figure it out and prepping to throw something through the monitor.

Recap

In this article I have shown how a simple mistake can lead to a really obtuse error message that doesn’t seem to make much sense. A little care and attention to properly typing the login names will save you a bit of time and hair on the troubleshooting end of creating principals.

Back in late December of 2015, a challenge of sorts was issued by Tim Ford (twitter) to write a blog post each month on a SQL Server Basic. Some have hash-tagged this as #backtobasics. Here is the link to that challenge sent via tweet.

I did not officially accept the challenge. Was an official acceptance required? I don’t know. I do know that I think it is a good challenge and that I intend to participate in the challenge. I hope I can meet the requirements and keep the posts to “basics”. Let’s hope this post holds up to the intent of the challenge.

With this being another installment in a monthly series, here is a link to review the other posts in the series – back to basics. Reviewing that link, you can probably tell I am a bit behind in the monthly series.

Active Queries

If you are like me, you have had the opportunity on more than one occasion to try and figure out what is currently active on your SQL Server. The reason to try and figure this is out is usually tied to some sort of performance issue that you have to dive in and troubleshoot.

When checking for current activity through sp_who2 or by querying the dmvs (sys.dm_exec_sessions, sys.dm_exec_requests, sys.dm_exec_connections), there is inevitably some sort of session that looks something like this:

This is not very helpful in this format. What I have seen most people do when they see this kind of result (and they care enough to know what is running) is to open up the “Job Activity Monitor” and then try to figure out manually what job is truly running. You can imagine the nightmare this becomes if there are more than a few jobs running.

Currently Running Agent Jobs

There is a significantly easier way to find the name and step of the agent job that is currently running when using your favorite dmv query to explore current activity. Let’s start with a simple query.

SELECT p.session_id,p.program_name
	FROM sys.dm_exec_sessions p
	WHERE p.program_name LIKE 'SQLAgent - TSQL JobStep (Job%';

If you have jobs that are currently executing, then this query should return some results representative of the running jobs. Unfortunately, you only know that the source of the spid happens to be the SQLAgent. I am going to dirty up this simple query with quite a bit more query so it looks like the following:

SELECT p.session_id,p.program_name
	,j.name AS JobName, js.step_name, j.description, js.step_id
	, st.Query AS JobCommand
	, js.subsystem
	FROM sys.dm_exec_sessions p
		INNER JOIN msdb.dbo.sysjobs j
			ON j.job_id = CONVERT(UNIQUEIDENTIFIER, CONVERT(BINARY(16), SUBSTRING(p.program_name,
																CHARINDEX('(',
																p.program_name)
																+ 5,
																CHARINDEX(':',
																p.program_name)
																- CHARINDEX('(',
																p.program_name)
																- 6),1))
		INNER JOIN msdb.dbo.sysjobsteps js
			ON j.job_id = js.job_id
				CROSS APPLY (
					SELECT 
						REPLACE
						(
							REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(
							REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(
							REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(
								CONVERT
								(
									NVARCHAR(MAX),
									N'--' + NCHAR(13) + NCHAR(10) + ist.command + NCHAR(13) + NCHAR(10) + N'--' COLLATE Latin1_General_Bin2
								),
								NCHAR(31),N'?'),NCHAR(30),N'?'),NCHAR(29),N'?'),NCHAR(28),N'?'),NCHAR(27),N'?'),NCHAR(26),N'?'),NCHAR(25),N'?'),NCHAR(24),N'?'),NCHAR(23),N'?'),NCHAR(22),N'?'),
								NCHAR(21),N'?'),NCHAR(20),N'?'),NCHAR(19),N'?'),NCHAR(18),N'?'),NCHAR(17),N'?'),NCHAR(16),N'?'),NCHAR(15),N'?'),NCHAR(14),N'?'),NCHAR(12),N'?'),
								NCHAR(11),N'?'),NCHAR(8),N'?'),NCHAR(7),N'?'),NCHAR(6),N'?'),NCHAR(5),N'?'),NCHAR(4),N'?'),NCHAR(3),N'?'),NCHAR(2),N'?'),NCHAR(1),N'?'),
							NCHAR(0),
							N''
						) AS [processing-instruction(query)]
						FROM msdb.dbo.sysjobsteps AS ist
							WHERE ist.job_id = j.job_id
								AND ist.step_id = SUBSTRING(p.program_name, CHARINDEX(')', p.program_name) - 2, 2)
					FOR XML
						PATH(''),
						TYPE
				) AS st(Query)
	WHERE p.program_name LIKE 'SQLAgent - TSQL JobStep (Job%'
		AND js.step_id = SUBSTRING(p.program_name, CHARINDEX(')', p.program_name) - 2, 2)
	;

There is a good reason for how much I have complicated the simple version of the query. I can leave the query significantly less complicated if not for the fact that I wanted to also know the sql text in a well formatted manner. That accounts for the entire segment in the cross apply.

To retrieve the name of the job that is running, I actually only need this little piece of code right here:

INNER JOIN msdb.dbo.sysjobs j
			ON j.job_id = CONVERT(UNIQUEIDENTIFIER, CONVERT(BINARY(16), SUBSTRING(p.program_name,
																CHARINDEX('(',
																p.program_name)
																+ 5,
																CHARINDEX(':',
																p.program_name)
																- CHARINDEX('(',
																p.program_name)
																- 6),1))

This takes the varbinary representation of the jobid string and converts to the human friendly form with the appropriate format of the string so we can compare it to the the actual job id and then finally get the job name. From there, I can then retrieve the job step to see exactly where in the process the job is presently executing.

Executing this query, I receive the following results for the job that I have executing right now.

If I take this code and then integrate it into my favorite query to check for running sessions while investigating issues on the server, I have become just that much more efficient as a DBA.

Recap

It is quite common to be required to investigate performance issues on the server. Even if not a performance issue, there are frequent needs that require us to know what queries are executing at various points in time throughout the day. A complication to this is the varbinary format of the job name that is represented as the program that is running during many of these spot checks.

The means to circumvent this complication is with a little extra code for your favorite script du jour used to investigate running sessions. I recommend adding a code segment, such as the code I have shown in this article, to help simplify your research tasks and help you look more like a rockstar. Of course, you could always resort to the other method touched on in this article – trial and guess through manual process of elimination via “Job Activity Monitor”.

Back in late December of 2015, a challenge of sorts was issued by Tim Ford (twitter) to write a blog post each month on a SQL Server Basic. Some have hash-tagged this as #backtobasics. Here is the link to that challenge sent via tweet.

I did not officially accept the challenge. Was an official acceptance required? I don’t know. I do know that I think it is a good challenge and that I intend to participate in the challenge. I hope I can meet the requirements and keep the posts to “basics”. Let’s hope this post holds up to the intent of the challenge.

With this being another installment in a monthly series, here is a link to review the other posts in the series – back to basics. Reviewing that link, you can probably tell I am a bit behind in the monthly series.

Database Backups

In my previous two articles, I touched on some data that is available to the data professional with regards to backups and database restores. In the article discussing the restore history data, I alluded to another topic related to restores that could have been discussed.

Today, I will be diving into that alternate direction. This direction is related to database backups. More specifically, how do you truly know if you have a successful backup? Here’s a hint: it has to deal with restores.

If you haven’t heard it before, here it is in simple terms “you do not have a backup until you have restored the backup“. If you ponder that for a minute, it is quite logical and makes plenty of sense. The only way to confirm that you have a backup is to test the purpose of the backup and that is to recover the database back to the state represented by the backup.

Testing Backups

Creating a database backup is a pretty straight-forward task. This is easy enough to do through the GUI or from tsql script, or even, *shudder*, from a maintenance plan. Creating the backup is the easy part. If you are not creating backups, I hope there is a good reason for it (and yes there are valid reasons for not creating a backup of certain databases).

This article will not explore the nuances of the ways to create a backup of a database. Suffice it to say, there are multiple options and methods. This article will focus on the second part of creating a successful backup – testing the backup. In other words, restoring the backup that was created.

So how does one go about testing a backup? The simple answer as already stated is to perform a restore. This means through the use of the restore command. But is it really that simple?

If I have the following backups available for my AdventureWorks2014 database, where should I test the validity of the backup?

Would it make much sense to take any of these backups and test the restore process on the same server as the source of the backup? To be honest, that is a question that may have an entirely different answer in each and every environment. That is a question that requires a little insight into the business needs, available resources and procedures and policies in place for the environment. For me, I typically like to create an automated restore process that will restore the previous night’s backup onto a test/stage/dev server that is not in production.

I have written previously on how I do this sort of automated restore. You can read all about it from the original article here. In that article I provide a script to assist with the restore of these backups. You are welcome to test it out and play around with it while setting up your backup validation environment. In using the script, the user assumes all risk.

I recommend an automated restore system to ensure the backups are tested on a routine basis. The benefits of doing this are more than just a few. That said here are a few of those benefits: 1. Confidence in backups, 2. Quick recovery in event of failure or disaster, 3. A usable environment for quick data comparison, 4. A passable environment for reporting, and the best benefit is that you can get a good nights sleep knowing your backups are reliable.

Caveat

With automated restores, there may be an occasional failure. Heck, you may run into a bit of a head-scratcher here or there trying to get them to work in the first place. It’s not technically easy the first time. With some practice, it gets much easier. This method is intended to be suitable for a cheap solution. There may be a pricier solution out there that can be bought. If that is in your budget – go with it. The main point is to do it.

After you get these restores working, I recommend using the scripts in this article to check the restore history from time to time. Take it an extra mile and generate some reports from that data.

Recap

I have provided some information on how and why to restore a database. The primary reason being that you never know how good your backup is until you have restored it. Some say you do not have a backup until you have restored it. This is a solution and some opinions on how to do that efficiently.

Back in late December of 2015, a challenge of sorts was issued by Tim Ford (twitter) to write a blog post each month on a SQL Server Basic. Some have hash-tagged this as #backtobasics. Here is the link to that challenge sent via tweet.

I did not officially accept the challenge. Was an official acceptance required? I don’t know. I do know that I think it is a good challenge and that I intend to participate in the challenge. I hope I can meet the requirements and keep the posts to “basics”. Let’s hope this post holds up to the intent of the challenge.

With this being another installment in a monthly series, here is a link to review the other posts in the series – back to basics. Reviewing that link, you can probably tell I am a bit behind in the monthly series.

Database Backup History

As a data professional, there is little to no doubt that one of the most important tasks is to ensure the data is backed up regularly. We do this in preparation for several reasons. Whether it be to be able to recover in the event of a problem or to compare yesterdays data to how it looks today, the backup is done.

When doing a backup, how often do you report on the backup success? How frequently do you refer to the backup history to compare multiple runs or even generate reports for interested parties? The fact of the matter is that this history should be referenced on a routine basis to generate reports that should be read and not ignored.

Backup Reporting

Right about here you may be asking yourself why would somebody generate reports on the backups. After all, the job succeeds or fails and you respond appropriately to that, right? Does one really need to generate a report about the success of the backup job? Well, that is for you to decide. I would say it wouldn’t hurt to generate a report about the success rate of your backups. Management often likes to see reports concerning important processes. That said, that is not the reporting that I am referencing here.

The reporting I believe is useful in this case is indeed related to the historical information of your backups, but it is less about the success of the job and so forth. Of great benefit from the backup history data is the information on frequency of backup and also the size of the backup. These pieces of information can be somewhat useful when looking to forecast data growth. That is one example of a possible report when working with the backup history data that is stored.

Let’s take a look at a query that will help retrieve some of this backup history data.

DECLARE @SQLVer SQL_VARIANT
	, @DBName VARCHAR(128)
	, @NumDays SMALLINT
	, @SQL VARCHAR(1024)
	, @WhereClause VARCHAR(256);
 
SET @DBName = NULL;
SET @NumDays = 10000;
SET @SQLVer = CONVERT(INTEGER, PARSENAME(CONVERT(VARCHAR(20), SERVERPROPERTY('ProductVersion')),
											4));

SET @WhereClause = 'WHERE a.type IN (''D'',''I'')
		AND a.backup_start_date > GETDATE()- ' + CAST(@NumDays AS VARCHAR) + '';
IF @DBName IS NOT NULL
	BEGIN
		SET @WhereClause = @WhereClause + '
		AND a.database_name = ''' + @DBName + '''';
	END;

SET @SQL = '
SELECT a.database_name,a.backup_start_date,a.backup_finish_date
		,datediff(second,a.backup_start_date,a.backup_finish_date)/60.0 as DurationMinute
		,b.physical_device_name AS BackupPath
		,a.position
		,a.type
		,a.backup_size/1024/1024 AS BackupSizeMB
		,' + CASE	WHEN @SQLVer < 10 THEN '0'
					ELSE 'a.compressed_backup_size/1024/1024'
				END + ' AS CompressedBackMB
	FROM msdb.dbo.backupset a
		INNER JOIN msdb.dbo.backupmediafamily b
			ON a.media_set_id = b.media_set_id
	' + @WhereClause + '
	ORDER BY a.database_name,a.backup_start_date;';

--PRINT @SQL
EXECUTE (@SQL);

Off the top, yes the script uses dynamic sql. I use dynamic sql because I use this script on multiple versions of SQL Server. One of the data points I gather in this script is the compressed backup size. That attribute is not available in SQL 2005. Due to that, and not wanting to maintain multiple scripts, I use the dynamic sql to account for that missing column.

I have also set the script to accept a database name parameter. If a name is provided, then only the backup history for that database is returned. If the parameter is left NULL, then the backup history for all databases will be returned. Additionally, I added a number of days parameter to limit the scope of the report to a specific range of days.

Among the data points returned in this script, you will note there is the duration of the backup, the date, and even the size of the backup. All of these attributes can help me to forecast future storage requirements both for the backup storage as well as for the data volume. Additionally, by knowing the duration of the backup and the trend of that duration, I can adjust maintenance schedules accordingly.

Caveat

In this particular script, I only retrieve the backup history for two backup types: FULL and DIFFERENTIAL. A small adjustment can be made to include other backup types if necessary. The line to change is:

SET @WhereClause = 'WHERE a.type IN (''D'',''I'')

Other possible values for backups are:

• D = Database
• I = Differential database
• L = Log
• F = File or filegroup
• G =Differential file
• P = Partial
• Q = Differential partial

If you are interested in learning more about backup information, you can read this msdn article – here.

Recap

I have provided a quick example of how to retrieve pertinent backup history information. This information can be used to assist in forecasting storage requirements as well as assist in the job scheduling as backup jobs begin to run longer and longer as the data gets larger and larger.

For a related article, check out this article on how to retrieve the restore history from the msdb database.

Back in late December of 2015, a challenge of sorts was issued by Tim Ford (twitter) to write a blog post each month on a SQL Server Basic. Some have hash-tagged this as #backtobasics. Here is the link to that challenge sent via tweet.

I did not officially accept the challenge. Was an official acceptance required? I don’t know. I do know that I think it is a good challenge and that I intend to participate in the challenge. I hope I can meet the requirements and keep the posts to “basics”. Let’s hope this post holds up to the intent of the challenge.

With this being another installment in a monthly series, here is a link to review the other posts in the series – back to basics. Reviewing that link, you can probably tell I am a bit behind in the monthly series.

Database Restore

As a data professional, one should be somewhat cognitive of backups with regards to the data. From the point of the creation of the backup, we can go in one of two directions with this article. Due to the importance of each of the directions, I will devote a basics article to each.

The order of publication of these articles in no means denotes a level of priority to the topic. I feel each is equally important. The two directions I see both deal with the restore of that backup that was created. One direction is in regards to reporting and the other direction is the actual restore.

Last Restore Date

Something that I seem to be getting asked more and more frequently is how to determine when a database was last restored. Maybe it is just a hot topic and maybe it will be short lived, but it sure seems to be something that is on the tip of peoples minds of late. So, how does one determine the last time a database was restored? The answer is actually quite simple. Within the msdb database, there is a table (that maintains the history of database restores) called restorehistory.

The restorehistory table stores information that will help you better understand what kind of restore was performed, who did the restore, when it was done, and even some information about the backup that was used to perform the restore. You can read more about the specifics of this table from the msdn article – here.

A quick means to query this table is via the following script:

SELECT r.restore_date
		, r.destination_database_name
		, CASE r.restore_type
			WHEN 'D' THEN 'Database'
			WHEN 'F' THEN 'File'
			WHEN 'G' THEN 'FileGroup'
			WHEN 'I' THEN 'Differential'
			WHEN 'L' THEN 'Log'
			WHEN 'V' THEN 'VerifyOnly'
			END AS restore_type
		, r.replace
		, r.recovery
		, r.user_name
	FROM msdb.dbo.[restorehistory] r
	WHERE r.destination_database_name = 'TestDB'
	ORDER BY r.destination_database_name,r.restore_date DESC;

Now, that is an extremely simple query looking at my TestDB database. The results of that query on my system are as follows:

As is shown in the preceding illustration, I have multiple restores and restore types of this particular database. This is a good start. Let’s take this query up a notch. Not all production environments will have databases that are restored on the same server. What do we do to report on databases that are restored on the production server vs. the databases that exist on the production server? That is where this next query comes in to play.

DECLARE @DBName VARCHAR(128) = 'AdventureWorks2014' --NULL
	, @MostRecentRestore TINYINT = 1 --NULL --1
	, @RestoredDBsOnly TINYINT = NULL; --1

SELECT DatabaseName = [d].[name]
		, [d].[create_date]
		, [d].[compatibility_level]
		, [d].[collation_name]
		, r.restore_date
		, r.user_name AS RestoredBy
		, r.destination_database_name
		, r.restore_type
		, r.replace
		, r.recovery
		--, r.RowNum
	FROM master.sys.databases d
		LEFT OUTER JOIN ( SELECT r.restore_date
									, r.destination_database_name
									, CASE r.restore_type
										WHEN 'D' THEN 'Database'
										WHEN 'F' THEN 'File'
										WHEN 'G' THEN 'FileGroup'
										WHEN 'I' THEN 'Differential'
										WHEN 'L' THEN 'Log'
										WHEN 'V' THEN 'VerifyOnly'
										END AS restore_type
									, r.replace
									, r.recovery
									, r.user_name
									, RowNum = ROW_NUMBER() OVER ( PARTITION BY r.[destination_database_name] ORDER BY r.restore_date DESC )
								FROM msdb.dbo.[restorehistory] r
						) r
			ON d.name = r.[destination_database_name]
				AND r.RowNum = ISNULL(@MostRecentRestore, r.RowNum)
	WHERE 1 = 1
		AND ISNULL(r.RowNum, -1) = COALESCE(@RestoredDBsOnly, r.RowNum, -1)
		AND d.name = ISNULL(@DBName, d.name);

In this latter query, I have added a few things to help for various reporting needs. You never know when the CIO or an auditor is going to come to you with a request to filter the data differently for restored databases. In this vein, I added parameters that will help filter the results down to a specific database, return all databases, or return all databases that have a restore history on the server. In this case, I decided to use the AdventureWorks2014 database as my example database. You will want to change the parameter as needed. If you pass a null value to the database name, then all databases can be returned in the result set – depending on the values used for the remaining parameters.

Caveat

This is all fine and well if the databases are restored onto the same server as the source database, right? What about the case where the database is restored to a different server? Well, the query still works for reporting the restored databases on the server where the database is restored. The complexity comes when trying to correlate back to the original production server. My recommendation here is good documentation and an administrative linked server that is not accessible via the application accounts or any user but the DBA group.

Recap

I have provided two quick examples of how to retrieve the restore history for your databases. I recommend that the restore history be checked on a routine basis. You never know when an over-permissioned user may decide to restore a database for you and then have all of the rest of the users coming to you to determine what happened. This will also be essential when we routinely test our database backups. Stay tuned for the next article where I will explore that aspect of database restores.

How well do you know your environment? You probably know all of the jobs that are running, the frequency that indexes need to be rebuilt, and even which users have which level of access to each object in the SQL Server instance. Do you know that your applications are accessing deprecated datatypes over 300 million times a week? What if your TSQL constructs are a bit archaic? Do you know that the code needs to be updated? Do you know how to find deprecated uses in SQL Server?

In this article, I will explore how to use Extended Events to track feature use and abuse. To be more precise, I will share how this tool can help you better understand all of the ways that your applications have been abusing your database by continuing to employ the use of deprecated features, syntax, or constructs in general. In case you are a bit behind in your exploration of XEvents, I have the perfect solution for you – my series on the topic that is continually growing. You can explore the full list of articles in the series by visiting the table of contents – here.

Audit Deprecated Uses

I would dare say that most data professionals think there is some use of deprecated constructs, datatypes or features within their environment. I would double down on that and say that most do not know just how bad it really may be. To find just how bad it really is, we need to audit for the use and abuse of these deprecation events.

Right here would be a good time to point out that your mileage may vary. Some items that are deprecated are more painful than others. Some may be deprecated and may have been on the list for 10+ years at this point. The point is, know your environment and then use good judgement to determine which items in your results need the most attention to fix and update. Why? Well, things really may break especially if you are looking to upgrade to a new version of SQL Server. Just because an item is still available in your current edition, that does not ensure it will still be available in a future release of SQL Server.

Now for the juicy stuff. As I mentioned, finding when and where a deprecated feature or syntax is employed, there are a couple of neat little events within Extended Events that can help to track each time a deprecated feature is accessed or employed. How do we do that? Use the deprecation_announcement and deprecation_final_support events. To help create sessions to track these events, I have the following script to create an “audit” session to do exactly that.

USE master;
GO
-- Create the Event Session
IF EXISTS ( SELECT *
				FROM sys.server_event_sessions
				WHERE name = 'AuditDeprecated' )
	DROP EVENT SESSION AuditDeprecated
    ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE';
GO


CREATE EVENT SESSION [AuditDeprecated] ON SERVER
ADD EVENT sqlserver.deprecation_announcement (
	ACTION ( sqlserver.database_id, sqlserver.database_name,
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname,
	sqlserver.context_info, sqlserver.client_connection_id )
	WHERE  [sqlserver].[database_name] = N'AdventureWorks2014' 
			AND [sqlserver].[client_app_name] <> 'Microsoft SQL Server Management Studio - Transact-SQL IntelliSense'
			AND [sqlserver].[client_app_name] NOT LIKE 'Red Gate Software Ltd SQL Prompt%'
		),
ADD EVENT sqlserver.deprecation_final_support (
	ACTION ( sqlserver.database_id, sqlserver.database_name,
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname,
	sqlserver.context_info, sqlserver.client_connection_id )
	WHERE  [sqlserver].[database_name] = N'AdventureWorks2014' 
			AND [sqlserver].[client_app_name] <> 'Microsoft SQL Server Management Studio - Transact-SQL IntelliSense'
			AND [sqlserver].[client_app_name] NOT LIKE 'Red Gate Software Ltd SQL Prompt%'
		)
ADD TARGET package0.event_file (  SET filename = N'C:\Database\XE\AuditDeprecated.xel'
									, max_rollover_files = ( 25 ) )
WITH (  MAX_MEMORY = 4096 KB
		, EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
		, MAX_DISPATCH_LATENCY = 30 SECONDS
		, MAX_EVENT_SIZE = 0 KB
		, MEMORY_PARTITION_MODE = NONE
		, TRACK_CAUSALITY = ON
		, STARTUP_STATE = OFF );
GO

ALTER EVENT SESSION AuditDeprecated ON SERVER
STATE = START;

And just in case you are using 2008 or 2008R2, use this version instead.

/* 2008r2 and older */
USE master;
GO
-- Create the Event Session
IF EXISTS ( SELECT *
				FROM sys.server_event_sessions
				WHERE name = 'AuditDeprecated' )
	DROP EVENT SESSION AuditDeprecated
    ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE';
GO


CREATE EVENT SESSION [AuditDeprecated] ON SERVER
ADD EVENT sqlserver.deprecation_announcement (
	ACTION ( sqlserver.database_id, 
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname)
	WHERE  [sqlserver].[database_id] > 4 --exclude system databases
			AND [sqlserver].[client_app_name] <> 'Microsoft SQL Server Management Studio - Transact-SQL IntelliSense'
		),
ADD EVENT sqlserver.deprecation_final_support (
	ACTION ( sqlserver.database_id, 
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname)
	WHERE  [sqlserver].[database_id] > 4
			AND [sqlserver].[client_app_name] <> 'Microsoft SQL Server Management Studio - Transact-SQL IntelliSense'
		)
ADD TARGET package0.asynchronous_file_target (  SET filename = N'C:\Database\XE\AuditDeprecated.xel'
									, max_rollover_files = ( 25 ) )
WITH (  MAX_MEMORY = 4096 KB
		, EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
		, MAX_DISPATCH_LATENCY = 30 SECONDS
		, MAX_EVENT_SIZE = 0 KB
		, MEMORY_PARTITION_MODE = NONE
		, TRACK_CAUSALITY = ON
		, STARTUP_STATE = OFF );
GO

ALTER EVENT SESSION AuditDeprecated ON SERVER
STATE = START;

Slight differences between these two sessions. First, in the 2008 version of the script, I have to provide database ids instead of names. That is a shortcoming of 2008 and 2008R2 implementations of Extended Events. Additionally, the file target is different between the two (recall that they renamed the file target). And lastly, there are a few actions that I included in the 2012 version of the script that are not available in 2008 and R2.

With the session in place, I am now going to run through some sample scripts that will generate deprecation events to occur. I am sticking with my 2014 instance for this segment. That is important to note because different events may occur for different versions of SQL Server. Additionally, the parse script I will share will require a slight change for 2008 and r2 (again related to the file target name).

USE AdventureWorks2014;
GO
--String alias
SELECT 'FullName' = FirstName + ' ' + MiddleName + '. ' + LastName
	FROM Person.Person
GO

--# table name
CREATE TABLE # ( c1 INT )
GO

--3 part column name
SELECT TOP 1 AdventureWorks2014.Person.Person.LastName
		, *
	FROM Person.Person

SELECT TOP 1 Person.Person.LastName
		, *
	FROM Person.Person
GO

--use :: for function calls
SELECT *
	FROM :: fn_virtualfilestats(2, 1)
GO

--using hints without WITH keyword
SELECT TOP 1 Person.LastName
		, *
	FROM Person.Person (TABLOCK)
GO

--use fn_get_sql
DECLARE @Handle VARBINARY(64);
SELECT @Handle = sql_handle
	FROM sys.dm_exec_requests
	WHERE session_id = @@SPID
		AND request_id = 0;
SELECT *
	FROM sys.fn_get_sql(@Handle);
GO

--remote servers
SELECT @@REMSERVER

--vardecimal
exec sp_db_vardecimal_storage_format 'Adventureworks2014', 'ON'
sp_tableoption 'Person.Person', 'vardecimal storage format', 1

Now to take a peek at the data with this next script.

IF EXISTS (SELECT OBJECT_ID('tempdb.dbo.#xmlprocess'))
BEGIN
DROP TABLE #xmlprocess
END

SELECT CAST ([t2].[event_data] AS XML) AS event_data, t2.file_offset,t2.file_name, cte1.event_session_id--, ' AS event_predicate
	INTO #xmlprocess
	FROM ( SELECT REPLACE(CONVERT(NVARCHAR(128),sesf.value),'.xel','*.xel') AS targetvalue, ses.event_session_id
				FROM sys.server_event_sessions ses
					INNER JOIN sys.server_event_session_fields sesf
						ON ses.event_session_id = sesf.event_session_id
					--INNER JOIN sys.server_event_session_events sese
					--	ON ses.event_session_id = sese.event_session_id
				WHERE sesf.name = 'filename'
					AND ses.name = 'AuditDeprecated'
					) cte1
				OUTER APPLY sys.fn_xe_file_target_read_file(cte1.targetvalue,	NULL, NULL, NULL) t2
		;


SELECT x.event_data.value('(event/@name)[1]', 'varchar(50)') AS event_name
		, x.event_data.value('(event/@package)[1]', 'varchar(50)') AS package_name
		, DATEADD(hh, DATEDIFF(hh, GETUTCDATE(), CURRENT_TIMESTAMP),
					x.event_data.value('(event/@timestamp)[1]',
										'datetime2')) AS [timestamp]
		,event_data.value('(event/data[@name="feature_id"]/value)[1]','bigint') AS feature_id
		,event_data.value('(event/data[@name="feature"]/value)[1]','varchar(max)') AS feature
		,event_data.value('(event/data[@name="message"]/value)[1]','varchar(max)') AS message
		,event_data.value('(event/action[@name="client_app_name"]/value)[1]', 'varchar(max)') AS client_app_name
		,event_data.value('(event/action[@name="client_connection_id"]/value)[1]', 'uniqueidentifier') AS client_connection_id
		,event_data.value('(event/action[@name="client_hostname"]/value)[1]', 'varchar(max)') AS client_hostname
		,event_data.value('(event/action[@name="context_info"]/value)[1]', 'varbinary(max)') AS context_info
		,event_data.value('(event/action[@name="database_id"]/value)[1]', 'int') AS database_id
		,event_data.value('(event/action[@name="database_name"]/value)[1]', 'varchar(max)') AS database_name
		,event_data.value('(event/action[@name="nt_username"]/value)[1]', 'varchar(max)') AS nt_username
		,event_data.value('(event/action[@name="session_id"]/value)[1]', 'int') AS session_id
		,event_data.value('(event/action[@name="session_nt_username"]/value)[1]', 'varchar(max)') AS session_nt_username
		,event_data.value('(event/action[@name="sql_text"]/value)[1]', 'varchar(max)') AS sql_text
		,event_data.value('(event/action[@name="username"]/value)[1]', 'varchar(max)') AS username
	FROM #xmlprocess x
		LEFT OUTER JOIN sys.server_event_session_events sese
			ON x.event_data.value('(event/@name)[1]', 'varchar(50)') = sese.name
				AND x.event_session_id = sese.event_session_id
	ORDER BY timestamp
		, event_data.value('(event/action[@name="event_sequence"]/value)[1]',
							'varchar(max)');

Now at long last, I can see what kind of data I am generating (they are really wide so I am just posting a snip).

With this, I can see the feature_id along with the description and even the tsql that generated the event. What you don’t see in this is that I also trap the source machine and the user name. If there is an application name included in the connection string, I also trap that. These pieces of data can prove critical to efficiently troubleshooting and finding the source of these events.

From here, one might wish to explore all of the events generated from this session in order to ensure the environment is properly prepared for upgrade. Most tools do not evaluate the code thoroughly to trap all of these events. Instead they do a cursory look through stored procedures or at the data types. As we all should know, not every piece of SQL code is actually stored in the database or even is it cached at the time of analysis. This is the type of thing that requires a long running trace to prove that you are that rockstar DBA.

Conclusion

In the article today, I have shown how it is possible to see the deprecation alerts that may be generated in your environment. This data is what can help set you apart as a rockstar when it comes time for that migration. If you have yet to read my series on Extended Events, I highly recommend it. You can find that series here.

What do you do when you run into that missing database situation and the inevitable denial that will ensue?  This is when an audit can save the day.  Through an audit, you can discover who dropped the database and when it happened.  Then you have hard data to take back to the team to again ask what happened.  Taking the info from a previous article of mine, we can alter the script I published there and re-use it for our needs here.

DECLARE @DBName sysname = 'AdventureWorks2014'
		,@d1 DATETIME
		,@diff INT;

SELECT ObjectName
		  , ObjectID
		  , DatabaseName
		  , StartTime
		  , EventClass
		  , EventSubClass
		  , ObjectType
		  , ServerName
		  , LoginName
		  , NTUserName
		  , ApplicationName
		  , CASE EventClass
				WHEN 46
					THEN 'CREATE'
				WHEN 47
					THEN 'DROP'
				--WHEN 164
				--	THEN 'ALTER'
			END AS DDLOperation
		INTO #temp_trace  
	FROM sys.fn_trace_gettable(CONVERT(VARCHAR(150), 
			( SELECT REVERSE(SUBSTRING(REVERSE(path),
					CHARINDEX('\',REVERSE(path)),256)) + 'log.trc'
				FROM    sys.traces
				WHERE   is_default = 1)), DEFAULT) T  
  WHERE EventClass in (46,47) 
			AND EventSubclass = 0
			AND ObjectType = 16964-- i just want database related events   
			AND DatabaseName = ISNULL(@DBName,DatabaseName);


SELECT @d1 = MIN(StartTime) 
	FROM #temp_trace;

SET @diff= DATEDIFF(hh,@d1,GETDATE());


SELECT @diff AS HrsSinceFirstChange    
		, @d1 AS FirstChangeDate    
		, sv.name AS obj_type_desc
		, tt.ObjectType
		, tt.DDLOperation
		, tt.DatabaseName,tt.ObjectName,tt.StartTime
		, tt.EventClass,tt.EventSubClass
		, tt.ServerName,tt.LoginName, tt.NTUserName
		, tt.ApplicationName
		, (dense_rank() OVER (ORDER BY ObjectName,ObjectType ) )%2 AS l1     
		, (dense_rank() OVER (ORDER BY ObjectName,ObjectType,StartTime ))%2 AS l2    
	FROM #temp_trace tt
		INNER JOIN sys.trace_events AS te 
			ON tt.EventClass = te.trace_event_id
		INNER JOIN sys.trace_subclass_values tsv
			ON tt.EventClass = tsv.trace_event_id
			AND tt.ObjectType = tsv.subclass_value
		INNER JOIN master.dbo.spt_values sv 
			ON tsv.subclass_value = sv.number
			AND sv.type = 'EOD'
	ORDER BY StartTime DESC;

DROP TABLE #temp_trace;

This script will now query the default trace to determine when a database was dropped or created.  I am limiting this result set through the use of this filter: ObjectType = 16964.  In addition to that, I have also trimmed the result-set down to just look for drop or create events.

This is the type of information that is already available within the default trace.  What if you wished to not be entirely dependent on the default trace for that information?  As luck would have it, you don’t need to be solely dependent on the default trace.  Instead you can use the robust tool called extended events.  If you would like to be able to take advantage of Extended Events to track this information, I recommend you read my follow-up article here.

This has been a republication of my original content first posted here.

With all of this change occurring within SQL Server, sometimes it is difficult to figure out how to troubleshoot or track issues that relate to new features. Within the need to figure out how to troubleshoot the new features, there is the need to understand what tools are out there to help troubleshoot.

It is no big secret that Profiler cannot help you trace any of the new features. To help perform the task of tracing events related to new features one must take advantage of the power of Extended Events. Even knowing that you can use XEvents, there is still a need to know what is related to the feature.

In this article, I will show how you can discover the events related to specific features. In addition, I will provide enough base information to help you perform other discovery type queries on your own as you continue your exploration of XEvents. In case you are a bit behind in your exploration of XEvents, I have the perfect solution for you – my series on the topic that is continually growing. You can explore the full list of articles in the series by visiting the table of contents – here.

Events and Features

More and more I am being asked how to track which events belong to which features. I have also been seeing more people ask for a way to list the new features supported by XEvents. Thankfully there is adequate information within the XEvent metadata to help retrieve this type of information. The downside is that there is a bit of homework that must be done across a few versions of SQL Server to help produce the desired information. I have done that work and pulled the information into the following query to help make it easier for anybody else wishing to dive in and compare features and events across versions of SQL Server.

DECLARE @ServerMajorVersion DECIMAL(4, 2)
SELECT @ServerMajorVersion = CONVERT(DECIMAL(4, 2), PARSENAME(dt.fqn, 4) + '.'
		+ PARSENAME(dt.fqn, 3))
	FROM ( SELECT CONVERT(VARCHAR(20), SERVERPROPERTY('ProductVersion'))
			) dt ( fqn );

IF OBJECT_ID('tempdb.dbo.#XEVersions') IS NOT NULL
BEGIN
	DROP TABLE #XEVersions;
END

CREATE TABLE #XEVersions
	(VersionID INT IDENTITY(1,1)
		, Product VARCHAR(32)
		, ServerMajorVersion DECIMAL(4, 2) --INDEX CI_ServerMajorVer CLUSTERED
	);
CREATE CLUSTERED INDEX CI_ServerMajorVer ON	#XEVersions (VersionID);

INSERT INTO	#XEVersions
		( Product
			, ServerMajorVersion
		)
	VALUES
		( 'SQL Server 2008', 10.00)
		,( 'SQL Server 2008R2', 10.50)
		,( 'SQL Server 2012', 11.00)
		,( 'SQL Server 2014', 12.00)
		,( 'SQL Server 2016', 13.00);


IF OBJECT_ID('tempdb.dbo.#XECounts') IS NOT NULL
BEGIN
	DROP TABLE #XECounts;
END

CREATE TABLE #XECounts
	(
		[Feature] [NVARCHAR](3072) NOT NULL
		, [FeatureEventCount] [INT] NULL
		, [SQLVersion] [DECIMAL](4, 2) NULL
	);

INSERT INTO #XECounts
		( [Feature], [FeatureEventCount], [SQLVersion] )
	VALUES	( N'_UnAssigned', 27, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'access_methods', 23, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'alwayson', 100, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'broker', 42, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'cdc_logscan', 2, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'change_tracking', 2, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'checkpoint', 17, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'ckpt_trace', 37, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'ckptworker_trace', 21, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'clr', 3, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'cursor', 18, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'database', 42, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'deadlock_monitor', 7, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'deploy', 14, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'errors', 20, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'exception', 4, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'execution', 115, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'filetable', 13, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'fulltext', 8, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'garbage_collection', 1, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'gc', 4, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'hadr', 1, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'index', 9, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'init', 1, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'io', 33, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'latch', 6, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'lock', 15, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'memory', 35, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'merge_trace', 23, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'oledb', 5, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'optimization', 2, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'process', 12, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'query_store', 31, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'replication', 8, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'scheduling', 26, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'security', 3, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'server', 19, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'session', 17, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'storage_management', 13, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'synchronization', 2, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'task', 2, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'transaction', 10, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'transactions', 48, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'transmitter', 9, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'transport', 9, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'ucs', 8, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'warnings', 7, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'xtp', 15, CAST(12.00 AS DECIMAL(4, 2)) )
		, ( N'_UnAssigned', 9, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'access_methods', 6, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'broker', 73, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'cdc_logscan', 2, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'change_tracking', 1, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'clr', 2, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'cursor', 11, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'database', 21, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'deadlock_monitor', 6, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'errors', 5, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'exception', 3, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'execution', 14, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'io', 20, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'latch', 5, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'lock', 6, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'memory', 12, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'process', 12, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'replication', 7, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'scheduling', 21, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'server', 21, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'synchronization', 1, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'transactions', 6, CAST(10.50 AS DECIMAL(4, 2)) )
		, ( N'_UnAssigned', 83, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'access_methods', 19, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'alwayson', 117, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'appdomain', 5, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'backup_restore', 1, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'broker', 42, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'cdc_logscan', 2, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'change_tracking', 2, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'checkpoint', 19, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'ckpt_close', 1, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'ckpt_controller', 9, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'ckpt_trace', 40, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'ckptworker_trace', 21, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'clr', 3, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'columnstore', 41, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'cursor', 18, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'data_masking', 3, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'database', 48, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'dbseed', 14, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'deadlock_monitor', 7, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'deploy', 17, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'errors', 24, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'exception', 6, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'execution', 119, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'extensibility_errors', 1, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'filetable', 13, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'fulltext', 8, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'garbage_collection', 1, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'gc', 4, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'hadr', 1, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'index', 9, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'init', 1, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'io', 50, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'json', 7, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'latch', 8, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'lock', 16, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'memory', 43, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'merge_trace', 27, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'oledb', 6, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'optimization', 3, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'process', 12, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'query_store', 68, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'redo', 5, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'replication', 10, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'satellite_authentication', 2, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'satellite_cargo_messages', 12, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'satellite_launchpad_communication', 5, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'scheduling', 26, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'security', 9, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'server', 23, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'session', 31, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'storage_management', 23, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'stretch', 62, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'synchronization', 2, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'task', 2, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'temporal', 14, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'transaction', 14, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'transactions', 53, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'transmitter', 9, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'transport', 10, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'ucs', 8, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'unload', 1, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'warnings', 7, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'xtp', 31, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'xtp_encryption', 3, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'xtp_io_governance', 2, CAST(13.00 AS DECIMAL(4, 2)) )
		, ( N'_UnAssigned', 5, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'access_methods', 6, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'broker', 73, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'cdc_logscan', 2, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'change_tracking', 1, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'clr', 2, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'cursor', 11, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'database', 21, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'deadlock_monitor', 6, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'errors', 5, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'exception', 3, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'execution', 14, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'io', 20, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'latch', 4, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'lock', 6, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'memory', 12, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'process', 12, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'replication', 7, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'scheduling', 21, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'server', 21, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'synchronization', 1, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'transactions', 2, CAST(10.00 AS DECIMAL(4, 2)) )
		, ( N'_UnAssigned', 26, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'access_methods', 18, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'alwayson', 97, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'broker', 42, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'cdc_logscan', 2, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'change_tracking', 2, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'clr', 3, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'cursor', 18, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'database', 40, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'deadlock_monitor', 6, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'errors', 19, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'exception', 4, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'execution', 106, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'filetable', 13, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'fulltext', 8, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'garbage_collection', 1, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'hadr', 1, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'index', 9, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'init', 1, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'io', 21, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'latch', 6, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'lock', 11, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'memory', 34, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'oledb', 5, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'process', 12, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'replication', 8, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'scheduling', 24, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'security', 3, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'server', 15, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'session', 14, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'storage_management', 1, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'synchronization', 2, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'task', 2, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'transactions', 46, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'transmitter', 9, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'transport', 9, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'ucs', 1, CAST(11.00 AS DECIMAL(4, 2)) )
		, ( N'warnings', 7, CAST(11.00 AS DECIMAL(4, 2)) );

IF OBJECT_ID('tempdb.dbo.#presel') IS NOT NULL
BEGIN
	DROP TABLE #presel;
END

SELECT oc.name AS EventName
		, oc.description AS column_description
		, ca.map_value AS SearchKeyword
		, @ServerMajorVersion AS SQLVersion
	INTO #presel
	FROM sys.dm_xe_objects oc
		OUTER APPLY ( SELECT mv.map_value
							FROM sys.dm_xe_object_columns occ
								INNER JOIN sys.dm_xe_map_values mv
									ON occ.type_name = mv.name
										AND occ.column_value = mv.map_key
										AND occ.object_package_guid = mv.object_package_guid
							WHERE occ.name = 'KEYWORD'
								AND occ.object_name = oc.name
					) ca
	WHERE oc.object_type = 'event';

/* version specific features */
SELECT *
	FROM #XECounts xc
	WHERE xc.SQLVersion = @ServerMajorVersion
		AND xc.Feature NOT IN ( SELECT Feature
									FROM #XECounts x
									WHERE x.SQLVersion < @ServerMajorVersion );

/* change in feature counts*/
SELECT xv.Product
		, xc.Feature
		, xc.FeatureEventCount
		, ISNULL(fo.FeatureEventCount, 0) AS PreviousEditionCount
		, xc.FeatureEventCount - ISNULL(fo.FeatureEventCount, 0) AS NewEventCount
		, CASE	WHEN ISNULL(fo.Feature, '') = '' THEN 'New Feature'
				ELSE 'Existing Feature'
			END AS NewOrOld
	FROM #XECounts xc
		INNER JOIN #XEVersions xv
			ON xc.SQLVersion = xv.ServerMajorVersion
		FULL OUTER JOIN ( SELECT x2.Feature
									, x2.FeatureEventCount
								FROM #XECounts x2
									INNER JOIN #XEVersions xev
										ON x2.SQLVersion = xev.ServerMajorVersion
								WHERE xev.VersionID = ( SELECT VersionID - 1
															FROM #XEVersions x
															WHERE x.ServerMajorVersion = @ServerMajorVersion
														)
						) fo
			ON xc.Feature = fo.Feature
	WHERE 1 = 1
		AND xc.SQLVersion = @ServerMajorVersion;

/* specific counts for the current server just in case new events rolled out with a patch*/
SELECT MAX(xev.Product) AS Product
		, ISNULL(ou.Feature, '        Grand Total') AS FeatureCount
		, SUM(ou.FeatureEventCount) AS FeatureEventCount
	FROM ( SELECT ISNULL(p.SearchKeyword, '_UnAssigned') AS Feature
					, COUNT(*) AS FeatureEventCount
					, p.SQLVersion
				FROM #presel p
				GROUP BY p.SearchKeyword, p.SQLVersion
			) ou
		INNER JOIN #XEVersions xev
			ON ou.SQLVersion = xev.ServerMajorVersion
	GROUP BY ou.Feature WITH ROLLUP;

In the preceding query, I have provided an aggregation of the events and features across multiple editions of SQL Server. This aggregate of data is necessary to help see how the features are changing and how the events are changing within the features that remain constant from one version to the next.

Once that data is aggregated, I then provide a couple of sample queries that show what can be done with the data. I need to stress here that I have not provided an exhaustive set of queries to explore this data seven ways to Sunday. I have left those types of exercises up to the reader to explore and experiment.

The first example query shows how to pull the new features that can be “Traced” in the current version of SQL Server on which the query is being executed. The second example provides a difference count between versions of SQL Server to help illustrate the evolution of XEvents within SQL Server. The third query is a simple query to list out the number of events for each feature in your version of SQL Server.

From here, one might wish to explore all of the events that are related to a specific feature. This would be easily accomplished by querying out the data from the #presel temp table based on the feature name. Orrrr…one could query the desired feature by following the guidelines in this article.

Conclusion

In the article today, I have shown how it is possible to see the new events and how they relate to the various features within SQL Server. Being able to correlate events that can trap information about new features can and will help you evolve into that rock-star DBA you are trying to become!

I bring that up for a couple of reasons. First and foremost being that the topic was recently raised as a “need” by some colleagues. The second being that I see the need and how it was lacking in coverage by anything I had already written.

What is this gaping hole in the coverage of Extended Events? To be honest, it is not a very complicated topic or very difficult gap to fill. It’s just something that has been overlooked. The gap boils down to this: how does one consistently find the correct path to the Extended Event Log file (XEL file)?

Filling the Gap

The gap I will be working to fill in this article deals with consistently finding the file path for Extended Event (XE) sessions. This gap rises due a few different things such as the ability to define a target in different manners, being able to move the logs directory, or even the fact that a target may not be added to the session (let alone a file target). These causes can all contribute to a bit of frustration and may pose as different symptoms when trying to get the file path.

One additional complication is tied to the running state of a session. For that complication, I talked briefly about it in my article about better practices. The state of the session could have an impact and could cause frustration when trying to retrieve the file path. All of these things are considerations that must be made when trying to retrieve the file path.

To find the file path, let’s start with some basics. The default path for XEL files is in the log directory of the instance. In addition to this default behavior, each session stores metadata about running sessions as well as deployed sessions that is accessible from DMVs and system catalogs.

Log Path

The first basic to tackle is the log path for the instance. I can query for the log path of the error log for the instance and rely on that as the path of my xel files. Should I choose this method, then I could execute the following query.

DECLARE @ErrorPath VARCHAR(256) = CONVERT(VARCHAR(256),SERVERPROPERTY('ErrorLogFileName'));

SELECT LEFT(@ErrorPath,
			LEN(@ErrorPath) - CHARINDEX('\', REVERSE(@ErrorPath)) + 1) AS ErrorPath;

Executing that query on my SQL 2014 instance produces the following results:

Now the obvious problems with this method come from the fact that relying on this data is relying upon an assumption that you have set all of your XE Sessions to use the default log path. If you have declared your sessions to use a file target and did not specify a path, then the assumption is safe. However, if you are detail oriented, you probably have been somewhat explicit in how you define your event file target. This brings us to the next topic – defining the path.

Defining The Path

While a bit of bird-walk, it is necessary to cover this topic at this juncture. This brief discussion will help to understand some of the other issues with retrieving the path consistently.

When defining the path of the event path, there is a bit of flexibility in how one can define the file to be used. You can either declare the file as just the file name, or you can define the file as the folder path along with the file name. Both methods are completely legitimate. Unfortunately, this flexibility is what causes some of the pain with retrieving the file path consistently.

Let’s take a look at two quick, and acceptable, ways to add an event file to an XE Session. I will be re-using a session from a previous article for these examples.

CREATE EVENT SESSION [AuditSelects] ON SERVER
ADD EVENT sqlserver.degree_of_parallelism (
	ACTION ( sqlserver.database_id, sqlserver.database_name,
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname,
	sqlserver.context_info, sqlserver.client_connection_id )
	WHERE  statement_type = ( 1 ) --type 1 is a select statement
	AND [sqlserver].[database_name] = N'AdventureWorks2014' 
			AND [sqlserver].[client_app_name] <> 'Microsoft SQL Server Management Studio - Transact-SQL IntelliSense'
			AND [sqlserver].[client_app_name] NOT LIKE 'Red Gate Software Ltd SQL Prompt%'
		)
ADD TARGET package0.event_file (  SET filename = N'C:\Database\XE\AuditSelects.xel'
									, max_rollover_files = ( 25 ) )
;
GO

This is a pretty standard format I use for creating my sessions. The section I want to highlight though is the set of the filename near the end of the script. I will generally define the entire path for my xel files when I create a session – just as I have done in this example. Defining this path helps me to know exactly where I am putting the session for starters. Equally as important is that this path is easier to retrieve from metadata because I have explicitly defined the path.

Take this next example of the same session but with one minor difference.

CREATE EVENT SESSION [AuditSelects] ON SERVER
ADD EVENT sqlserver.degree_of_parallelism (
	ACTION ( sqlserver.database_id, sqlserver.database_name,
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname,
	sqlserver.context_info, sqlserver.client_connection_id )
	WHERE  statement_type = ( 1 ) --type 1 is a select statement
	AND [sqlserver].[database_name] = N'AdventureWorks2014' 
			AND [sqlserver].[client_app_name] <> 'Microsoft SQL Server Management Studio - Transact-SQL IntelliSense'
			AND [sqlserver].[client_app_name] NOT LIKE 'Red Gate Software Ltd SQL Prompt%'
		)
ADD TARGET package0.event_file (  SET filename = N'AuditSelects.xel'
									, max_rollover_files = ( 25 ) )
;
GO

The minor difference in this example is just in how the filename was defined. It is technically accurate and acceptable to only use the filename instead of the path and filename as I did in the previous example. The problem here comes from the retrieval of the path from metadata.

Getting the Path

In the previous two examples, I showed two methods of defining the filename within a session. In the former example, this means I can more easily find the path. However, if I use the script in the beginning of the article to find my xel file paths, then I have made an inaccurate assumption. This leads to an inconsistency in how the file path is fetched. In the latter example, the assumptions concerning the default log path would be valid and could be applied here.

Since the assumptions do not prove to be consistent or accurate across the board, we need to both evaluate how the data for each would look and we would need to see how to retrieve this path more consistently. First up is looking at how the data for each of these example sessions would be stored in metadata.

SELECT sesf.*
		, ses.name AS SessionName
	FROM sys.server_event_sessions ses
		INNER JOIN sys.server_event_session_fields sesf
			ON ses.event_session_id = sesf.event_session_id
	WHERE sesf.name = 'filename'
		AND ses.name LIKE 'auditselect%'

In the preceding query, I am looking at the deployed session metadata for two sessions, both of which start with “AuditSelect”. In the sys.server_event_session_fields view, there is an attribute called name that contains the value “filename”. This value will only exist if the session has an event file target defined for the session. Running the query will yield the following results (on my system).

In this result set, I can see there is one of each of the two event file definition methods I described in the previous section. In the green highlight you will see that I have the entire filepath. In the red highlight, you will only see the filename. Nothing special has been done to this data in the return as you can verify from the posted script. These are the actual stored values. So this would indeed seem like we have a bit of a problem, right?

Well, this is where we get to be a little bit imaginative and employ a script such as the following:

/* Deployed Session Metadata */
DECLARE @SessionName VARCHAR(128) = 'AuditSelects';

IF EXISTS ( SELECT 1
				FROM sys.server_event_sessions ses
					INNER JOIN sys.server_event_session_fields sesf
						ON ses.event_session_id = sesf.event_session_id
				WHERE sesf.name = 'filename'
					AND ses.name = @SessionName )
	BEGIN
		IF ( SELECT CHARINDEX('\', CONVERT(VARCHAR(256),sesf.value))
						FROM sys.server_event_sessions ses
							INNER JOIN sys.server_event_session_fields sesf
								ON ses.event_session_id = sesf.event_session_id
						WHERE sesf.name = 'filename'
							AND ses.name = @SessionName ) > 0
			BEGIN
				SELECT ses.name AS SessionName
						, LEFT(CONVERT(VARCHAR(256),sesf.value),
								LEN(CONVERT(VARCHAR(256),sesf.value)) - CHARINDEX('\',
																REVERSE(CONVERT(VARCHAR(256),sesf.value)))+1) AS XEFilePath
					FROM sys.server_event_sessions ses
						INNER JOIN sys.server_event_session_fields sesf
							ON ses.event_session_id = sesf.event_session_id
					WHERE sesf.name = 'filename'
						AND ses.name = @SessionName;
			END;
		ELSE
			BEGIN
				SELECT TOP 1 cte1.SessionName
						, LEFT(t2.file_name,
								LEN(t2.file_name) - CHARINDEX('\',
																REVERSE(t2.file_name))
								+ 1) AS XEFilePath
					FROM ( SELECT REPLACE(CONVERT(NVARCHAR(128), sesf.value),
											'.xel', '*.xel') AS targetvalue
									, ses.name AS SessionName
								FROM sys.server_event_sessions ses
									INNER JOIN sys.server_event_session_fields sesf
										ON ses.event_session_id = sesf.event_session_id
								WHERE sesf.name = 'filename'
									AND ses.name = @SessionName
							) cte1
						CROSS APPLY sys.fn_xe_file_target_read_file(cte1.targetvalue,
																NULL, NULL, NULL) t2;
			END;
	END;

Wow, that is considerably longer than the first example in the article. That is true! This script accounts for a few of the conditions that cause inconsistencies (but not yet all of them) in retrieving the file path for a deployed event session. The key here is to understand this will fetch the path whether you enter a complete path or not while creating the session. I also throw a check in there to confirm that the session has a file target as well as a check to ensure the session is deployed to the server.

Looking at the output, I get results that are somewhat friendly and easy to understand without any of the guesswork.

What if the session has never been started and did not have the full path declared? Well, that is one of the inconsistencies I am working on still. (Update 2019/05/17 – An updated script is coming out soon that handles this little issue.)

There is an alternative to this method as well. For the time being, this method would also be recommended in the event the session being researched happens to be one of the system sessions that is “private”. This next code chunk will show two different methods to parse the file path from running session metadata in the DMVs.

/* use query() method for running session */
SELECT cte1.SessionName
		,CAST(cte1.target_data.query('data(/EventFileTarget/File/@name)') AS VARCHAR(256)) AS XEFilePath
	INTO #presel
	FROM ( SELECT target_data = CONVERT(XML, target_data), s.name AS SessionName
				FROM sys.dm_xe_session_targets t
					INNER JOIN sys.dm_xe_sessions s
						ON t.event_session_address = s.address
				WHERE t.target_name = 'event_file'
					AND s.name = 'system_health'
			) cte1;

SELECT p.SessionName
		, LEFT(p.XEFilePath,
				LEN(p.XEFilePath) - CHARINDEX('\', REVERSE(p.XEFilePath)) + 1) AS XEFilePath
	FROM #presel p;

DROP TABLE #presel;
GO

/* use value() method on Running Session Metadata */
SELECT FileEvent.FileTarget.value('(@name)[1]',
										'varchar(256)') AS [FilePath]
		,cte1.SessionName
	INTO #presel
	FROM ( SELECT target_data = CONVERT(XML, target_data), s.name AS SessionName
				FROM sys.dm_xe_session_targets t
					INNER JOIN sys.dm_xe_sessions s
						ON t.event_session_address = s.address
				WHERE t.target_name = 'event_file'
					AND s.name = 'system_health'
			) cte1
	CROSS APPLY cte1.target_data.nodes('//EventFileTarget/File')FileEvent ( FileTarget );	

SELECT p.SessionName
		, LEFT(p.FilePath,
				LEN(p.FilePath) - CHARINDEX('\', REVERSE(p.FilePath)) + 1) AS XEFilePath
	FROM #presel p;

DROP TABLE #presel;
GO

Conclusion

In the article today, I have shown some of the internals to retrieving file paths for Extended Event Sessions. I dove into metadata to pull out the path for the session and discussed some concerns for some of these methods. In the end, you have a few viable options to help retrieve the file path in a more consistent fashion.

To read more about Extended Events, I recommend this series of articles.

In case you missed it (many did), TSQL Tuesday was a challenging event this month. I invited people to do a put a little more into writing a post than what they may usually do. There were some very good reasons for this. If you are interested, take a look back at the invite and see if you maybe want to give it a go outside the bounds of TSQL Tuesday. You can check out the original post here.

Before I get into the nitty gritty I have a confession. This topic was a reminder for myself as much as it was a challenge to others to help me continue to drive and improve in various areas as I see fit.

There are other dirty little secrets too. Some may become apparent as you read through the recap.

Recap of the Event

One of the tricks to becoming and staying a top tier data talent or professional is a perpetual cycle to learn, adapt, change, and evolve. We must be in a continual cycle of self evaluation and self modification. Let’s call this by something else – we must be agile. There I said the five letter word. Think about it in broad strokes with your career – it is a development process with perpetual evaluation, review and tweaks.

Now think about the invite and see how that fits with what I just said or with the, cough cough, agile flow. You start (albeit very basically) with a need for enhancement, then you plan which pieces of the enhancement you can accomplish, you then do the work (whether successful or not), then after you deliver the work you conclude with a retrospective (what went well and what needs to change). Yes! I do feel rather dirty for sneaking this on everybody like this. That said, when you think about the model and apply it in broad strokes to your career path – it has merit.

Another way of viewing this is to think in terms of the following flowchart to help improve your personal mindset or maybe improve your personal mental power. The process is repetitive and follows a natural course. Once you have acted on some plan, you must review the performance and results and then gauge where your mindset needs to go from there to improve.

In other Words

Did you just look at the picture or did you explore the picture? If you hover the picture, you will find there are links to this months participants. There were only eight so not a ton of exploration is necessary.

Here are my thoughts on each of the posts submitted this month:

Wayne Sheffield (blog | twitter) – You can find his link in the big arrow that restarts the cycle. I put his link here because he ran into a ton of blockers during his experiment and he is at a spot of practically restarting – again. This is not the first time he has restarted in his quest to learn more about Availability Groups. Wayne fully admits he is deficient in AG and states near the end of the post that he had to humble himself going through this exercise. That is awesome! We could all use a little humility on a more regular basis.

Mala Mahadevan (blog | twitter) – You can find her link in the “Results” circle. The reason for this choice is that Mala discusses her midlife crisis – erm career change. Mala held out for quite a while looking for just the right opportunity. When it came, she snatched it up. Along with that career change, she has implemented a plan to become more active in blogging and to learn more and more through various avenues. The increase in blogging and the ability to stick to her guns resulted in a new job/career she seems to be happy with at the moment.

Robert Davis (blog | twitter) – Robert found himself placed in the performance circle thanks to his article involving a third party backup utility that should be heavy on the performance side. Robert needed something interesting to push him to reacquaint himself with this tool. Once he found that project that required just a touch of ingenuity, performance and a way to avoid the GUI, Robert found himself right at home with a great solution for his environment.

Kennie Nybo Pontoppidan (blog | twitter) – Kennie landed in the Actions node mostly because he decided to take the challenge and act on his long time desire to get better at the new temporal features. To do that, he decided to read a book by Snodgrass which seriously sounded like something from Harry Potter to me. Kennie outlines a bunch of information that he learned from the book such as tracking time based data from either a transaction or valid-time perspective.

Kenneth Fisher (blog | twitter) – I placed this one into the behavior node. Maybe it is a bit of a stretch, but it seems to make sense since he discussed some behavioral differences between Azure DB and SQL Server. Things just do not work exactly the same between the two. You will need to understand these differences if you find yourself in a spot where you must work with both.

Steve Jones (blog | twitter) – When looking through the image, you will find that Steve landed solidly in the mindset node. When I read his contribution, I got the full impression that his mind was 100% in the right place. He set out to learn something and try to get better at it. Additionally, he blogged about a topic that is near and dear to me – Extended Events. Have I mentioned before that I have a lot of content about XE? You can read a bunch of it here. Like Wayne, Steve was humble near the end of his article. He notes that he was clumsy as he started working with XE but that he is glad he did it as well. Read his article. He gave me a great idea of another use for XE and I am sure it may sound good to you too!

Rob Farley (blog | twitter) – I planted Rob firmly on the attitude node. It seems clear to me that Rob had loads of attitude throughout his article about Operational Analytics. The attitude I perceived was that of humility and yearning. Rob feels like he has a lot to learn and his attitude is in the right place it seems to keep him going while he tries to learn more in the field of Operational Analytics.

My Contribution link can be found by clicking on any spot in the image that is not already described. I wrote about my experiences with trying to pick up a little on JSON.

That is a wrap of all eight contributions. If you did not contribute this month, I recommend that you still try to do something with the challenge issued with this months TSQL Tuesday.

Edit: Added links to the articles with each persons name in the event this page is being viewed with Firefox. There seems to be an issue with the links in the image map within Firefox.

Over the course of a few recent articles, I have included little hints here and there alluding to this article. In fact, now would be a good time to review one of these articles in preparation for what I will be sharing today. Take a moment to refresh your memory on this article about finding the right event – here.

Backstory

I have to be a little honest here. Prior to somebody asking how they could possibly achieve a statement audit via extended events, I had not considered it as a tool for the job. I would have relied on Audit (which is Extended Event related), or some home grown set of triggers. In this particular request, Audit was not fulfilling the want and custom triggers was not an option. Another option might have included the purchase of third party software but there are times when budget does not allow for nice expensive shiny software.

So, with a little prodding, I hopped into the metadata and poked around a bit to see what I could come up with to achieve this low-budget audit solution.

Low-Budget Audit

Using the handy scripts I have shown previously (and that I just linked to), I was able to explore the Extended Events metadata and find just what may work properly. The requirements in this case were that it needed to be done in XEvents and that it must capture SELECT statements. To find the events that seemed plausible for this task, I plugged the term “SELECT” into my queries as follows:

DECLARE @TermDescription VARCHAR(64) = 'Select'
	, @ColumnDesc VARCHAR(64) = 'statement'
	, @ReadFlag VARCHAR(64) = NULL; --readonly' --ALL if all columntypes are desired --data --customizable
 
SELECT oc.object_name AS EventName
		, oc.name AS column_name
		, oc.type_name
		, oc.column_type AS column_type
		, oc.column_value AS column_value
		, oc.description AS column_description
		, ca.map_value AS SearchKeyword
		, oc.column_type
	FROM sys.dm_xe_object_columns oc
		CROSS APPLY ( SELECT mv.map_value
							FROM sys.dm_xe_object_columns occ
								INNER JOIN sys.dm_xe_map_values mv
									ON occ.type_name = mv.name
										AND occ.column_value = mv.map_key
										AND occ.object_package_guid = mv.object_package_guid
							WHERE occ.name = 'KEYWORD'
								AND occ.object_name = oc.object_name
					) ca
	WHERE (oc.object_name = @TermDescription
		OR oc.description LIKE '%' + ISNULL(@ColumnDesc,@TermDescription) + '%')
		AND oc.column_type = ISNULL(@ReadFlag, oc.column_type)
	ORDER BY EventName, column_name;

GO

From this query, there was really only one event that made any sense for my task – “degree_of_parallelism”. There was another event returned in the result set, but it made no sense to me as a possible candidate event for auditing select statements (it was ucs_transmitter_reclassify). From the results, I then took the keyword associated to degree_of_parallelism to see what else might be pertinent. I plugged that keyword “execution” into the following query:

DECLARE @Keyword VARCHAR(64) = 'execution' --'synchronization'
 
SELECT oc.OBJECT_NAME AS EventName
		,oc.name AS column_name, oc.type_name
		,oc.column_type AS column_type
		,oc.column_value AS column_value
		,oc.description AS column_description
		,REVERSE(LEFT(REVERSE(olm.name),CHARINDEX('\',REVERSE(olm.name))-1)) AS DLLName
		,olm.file_version
		,xp.name AS PackageName
		,mv.map_value AS SearchKeyword
		,ch.Channel
	FROM sys.dm_xe_object_columns oc
		INNER JOIN sys.dm_xe_map_values mv
			ON oc.type_name = mv.name
			AND oc.column_value = mv.map_key
			AND oc.name = 'KEYWORD'
		INNER JOIN sys.dm_xe_packages xp
			ON oc.object_package_guid = xp.guid
		INNER JOIN sys.dm_os_loaded_modules olm
			ON xp.module_address = olm.base_address
		INNER JOIN (SELECT c.object_name AS EventName,c.object_package_guid AS PkgGuid, v.map_value AS Channel
					FROM sys.dm_xe_object_columns c
						INNER JOIN sys.dm_xe_map_values v
							ON c.type_name = v.name
								AND c.column_value = CAST(v.map_key AS NVARCHAR)
					WHERE c.name = 'channel') ch
			ON ch.EventName = oc.object_name
			AND ch.PkgGuid = oc.object_package_guid
	WHERE mv.map_value = @Keyword
	;
GO

The results of the previous query will be quite a bit more substantial. Some make sense to include in the audit and some do not. I opted to not included any of the events to keep things as clean and simple as possible.

More Requirements

I now have the base events covered that I want to use for my audit. That said, my base requirements are just not extensive enough. When I audit something, I really want to know who did it, when it was done, where it originated and what was it that they did. The degree_of_parallelism event will capture all of the select statements but it does not meet these additional requirements.

In order to meet the additional requirements, I will attach a handful of actions to the degree_of_parallelism event. The addition of the actions will provide all of the data I want and need. Combine the event and actions together, I came up with the following session definition.

USE master;
GO
-- Create the Event Session
IF EXISTS ( SELECT *
				FROM sys.server_event_sessions
				WHERE name = 'AuditSelects' )
	DROP EVENT SESSION AuditSelects
    ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE';
GO


CREATE EVENT SESSION [AuditSelects] ON SERVER
ADD EVENT sqlserver.degree_of_parallelism (
	ACTION ( sqlserver.database_id, sqlserver.database_name,
	sqlserver.nt_username, sqlserver.sql_text, sqlserver.username,sqlserver.session_nt_username,
	sqlserver.client_app_name, sqlserver.session_id, sqlserver.client_hostname,
	sqlserver.context_info, sqlserver.client_connection_id )
	WHERE  statement_type = ( 1 ) --type 1 is a select statement
	AND [sqlserver].[database_name] = N'AdventureWorks2014' 
			AND [sqlserver].[client_app_name] <> 'Microsoft SQL Server Management Studio - Transact-SQL IntelliSense'
			AND [sqlserver].[client_app_name] NOT LIKE 'Red Gate Software Ltd SQL Prompt%'
		)
ADD TARGET package0.event_file (  SET filename = N'C:\Database\XE\AuditSelects.xel'
									, max_rollover_files = ( 25 ) )
WITH (  MAX_MEMORY = 4096 KB
		, EVENT_RETENTION_MODE = ALLOW_SINGLE_EVENT_LOSS
		, MAX_DISPATCH_LATENCY = 30 SECONDS
		, MAX_EVENT_SIZE = 0 KB
		, MEMORY_PARTITION_MODE = NONE
		, TRACK_CAUSALITY = ON
		, STARTUP_STATE = OFF );
GO

This is a very simple session to pull together. I have added a few things in on my predicate to help limit the scope of the session to the AdventureWorks2014 database and to make sure I am not trapping events related to the code completion tools shown. Notice here also that there is a potential chance to optimize this session. Can you find it? Hint: It may be in the predicate. Second hint: re-read this article about predicates.

Now the trick to what makes this work to audit only the select statements is right there in the predicate. I have instructed the session to ignore any statement_type that is not a value of 1. As it turns out, statement_type of 1 is a select statement. To see how these values map, here is a quick query and the correlated results.

DECLARE @EventName VARCHAR(64) = 'degree_of_parallelism'
	,@ReadFlag VARCHAR(64) = 'readonly' --readonly' --ALL if all columntypes are desired
 
SELECT oc.object_name as EventName,oc.name as ColName,mv.name as MapName, map_key, map_value 
	FROM sys.dm_xe_map_values mv
		Inner Join sys.dm_xe_object_columns oc
			on mv.name = oc.type_name
			AND mv.object_package_guid = oc.object_package_guid
	WHERE oc.object_name = @EventName
		AND oc.column_type <> @ReadFlag
	ORDER BY MapName, mv.map_key;
GO

Based on these values, should you want to audit a different query type, just change the predicate to use the map_key value that corresponds to the desired statement type.

Testing

Testing this session is rather simple. Here is a quick and dirty script to help test it.

ALTER EVENT SESSION AuditSelects ON SERVER
STATE = START;

USE AdventureWorks2014;
GO

SELECT *
	FROM Person.Person;


/* parse the data */
IF EXISTS (SELECT OBJECT_ID('tempdb.dbo.#xmlprocess'))
BEGIN
DROP TABLE #xmlprocess
END

SELECT CAST ([t2].[event_data] AS XML) AS event_data, t2.file_offset,t2.file_name, cte1.event_session_id--, '' AS event_predicate
	INTO #xmlprocess
	FROM ( SELECT REPLACE(CONVERT(NVARCHAR(128),sesf.value),'.xel','*.xel') AS targetvalue, ses.event_session_id
				FROM sys.server_event_sessions ses
					INNER JOIN sys.server_event_session_fields sesf
						ON ses.event_session_id = sesf.event_session_id
					--INNER JOIN sys.server_event_session_events sese
					--	ON ses.event_session_id = sese.event_session_id
				WHERE sesf.name = 'filename'
					AND ses.name = 'AuditSelects'
					) cte1
				OUTER APPLY sys.fn_xe_file_target_read_file(cte1.targetvalue,	NULL, NULL, NULL) t2
		;


SELECT x.event_data.value('(event/@name)[1]', 'varchar(50)') AS event_name
		, x.event_data.value('(event/@package)[1]', 'varchar(50)') AS package_name
		, DATEADD(hh, DATEDIFF(hh, GETUTCDATE(), CURRENT_TIMESTAMP),
					x.event_data.value('(event/@timestamp)[1]',
										'datetime2')) AS [timestamp]
		, event_data.value('(event/data[@name="dop"]/value)[1]',
							'bigint') AS dop
		, event_data.value('(event/data[@name="statement_type"]/value)[1]',
							'varchar(max)') AS statement_type
		, event_data.value('(event/data[@name="workspace_memory_grant_kb"]/value)[1]',
							'bigint') AS workspace_memory_grant_kb
		, event_data.value('(event/action[@name="client_app_name"]/value)[1]',
							'varchar(max)') AS client_app_name
		, event_data.value('(event/action[@name="client_connection_id"]/value)[1]',
							'uniqueidentifier') AS client_connection_id
		, event_data.value('(event/action[@name="client_hostname"]/value)[1]',
							'varchar(max)') AS client_hostname
		, event_data.value('(event/action[@name="context_info"]/value)[1]',
							'varbinary(max)') AS context_info
		, event_data.value('(event/action[@name="database_id"]/value)[1]',
							'int') AS database_id
		, event_data.value('(event/action[@name="database_name"]/value)[1]',
							'varchar(max)') AS database_name
		, event_data.value('(event/action[@name="nt_username"]/value)[1]',
							'varchar(max)') AS nt_username
		, event_data.value('(event/action[@name="session_id"]/value)[1]',
							'int') AS session_id
		, event_data.value('(event/action[@name="session_nt_username"]/value)[1]',
							'varchar(max)') AS session_nt_username
		, event_data.value('(event/action[@name="sql_text"]/value)[1]',
							'varchar(max)') AS sql_text
		, event_data.value('(event/action[@name="username"]/value)[1]',
							'varchar(max)') AS username
	FROM #xmlprocess x
		LEFT OUTER JOIN sys.server_event_session_events sese
			ON x.event_data.value('(event/@name)[1]', 'varchar(50)') = sese.name
				AND x.event_session_id = sese.event_session_id
	ORDER BY timestamp
		, event_data.value('(event/action[@name="event_sequence"]/value)[1]',
							'varchar(max)');

Conclusion

Building low cost solutions is a common requirement for the data professional. When given the opportunity, try things out to see what you can build to provide the solution. In this case, I have demonstrated how Extended Events, however unlikely a candidate, can provide a working solution to help audit any select statements occurring within your database.

This has been another article in the 60 Days of XE series. If you have missed any of the articles, or just want a refresher, check out the TOC.

Why?

There are some very good reasons as a matter of fact. One really big reason is that I could find no decent information about this issue. The bigger reason is the seemingly indifferent level of response and the delays I saw in response from the Bit9 support channels.

In this specific case, the first response from their support channels took more than two days. Subsequent responses were more than a week later. For the client, this actually caused delays in a project they were working on. Was the issue significant? It was not an outage causing issue, but it was one that did seem to continue to grow and cause concerns with disk space.

The Issue

It may be appropriate to discuss what Bit9 does prior to breaking into the details about the issue. If you have never had any experience with this product before, you may be running a product from a competitor such as Symantec. Bit9 is a an endpoint protection tool from Carbon Black. This helps to prevent against malware and endpoint attacks. In short it is a security based tool to protect your computing enterprise.

Go ahead and click the image to link to their site if you wish to learn more about how it works. For me, the nitty gritty specifics on how it works is a bit outside the scope of this article.

In short, Bit9 records a record of every filename and every filepath for every client machine. It stores these in a database in SQL Server. This shouldn’t be too much of an issue. For one instance serving about 300 client machines we saw about 1.2 million paths being stored. One another instance with far fewer clients and far fewer actual paths on those clients (physically checked) we had 114 million paths stored in the database. The difference in size was 20GB vs ~160GB. When the server is installed with all defaults, you can imagine how this looked for this instance – a very bloated OS volume.

Investigating this issue from a database perspective, I looked to find what was consuming so much space in the database. To do that, I ran my tablespace script and discovered the following:

A significant standout there with the pathnames table. Looking at the definition of the table I see something far more interesting and disturbing all at once.

I see a table with three columns and two large string fields. Each of these fields has a non-clustered index on it. This may or may not be such a big problem (other than the fact that the size of the non-clustered indexes on this table are much larger than the data) except that each of the string fields is an exact duplicate of the other. That’s right. Within this table, the data is duplicated into this second string field and each field has its own index. Not only does it appear that I have a ton of duplicated data, it appears I have entirely useless indexes (neither had been touched for a read since the server had been up)

I inquired about this to the folks at Bit9, both from a design perspective and from an archival perspective. Absolute silence on the design (expected). The inquiry about archival (or purge) of non-essential data did fetch a response – albeit a painfully slow response. The basic question is: “Is it safe to purge or archive old or unnecessary data.” The response we received was “run this and we will tell you what to do next.”

Great, they sent a script to help determine the state of data within the database. I am not posting their script here. Suffice it to say that the script they sent was not very pretty. They query about 20 tables, union the results from those tables, then perform a not in operation to see how many of the pathnames are invalid. No problem. Executing the script did reveal the following:

There seems to be the problem. 95.89% of the rows being stored in the pathnames table are orphaned records! This is a bit of a problem. The software does not appear to manage removal of invalid paths. From here, I knew what the course of action needed to be and acted on it. A big piece of the equation was provided from the results of the script. Another piece was provided in the makeup of the script. From these pieces of information, I created a purge script to help manage the orphan problem. I then put that script into an agent job and set it to run on a weekly basis.

If you find yourself in the same boat, here is the script needed to purge the data. If running this, I recommend disabling the two non-clustered indexes and then performing the delete (especially if you sit at 96% orphaned and over 100 million rows). After deleting the mass amount of orphans, go ahead and rebuild the indexes to stay in compliance with the software contract until Bit9 responds about the index requirement and the schema of the table.

-- select count(1) from [dbo].[OrphanedPathnameIds] (nolock)
CREATE TABLE #pathname ( pathname_id BIGINT );
GO

INSERT INTO #pathname
		( pathname_id
		)
		SELECT pathname_id
			FROM dbo.pathnames
			WHERE pathname_id NOT IN (
				SELECT DISTINCT pathname_id
					FROM dbo.antibodies
					WHERE pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT pathname_id
					FROM dbo.antibody_instances
					WHERE pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT pathname_id
					FROM dbo.antibody_instances_deleted
					WHERE pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT pathname_id
					FROM dbo.antibody_instance_groups
					WHERE pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT pathname_id
					FROM dbo.temp_antibody_instances
					WHERE pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT pathname_id
					FROM dbo.processing_temp_antibody_instances1
					WHERE pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT pathname_id
					FROM dbo.processing_temp_antibody_instances2
					WHERE pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT pathname_id
					FROM dbo.processing_temp_antibody_instances3
					WHERE pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT old_pathname_id
					FROM dbo.temp_antibody_instances
					WHERE old_pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT old_pathname_id
					FROM dbo.processing_temp_antibody_instances1
					WHERE old_pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT old_pathname_id
					FROM dbo.processing_temp_antibody_instances2
					WHERE old_pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT old_pathname_id
					FROM dbo.processing_temp_antibody_instances3
					WHERE old_pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT pathname_id
					FROM dbo.antibody_instances_snapshots
					WHERE pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT pathname_id
					FROM dbo.approval_requests
					WHERE pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT pathname_id
					FROM dbo.internal_events
					WHERE pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT process_pathname_id
					FROM dbo.internal_events
					WHERE process_pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT process_pathname_id
					FROM dbo.events
					WHERE process_pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT process_pathname_id
					FROM dbo.remote_events
					WHERE process_pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT process_pathname_id
					FROM dbo.remote_internal_events
					WHERE process_pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT process_pathname_id
					FROM dbo.approval_requests
					WHERE process_pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT pathname_id
					FROM dbo.events
					WHERE pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT pathname_id
					FROM dbo.remote_events
					WHERE pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT pathname_id
					FROM dbo.remote_internal_events
					WHERE pathname_id IS NOT NULL
				UNION
				SELECT DISTINCT file_path_id
					FROM dbo.notification_files
					WHERE file_path_id IS NOT NULL
				UNION
				SELECT DISTINCT proc_path_id
					FROM dbo.notification_files
					WHERE proc_path_id IS NOT NULL
				UNION
				SELECT DISTINCT proc_path_id
					FROM dbo.notification_directories
					WHERE proc_path_id IS NOT NULL
				UNION
				SELECT DISTINCT dir_path_id
					FROM dbo.notification_directories
					WHERE dir_path_id IS NOT NULL
				UNION
				SELECT DISTINCT proc_path_id
					FROM dbo.notification_regkeys
					WHERE proc_path_id IS NOT NULL
				UNION
				SELECT DISTINCT pathname_id
					FROM dbo.uploaded_files
					WHERE pathname_id IS NOT NULL );
GO

CREATE TABLE #filenames ( filename_id BIGINT );
GO
INSERT INTO #filenames
		( filename_id
		)
		SELECT filename_id
			FROM dbo.filenames
			WHERE filename_id NOT IN (
				SELECT DISTINCT filename_id
					FROM dbo.antibodies
					WHERE filename_id IS NOT NULL
				UNION
				SELECT DISTINCT filename_id
					FROM dbo.antibody_instances
					WHERE filename_id IS NOT NULL
				UNION
				SELECT DISTINCT filename_id
					FROM dbo.antibody_instances_deleted
					WHERE filename_id IS NOT NULL
				UNION
				SELECT DISTINCT filename_id
					FROM dbo.antibody_instance_groups
					WHERE filename_id IS NOT NULL
				UNION
				SELECT DISTINCT filename_id
					FROM dbo.temp_antibody_instances
					WHERE filename_id IS NOT NULL
				UNION
				SELECT DISTINCT filename_id
					FROM dbo.processing_temp_antibody_instances1
					WHERE filename_id IS NOT NULL
				UNION
				SELECT DISTINCT filename_id
					FROM dbo.processing_temp_antibody_instances2
					WHERE filename_id IS NOT NULL
				UNION
				SELECT DISTINCT filename_id
					FROM dbo.processing_temp_antibody_instances3
					WHERE filename_id IS NOT NULL
				UNION
				SELECT DISTINCT old_filename_id
					FROM dbo.temp_antibody_instances
					WHERE old_filename_id IS NOT NULL
				UNION
				SELECT DISTINCT old_filename_id
					FROM dbo.processing_temp_antibody_instances1
					WHERE old_filename_id IS NOT NULL
				UNION
				SELECT DISTINCT old_filename_id
					FROM dbo.processing_temp_antibody_instances2
					WHERE old_filename_id IS NOT NULL
				UNION
				SELECT DISTINCT old_filename_id
					FROM dbo.processing_temp_antibody_instances3
					WHERE old_filename_id IS NOT NULL
				UNION
				SELECT DISTINCT filename_id
					FROM dbo.antibody_instances_snapshots
					WHERE filename_id IS NOT NULL
				UNION
				SELECT DISTINCT filename_id
					FROM dbo.approval_requests
					WHERE filename_id IS NOT NULL
				UNION
				SELECT DISTINCT filename_id
					FROM dbo.internal_events
					WHERE filename_id IS NOT NULL
				UNION
				SELECT DISTINCT filename_id
					FROM dbo.events
					WHERE filename_id IS NOT NULL
				UNION
				SELECT DISTINCT process_filename_id
					FROM dbo.internal_events
					WHERE process_filename_id IS NOT NULL
				UNION
				SELECT DISTINCT process_filename_id
					FROM dbo.events
					WHERE process_filename_id IS NOT NULL
				UNION
				SELECT DISTINCT filename_id
					FROM dbo.remote_internal_events
					WHERE filename_id IS NOT NULL
				UNION
				SELECT DISTINCT filename_id
					FROM dbo.remote_events
					WHERE filename_id IS NOT NULL
				UNION
				SELECT DISTINCT process_filename_id
					FROM dbo.remote_internal_events
					WHERE process_filename_id IS NOT NULL
				UNION
				SELECT DISTINCT process_filename_id
					FROM dbo.remote_events
					WHERE process_filename_id IS NOT NULL
				UNION
				SELECT DISTINCT process_filename_id
					FROM dbo.approval_requests
					WHERE process_filename_id IS NOT NULL
				UNION
				SELECT DISTINCT file_name_id
					FROM dbo.notification_files
					WHERE file_name_id IS NOT NULL
				UNION
				SELECT DISTINCT proc_name_id
					FROM dbo.notification_files
					WHERE proc_name_id IS NOT NULL
				UNION
				SELECT DISTINCT proc_name_id
					FROM dbo.notification_directories
					WHERE proc_name_id IS NOT NULL
				UNION
				SELECT DISTINCT proc_name_id
					FROM dbo.notification_regkeys
					WHERE proc_name_id IS NOT NULL
				UNION
				SELECT DISTINCT filename_id
					FROM dbo.uploaded_files
					WHERE filename_id IS NOT NULL );

GO


--select *
DELETE fn
	FROM #filenames f
		INNER JOIN dbo.filenames fn
			ON f.filename_id = fn.filename_id;


SELECT TOP 5000000 p.pathname_id
	INTO #presel
	FROM #pathname p;

--select *
DELETE pn
	FROM #presel p
		INNER JOIN pathnames pn
			ON p.pathname_id = pn.pathname_id;

DELETE pn
	FROM #presel p
		INNER JOIN #pathname pn
			ON p.pathname_id = pn.pathname_id;

DROP TABLE #presel;
DROP TABLE #pathname;
DROP TABLE #filenames;

Yep! It is just like that. The worst part of the whole thing is actually the delete. The delete operation will touch about 37 indexed views to update and delete. Be prepared for a long running process if you have a situation like I encountered.

For kicks, here is what that plan would look like in one of my favorite tools (SQL Sentry Plan Explorer).

What is your pain scale and story?

Last week I sent out the invite for the August TSQL Tuesday blog party. In that invite I promised to send out a reminder seven days prior to the event. Well – it is that time.

You are cordially invited to read the invite for TSQL Tuesday 81 and plan your entry for the party.

In the invite, I shared the details for the event including examples of what I am looking for as an entry for the event.

I hope we will be seeing you next Tuesday, August 9th in attendance at this month’s party. I am sure it will prove to be an interesting experience one way or another.

Bonus Example

In the original invite I provided a list of examples of what one could do for this TSQL Tuesday. Today, I am providing one more example in a slightly different format. Recall that the invite requested that participants set out to accomplish something, make a plan and report on that “goal”, the plan, and the outcome.

So, let’s say I have discovered that I write too much in the passive voice. Based on that information, I would like to overcome the passivity in my writing voice, therefore my goal would be to learn how to write more assertively (less passively). In order to accomplish that goal, I may need to read up on the topic and learn exactly what it means to write passively. Then I would need to examine articles that I have written. And then I would need to practice writing more assertively. After all of that is done, I may have somebody (or something) analyze a brand new article or two to determine if I have achieved my desire.

After having executed on that plan, I will write about the experience including what the initial goal and plan were and also on what worked or didn’t work while trying to reach that goal. To summarize, here is an outline of that example:

What I will Accomplish

I will learn how to write more assertively (or just Write more assertively)

How Will I do that

Research what it means to write passively

Research what it means to write assertively

Evaluate “assertively” written articles

Take Notes on how to write assertively

Evaluate my articles

Practice writing assertively

Write a new article and have it reviewed to judge the voice whether it seems too passive or not

Report on The Successes and Failures

Write whether or not each step succeeded or failed.

Write if a step was unnecessary

Write about the experience and your thoughts about the experience.

Did you achieve or fail overall?

What is T-SQL Tuesday?

T-SQL Tuesday is a monthly blog party hosted by a different blogger each month. This blog party was started by Adam Machanic (blog|twitter). You can take part by posting your own participating post that fits the topic of the month and follows the requirements below. Additionally, if you are interested in hosting a future T-SQL Tuesday, contact Adam Machanic on his blog.

How to Participate

• Your post must be published between 00:00 GMT Tuesday, Août 9e, 2016, and 00:00 GMT Wednesday Août 10e, 2016.
• Your post must contain the T-SQL Tuesday logo from above and the image should link back to this blog post.
• Trackbacks should work. But, please do add a link to your post in the comments section below so everyone can see your work.
• Tweet about your post using the hash tag #TSQL2sDay.

Among the batch of articles preceding the 60 day series one can find an article about shredding extended event data. Then as a part of the series (the first article in the 60 day series) one can find how to shred the action payload data attached to an extended event session.

You can read the article on shredding XEvent payload data here. Then you can follow that up by reading how to shred the Action data here. And once you are done with those and you really want to learn more about Extended Events, please read the whole series which can be found here.

All of that reading to catch up should keep you busy for a tad bit.

Upgrade

A year is a really long time to go without updating a script – according to some. Today, I have an update for both of the XML shredding scripts you just read about in the list of articles mentioned moments earlier in this post.

Before I dive into the script, let’s revisit a sample of the XML from an event session payload. Within an event session, you can see that there are nodes for both the data and the actions (highlighted in green and red). With all of this information found within the same XML for the session, it somewhat makes sense to try and parse all of the data at once.

In addition to parsing all of the XML for the data and the actions at the same time, it seems to also make sense to generate the statements that would parse the XML within a single effort. As you would have noted, my previous scripts were just that – scripts. That implies executing a separate script for each the data and the actions. Maybe it would make more sense to execute a single script.

It is that notion of a single script that constitutes this upgraded version of the script.

One Script

In the following script, I have tried to accomplish just that – a single script to create the entire XML parser for me, for you, and for anybody wishing to use it. I don’t want to have to remember the subtle nuances of how to parse each of the events each time I need to parse the session data. I want something that is quick, easy, and repeatable.

With all of that said, here is the script that I now use to parse my session data. You should notice that it has been simplified and is more extensive now.

DECLARE @EventName VARCHAR(64) = NULL --'query_post_execution_showplan'--'sp_statement_completed'
	,@ReadFlag VARCHAR(64) = 'readonly' --NULL if all columntypes are desired
	,@SessionName	VARCHAR(128) = 'AuditDeprecated' --'system_health' --NULL if all Sessions are desired --'Deadlock' --NULL
	,@UniqueEventColumns	TINYINT = 1 --1 for unique payload columns across all events within the session, NULL for all columns in session
	,@UniqueActions TINYINT = 1 --1 for unique actions, NULL for all actions in the session

/*gen tsql to parse session payload */
SELECT '/* parse the data */
IF EXISTS (SELECT OBJECT_ID(''tempdb.dbo.#xmlprocess''))
BEGIN
DROP TABLE #xmlprocess
END

SELECT CAST ([t2].[event_data] AS XML) AS event_data, t2.file_offset,t2.file_name, cte1.event_session_id--, '' AS event_predicate
	INTO #xmlprocess
	FROM ( SELECT REPLACE(CONVERT(NVARCHAR(128),sesf.value),''.xel'',''*.xel'') AS targetvalue, ses.event_session_id
				FROM sys.server_event_sessions ses
					INNER JOIN sys.server_event_session_fields sesf
						ON ses.event_session_id = sesf.event_session_id
					--INNER JOIN sys.server_event_session_events sese
					--	ON ses.event_session_id = sese.event_session_id
				WHERE sesf.name = ''filename''
					AND ses.name = ''' + @SessionName +'''
					) cte1
				OUTER APPLY sys.fn_xe_file_target_read_file(cte1.targetvalue,	NULL, NULL, NULL) t2
		;


SELECT x.event_data.value(''(event/@name)[1]'', ''varchar(50)'') AS event_name
		, x.event_data.value(''(event/@package)[1]'', ''varchar(50)'') AS package_name
		, DATEADD(hh, DATEDIFF(hh, GETUTCDATE(), CURRENT_TIMESTAMP),
					x.event_data.value(''(event/@timestamp)[1]'',
										''datetime2'')) AS [timestamp]';

WITH presel AS (
SELECT oc.OBJECT_NAME AS EventName
		,oc.name AS column_name, oc.type_name
		,',event_data.value(''(event/data[@name="' + oc.name + '"]/value)[1]'',''' + 
			CASE 
				WHEN ISNULL(xmv.name,'') = ''
					AND oc.type_name = 'guid'
				THEN 'uniqueidentifier'
				WHEN ISNULL(xmv.name,'') = ''
					AND oc.type_name = 'boolean'
				THEN 'bit'
				WHEN ISNULL(xmv.name,'') = ''
					AND oc.type_name = 'callstack'
				THEN 'varchar(max)'
				WHEN ISNULL(xmv.name,'') = ''
					AND oc.type_name <> 'unicode_string'
					AND oc.type_name <> 'ansi_string'
					AND oc.type_name <> 'ptr'
					AND oc.type_name NOT LIKE '%int%'
				THEN oc.type_name
				WHEN ISNULL(xmv.name,'') = ''
					AND oc.type_name LIKE '%int%'
				THEN 'bigint'
				ELSE 'varchar(max)' END + ''') AS ' + oc.name + '' AS ColumnXML
		,oc.column_type AS column_type
		,oc.column_value AS column_value
		,oc.description AS column_description
		,ca.map_value AS SearchKeyword
		,sesea.SessionName
		,rownum = ROW_NUMBER() OVER (PARTITION BY sesea.SessionName, oc.name ORDER BY oc.object_name)
	FROM sys.dm_xe_object_columns oc
	-- do we have any custom data types
		OUTER APPLY (SELECT DISTINCT mv.name FROM sys.dm_xe_map_values mv
			WHERE mv.name = oc.type_name
			AND mv.object_package_guid = oc.object_package_guid) xmv
	--just get the unique events that are tied to a session on the server (stopped or started state)
		CROSS APPLY (SELECT DISTINCT sese.name,ses.name AS SessionName
						FROM sys.server_event_session_events sese
							INNER JOIN sys.server_event_sessions ses
								ON sese.event_session_id = ses.event_session_id) sesea
	--keyword search phrase tied to the event
		CROSS APPLY (SELECT TOP 1 mv.map_value
						FROM sys.dm_xe_object_columns occ
						INNER JOIN sys.dm_xe_map_values mv
							ON occ.type_name = mv.name
							AND occ.column_value = mv.map_key
						WHERE occ.name = 'KEYWORD'
							AND occ.object_name = oc.object_name) ca
	WHERE oc.column_type <> @ReadFlag
		AND sesea.name = oc.object_name
		AND oc.object_name = ISNULL(@EventName,oc.object_name)
		AND sesea.SessionName = ISNULL(@SessionName,sesea.SessionName)
)

SELECT p.EventName
	 , p.column_name
	 , p.type_name
	 , p.ColumnXML
	 , p.column_type
	 , p.column_value
	 , p.column_description
	 , p.SearchKeyword
	 , p.SessionName
	FROM presel p
	WHERE p.rownum = ISNULL(@UniqueEventColumns,p.rownum)
	ORDER BY p.SessionName,p.EventName
	;

/* auto generate the xml associated to an event session action deployed to the server */
WITH presel AS (
	SELECT p.name AS package_name
	        ,o.name AS action_name
			,',event_data.value(''(event/action[@name="' + esa.name + '"]/value)[1]'', ''' + 
			CASE 
				WHEN o.type_name = 'guid'
				THEN 'uniqueidentifier'
				WHEN o.type_name = 'boolean'
				THEN 'bit'
				WHEN o.type_name = 'binary_data'
				THEN 'varbinary(max)'
				WHEN o.type_name = 'callstack'
				THEN 'varbinary(max)'
				WHEN o.type_name = 'filetime'
				THEN 'varbinary(max)'
				WHEN o.type_name = 'cpu_cycle'
				THEN 'varbinary(max)'
				WHEN ISNULL(o.type_name,'') = ''
				THEN NULL
				WHEN o.type_name <> 'unicode_string'
					AND o.type_name <> 'ansi_string'
					AND o.type_name <> 'ptr'
					AND o.type_name NOT LIKE '%int%'
				THEN o.type_name
				WHEN o.type_name LIKE '%int%'
				THEN 'int'
				ELSE 'varchar(max)' END + ''') AS ' + esa.name +'' AS ActionXML
			,ses.name AS EventSessionName
			, ese.name AS EventName
	        ,o.description
			, rownum = ROW_NUMBER() OVER (PARTITION BY ses.name,o.name ORDER BY ese.name)
	FROM sys.dm_xe_packages AS p
		INNER JOIN sys.dm_xe_objects AS o
			ON p.guid = o.package_guid
		INNER JOIN sys.server_event_session_actions esa
			ON o.name = esa.name
		INNER JOIN sys.server_event_sessions ses
			ON esa.event_session_id = ses.event_session_id
		INNER JOIN sys.server_event_session_events ese
			ON esa.event_session_id = ese.event_session_id
			AND ese.event_id = esa.event_id
	WHERE o.object_type = 'action'
		AND (o.capabilities IS NULL OR o.capabilities & 1 = 0)
		AND (p.capabilities IS NULL OR p.capabilities & 1 = 0)
		AND ese.name = ISNULL(@EventName,ese.name)
		AND ses.name = ISNULL(@SessionName,ses.name)
)
SELECT p.package_name
	 , p.action_name
	 , p.ActionXML
	 , p.EventSessionName
	 , p.EventName
	 , p.description
	FROM presel p
	WHERE p.rownum = ISNULL(@UniqueActions,p.rownum)
	ORDER BY p.EventSessionName, p.action_name, p.EventName
	;

/* gen last piece of xml parsing sql code block*/
SELECT '	FROM #xmlprocess x
		LEFT OUTER JOIN sys.server_event_session_events sese
			ON x.event_data.value(''(event/@name)[1]'', ''varchar(50)'') = sese.name
				AND x.event_session_id = sese.event_session_id
	ORDER BY timestamp
		, event_data.value(''(event/action[@name="event_sequence"]/value)[1]'',
							''varchar(max)'');'

Not only does this script slice and dice for you…Wait that’s not quite right.

This script doesn’t just generate the XML shredding strings for the payload and action data. This script also will produce the rest of the tsql statements that should precede and conclude the XML parsing statements. Yes it uses dynamic sql. That is to your advantage in this case.

Could there be other enhancements? Absolutely! More will be coming.

Enjoy this script and happy parsing.

Extra Extra – Read All About It!

From the comments, you will see that Brent Ozar (blog | twitter) made an excellent suggestion. To be honest, I considered sharing some examples during the initial write of this article. I had opted against it then, not considering it in the same way that Brent puts it. So, in this addendum I have a quick example using an XEvent Session that I have not yet written about in all of my articles on Extended Events (pseudo spoiler alert).

Let’s use the AuditSelects XEvent Session I have deployed and running (remember a session can be deployed but may be in the “stopped” state). If I want to figure out what data has been captured for this session, I can take the session name and plug it into the script shown in this article.

Ignore the green tsql comments for now (you can read them direct from the script). Highlighted in gold is the variable for the session name. By entering a valid session name for this variable, I can restrict the results to just the metadata for that specific session. In addition, note that I have highlighted, in a pale chartreuse, two variables that will give me unique column names for each of the events and actions within each session. This is important because some events have the same name for different attributes as other events. If events with the same attribute names are used in the same session, you will get a duplication of data. Maybe you want the duplicate data. Maybe you don’t.

The session name variable is used in each of the next three code segments. Each segment in the script will generate sql statements that will need to be copied into a new query window. I will leave the further breakdown of the script as an exercise for you. What I want to do here is show the results from executing this script.

When I execute the script, I will have four result sets returned to me (in grid mode). It will look something like shown in the following image.

Breaking this down into three colors is probably easiest to see how things fit together. Highlighted in red, I have the build out of the pre-processing statements and the build of the “select” statement that will be returning our data to us. Then in blue comes the guts of the query – all of the beautiful XML parsing statements. The segments in blue correlates to the columns in the select list. And at the bottom of the image I have the finishing pieces that includes my predicates, from, and joins. Each of the highlighted segments will then be copied and pasted to a new query window and look something like the following.

And there you have it. Sure there are still some manual steps in it, but you no longer need to memorize all of that glorious XML parsing syntax. You only need to copy and paste with this version of the script.

Enjoy your adventures in parsing XML!

It has now been 30 months since the last time I hosted a TSQL Tuesday, that was TSQL Tuesday 51. I recapped that event here with the original invite here. I can’t believe it has been that long since I last hosted. It only seems like yesterday.

Coming into the present day, we are now at TSQL Tuesday 81. For this month, I would like to try and up the ante a bit. Usually we only get about a weeks notice prior to the event to think about the article to write for the event.

This time, I want to invite everybody just a little bit sooner and will follow-up with a reminder seven days prior to the event. The reason I want to do this is because I think this may be a touch more difficult this time.

This month I am asking you to not only write a post but to do a little homework – first. In other words, plan to do something, carry out that plan, and then write about the experience. There is a lot going into that last sentence. Because of that, let me try to explain through a few examples of what I might like to see. Hopefully these examples will help you understand the intent and how this month the topic relates to “Sharpening Something“.

EXAMPLES

1. You have learned about a really cool feature called Azure DevTest Lab. Having heard about it, you wish to implement this feature to solve some need in your personal development or corporate environment. Develop a plan to implement the feature and tell us the problem it solves and about your experiences in getting it to work from start to end. An example of how I might try to use this might involve the creation of a disposable and easy setup environment for Precons, Workshops, and various other types of training.
2. There is a really awesome book about SQL Server you heard about and you decided to buy it. Plan to sit down and read the book. Take a nugget or two from the book and tell us how you can use that nugget of information within your personal or professional environment.
3. You know you are extremely deficient at a certain SQL Skill. Tell me what that skill is and develop a plan to get better at that skill. Report on the implementation of this skill and how you are doing at improving. Maybe that skill is about Extended Events, PoSH or availability groups.
4. Similar to the skill deficiency, you know you do not understand a certain concept within SQL Server as well as you feel you should. Maybe that concept is indexing or statistics (for example). Create a two week plan to become more proficient at that concept. Follow that plan and report on your progress.

In recap, this is an invite to make a short term goal covering the next two weeks. Tell everybody what that goal is (in your tsql tuesday post of course) and how you went about creating a plan for that goal and how you have progressed during the two week interval.

What is T-SQL Tuesday?

T-SQL Tuesday is a monthly blog party hosted by a different blogger each month. This blog party was started by Adam Machanic (blog|twitter). You can take part by posting your own participating post that fits the topic of the month and follows the requirements below. Additionally, if you are interested in hosting a future T-SQL Tuesday, contact Adam Machanic on his blog.

How to Participate

• Your post must be published between 00:00 GMT Tuesday, Août 9e, 2016, and 00:00 GMT Wednesday Août 10e, 2016.
• Your post must contain the T-SQL Tuesday logo from above and the image should link back to this blog post.
• Trackbacks should work. But, please do add a link to your post in the comments section below so everyone can see your work.
• Tweet about your post using the hash tag #TSQL2sDay.

Today I will be sharing a new query or two that I have used on more than one occasion to help track down the event(s) that I wanted to, at least, try while troubleshooting specific problems within the SQL Server Instance. I found these queries useful once again in a recent opportunity to help out some friends from the SQL Community. (Maybe, just maybe, these are really just a spin of other previously shared queries.) I will share the most recent experience in another follow-up post.

For the sake of posterity, I am also adding this to the 60 Days of Extended Events Table of Contents.

Is There an Event that Contains pertinent Data?

In my previous article I demonstrated how to find an event based solely on the name or description of the event. This is fantastic if the event name (or description) contains one of the magical words you have used. What if the event name or description has nothing to do with the terms you selected? Or, what if the data you seek may be attached to the event but wouldn’t necessarily stand out as a description for that event (by name or description details for that event)?

Now comes the more difficult task right? If the name or description of the event doesn’t relate to the search terms then you just might overlook a few events and be stuck trying to troubleshoot a problem. An equally big problem this could cause is yet another invisible barrier to using Extended Events. It would be easy to slide down the slippery slope and not transition to Extended Events just because an event, applicable to the problem at hand, could not be found.

This is where searching on the payload comes in handy. Remember from a previous article that the payload is the event data that is attached to each event. Imagine how much more accurate we can become by extending our search from the name and description of the event object to next search on the payload names and descriptions? That is exactly what I will now show.

DECLARE @TermDescription VARCHAR(64) = 'Select'
	, @ColumnDesc VARCHAR(64) = 'statement'
	, @ReadFlag VARCHAR(64) = NULL; --readonly' --ALL if all columntypes are desired --data --customizable
 
SELECT oc.object_name AS EventName
		, oc.name AS column_name
		, oc.type_name
		, oc.column_type AS column_type
		, oc.column_value AS column_value
		, oc.description AS column_description
		, ca.map_value AS SearchKeyword
		, oc.column_type
	FROM sys.dm_xe_object_columns oc
		CROSS APPLY ( SELECT mv.map_value
							FROM sys.dm_xe_object_columns occ
								INNER JOIN sys.dm_xe_map_values mv
									ON occ.type_name = mv.name
										AND occ.column_value = mv.map_key
										AND occ.object_package_guid = mv.object_package_guid
							WHERE occ.name = 'KEYWORD'
								AND occ.object_name = oc.object_name
					) ca
	WHERE (oc.object_name = @TermDescription
		OR oc.description LIKE '%' + ISNULL(@ColumnDesc,@TermDescription) + '%')
		AND oc.column_type = ISNULL(@ReadFlag, oc.column_type)
	ORDER BY EventName, column_name;

GO

Basically, I am searching against the object_columns metadata dmv for Extended Events just to find any columns that match my theoretical needs. In this case, I am trying to find anything that may help me find events related to the terms “select” or “statement”. When run, there will be quite a few results for these terms. That may actually be a bit of a problem (just the same as if nothing were returned in the results). There is a fix for that (well probably several fixes as you begin to tweak the query).

One possible fix is to also include map_values to refine the search. Another fix might be to change the search terms. And still other fixes might include tweaking the predicate. For today, I am going to demonstrate the inclusion of map_values as a secondary tweak to help isolate the needed event.

DECLARE @TermDescription VARCHAR(64) = 'Select'
	, @ColumnDesc VARCHAR(64) = 'statement'
	, @ReadFlag VARCHAR(64) = NULL; --readonly' --ALL if all columntypes are desired --data --customizable
 
SELECT oc.object_name AS EventName
		, oc.name AS column_name
		, oc.type_name
		, oc.column_type AS column_type
		, oc.column_value AS column_value
		, oc.description AS column_description
		, ca.map_value AS SearchKeyword
		, mv.map_value AS MapValue
		, mv.map_key AS MapKey
		, oc.column_type
	FROM sys.dm_xe_object_columns oc
		INNER JOIN sys.dm_xe_map_values mv
			ON oc.type_name = mv.name
			AND mv.object_package_guid = oc.object_package_guid
		CROSS APPLY ( SELECT mv.map_value
							FROM sys.dm_xe_object_columns occ
								INNER JOIN sys.dm_xe_map_values mv
									ON occ.type_name = mv.name
										AND occ.column_value = mv.map_key
										AND occ.object_package_guid = mv.object_package_guid
							WHERE occ.name = 'KEYWORD'
								AND occ.object_name = oc.object_name
					) ca
	WHERE (oc.object_name = @TermDescription
		OR oc.description LIKE '%' + ISNULL(@ColumnDesc,@TermDescription) + '%')
		AND mv.map_value = @TermDescription
		AND oc.column_type = ISNULL(@ReadFlag, oc.column_type);

In this iteration, my result-set is trimmed significantly due to the predicate and the requirement for a match based on the map_value name. Ultimately, this trimmed my results down to precisely the events that I needed in this case. With this slightly more advanced version, I have greater visibility into the Extended Event realm to help me find just the right event to match my needs.

Are there other Events related to the new found Event?

This is some pretty awesome stuff so far. But it doesn’t end right there. Many events are tied together to help provide a more holistic view of the problem. These events are tied together via keyword. You can read more about keywords here. By searching for events based on a keyword, we can get a lot more intelligent about the sessions we create with Extended Events.

Taking the single result from the last query and then using the SearchKeyword value, for that event, in this next query – I can potentially go from a Great DBA to a Super DBA.

DECLARE @Keyword VARCHAR(64) = 'execution' --'synchronization'
 
SELECT oc.OBJECT_NAME AS EventName
		,oc.name AS column_name, oc.type_name
		,oc.column_type AS column_type
		,oc.column_value AS column_value
		,oc.description AS column_description
		,REVERSE(LEFT(REVERSE(olm.name),CHARINDEX('\',REVERSE(olm.name))-1)) AS DLLName
		,olm.file_version
		,xp.name AS PackageName
		,mv.map_value AS SearchKeyword
		,ch.Channel
	FROM sys.dm_xe_object_columns oc
		INNER JOIN sys.dm_xe_map_values mv
			ON oc.type_name = mv.name
			AND oc.column_value = mv.map_key
			AND oc.name = 'KEYWORD'
		INNER JOIN sys.dm_xe_packages xp
			ON oc.object_package_guid = xp.guid
		INNER JOIN sys.dm_os_loaded_modules olm
			ON xp.module_address = olm.base_address
		INNER JOIN (SELECT c.object_name AS EventName,c.object_package_guid AS PkgGuid, v.map_value AS Channel
					FROM sys.dm_xe_object_columns c
						INNER JOIN sys.dm_xe_map_values v
							ON c.type_name = v.name
								AND c.column_value = CAST(v.map_key AS NVARCHAR)
					WHERE c.name = 'channel') ch
			ON ch.EventName = oc.object_name
			AND ch.PkgGuid = oc.object_package_guid
	WHERE mv.map_value = @Keyword
	ORDER BY EventName
	;
GO

I intentionally included the columns from each of the matching events. Doing this, I can also see what potential payload may be trapped when I attempt to trap any of the events from this result set.

Conclusion

There is great power in Extended Events. There is more power in being able to query the metadata for Extended Events in order to trap the appropriate information to properly troubleshoot a problem.

Have fun with these scripts and I hope you enjoy.

Sometimes, this information is easy enough to find.  Sometimes, it is less than evident how to find the wanted information.  Some of the information may be visible through the GUI (if you are using SQL Server 2012 or later), and sometimes it is just flat out easier to run a quick script.  Today, I would like to share a few quick scripts to help find some of this information that can make the life of the DBA just a bit easier.

Does an Event Exist?

First let’s tackle the problem of discovery.  When we want to use extended events to try and troubleshoot a problem or to capture more information, it is really good to know if such an event exists.  There are many events that capture data for various different things within SQL Server.  More and more events are being added with each release.  More and more data is being made available to the DBA to help perform a better job and to help the DBA better understand what is really happening within the database environment.

In order to determine if there might be an event, that can provide the data for that one “thing” that may be happening within your environment, we could start by querying the SQL Server Internals.  This next query will do just that for us.

SELECT c.object_name AS EventName
		,p.name AS PackageName
		,o.description AS EventDescription
	FROM sys.dm_xe_objects o
		INNER JOIN sys.dm_xe_object_columns c 
			ON o.name = c.object_name
			and o.package_guid = c.object_package_guid
		INNER JOIN sys.dm_xe_packages p
			ON o.package_guid = p.guid
	WHERE object_type='event'
		AND c.name = 'channel'
		AND (c.object_name like '%file_size%'
			or c.object_name like '%growth%')
	ORDER BY o.package_guid;

This will query the DMVs related to Extended Events and provide us with the event names that contain the terms used in the filter / predicate.  In this case, I am looking for any events that contain the terms “growth” or “file_size”.  This is in an effort to try and find anything that might be related to database files changing sizes.  You may have seen one of the previous articles on tracking file size changes here or here.

The use of the right term to try and trap the right data for the problem that is happening can greatly decrease the time required to find the correct event.  There may be a bit of an art involved in finding the correct search term.  A little practice can help improve the ability to find the appropriate event more quickly.

Is the Data any Good?

Knowing if an Extended Event exists for the desired problem is one thing.  Knowing what data can be parsed from that event is another thing.  Many times, we may not know what kind of data can be trapped with each of the events.  Many times, we may just not be trapping enough data.  And of course, knowing what kind of data can be trapped by the event may help us in filtering that data down to what is usable for efficient troubleshooting.

Finding that data is just a matter of another simple query against the DMVs.  Let’s take a look at the query that can help us find the data we need:

DECLARE @EventName VARCHAR(64) = 'trace_flag_changed'
	,@ReadFlag VARCHAR(64) = 'readonly' --NULL if all columntypes are desired

SELECT oc.object_name as EventName
		,oc.name AS column_name, oc.type_name,
		oc.column_type AS column_type,
		oc.column_value AS column_value,
		oc.description AS column_description
	FROM sys.dm_xe_objects AS o
		INNER JOIN sys.dm_xe_object_columns AS oc
			ON o.name = oc.object_name
			AND o.package_guid = oc.object_package_guid
	WHERE o.name = @EventName
		AND (oc.capabilities IS NULL 
				OR oc.capabilities & 1 = 0)
		AND oc.column_type <> @ReadFlag
	   ;
GO

In this case, I have taken the name of the event about which I want to learn more details.  In addition, I only want to retrieve the data columns that retrieve data that could change.  As you can see, I am investigating “trace_flag_changed”.  You may recall that particular event from a previous article and you can read more about it here.

You may also be looking at the query and thinking it seems a little overly complex for what we want.  Well, it is.  Here is an alternate version:

DECLARE @EventName VARCHAR(64) = 'trace_flag_changed'
	,@ReadFlag VARCHAR(64) = 'readonly' --NULL if all columntypes are desired

SELECT oc.object_name as EventName
		,oc.name AS column_name, oc.type_name
		,oc.column_type AS column_type
		,oc.column_value AS column_value
		,oc.description AS column_description 
	FROM sys.dm_xe_object_columns oc
	WHERE oc.object_name = @EventName
		AND oc.column_type <> @ReadFlag
	;

The discovery of the columns available in each event will give us some options to filter as well as some options in what kind of data we parse from the resulting event.

Did I Already Create that Session?

This is arguably less critical than the others.  After all, some will just drop and recreate the session.  But why drop the session if you don’t need to?  On the other hand, a quick glimpse in the GUI could also display that information.  Sadly that doesn’t work for all versions of SQL Server (e.g. SQL Server 2008).  Maybe you have the session running, or maybe it is simply created and not in the running state.

Finding these sessions and whether they are running or not is rather simple.  Just run this next query and it will help you figure out if the session is there or not.  All you need to do is plug in the name of the session and away you go.

DECLARE @SessionName VARCHAR(64) = 'TrackDBFileChange';

SELECT sn.SessionName
	, CASE WHEN ISNULL(es.name,'No') = 'No'
		THEN 'NO'
		ELSE 'YES'
		End as XESessionExists
	, CASE WHEN ISNULL(xe.name,'No') = 'No'
		THEN 'NO'
		ELSE 'YES'
		End as XESessionRunning
	FROM (Select @SessionName as SessionName) sn
		LEFT OUTER JOIN sys.server_event_sessions es
			ON sn.SessionName = es.name
		LEFT OUTER JOIN sys.dm_xe_sessions xe
			ON es.name = xe.name
	;

This query has been confirmed to work in SQL 2008 and up through SQL 2014.  It should simplify some of the attempts to better understand what is happening with any sessions that might be out there.

These are a few quick and simple queries to help you along your way to becoming better acquainted with Extended Events.  I hope these will be useful to you and that you will get to know and use extended events regularly.

Update: This post is being added to the 60 day series which can be found here. While in here fixing a formatting issue that was discovered, it made sense to also include it in the ongoing library of articles on Extended Events.

In a very similar fashion, today I want to share how easy it is to turn the internal schedules for SQL Agent jobs into something that is more human friendly. SQL Agent job schedules should be easy to read for humans if for nothing else than the fact that you have to provide this information to the auditors from time to time. We all know they can’t properly interpret the numeric job schedules and properly determine what they mean.

The job schedules work quite well with the current design – which is not human friendly. The use of numbers is a streamlined approach and far more efficient. It may be a bit more useful to us if there were some sort of translation table that was stored. A stored translation table would make it terribly easy to figure out what is happening within the schedules for the SQL Agent jobs. We don’t have that, so we often have to do a little trial and error. Or we rely on people that have done that trial and error – with a little verification on our part. Afterall, these scripts are posted on the internet.

SQL Agent Job Schedules

I am not going to break down the script. With the similarity to the aforementioned SSRS script, I would recommend looking there for a little bit of an explanation. Here is the script to retrieve SQL Agent Job Schedules from your MSDB instance.

USE msdb;
GO
DECLARE @weekDay TABLE
	(
		mask INT
		, maskValue VARCHAR(32)
	);

INSERT INTO @weekDay
		SELECT 1
				, 'Sunday'
		UNION ALL
		SELECT 2
				, 'Monday'
		UNION ALL
		SELECT 4
				, 'Tuesday'
		UNION ALL
		SELECT 8
				, 'Wednesday'
		UNION ALL
		SELECT 16
				, 'Thursday'
		UNION ALL
		SELECT 32
				, 'Friday'
		UNION ALL
		SELECT 64
				, 'Saturday';

WITH	myCTE
			AS ( SELECT sched.name AS 'scheduleName'
							, sched.enabled AS SchedEnabled
							, sched.schedule_id
							, jobsched.job_id
							, CASE	WHEN sched.freq_type = 1 THEN 'Once'
									WHEN sched.freq_type = 4
											AND sched.freq_interval = 1
									THEN 'Daily'
									WHEN sched.freq_type = 4
									THEN 'Every '
											+ CAST(sched.freq_interval AS VARCHAR(5))
											+ ' days'
									WHEN sched.freq_type = 8
									THEN REPLACE(REPLACE(REPLACE(( SELECT maskValue
																FROM @weekDay AS x
																WHERE sched.freq_interval
																& x.mask <> 0
																ORDER BY mask
																FOR
																XML RAW
																),
																'"/><row maskValue="',
																', '),
															'<row maskValue="',
															''), '"/>', '')
											+ CASE	WHEN sched.freq_recurrence_factor <> 0
															AND sched.freq_recurrence_factor = 1
													THEN '; weekly'
													WHEN sched.freq_recurrence_factor <> 0
													THEN '; every '
															+ CAST(sched.freq_recurrence_factor AS VARCHAR(10))
															+ ' weeks'
												END
									WHEN sched.freq_type = 16
									THEN 'On day '
											+ CAST(sched.freq_interval AS VARCHAR(10))
											+ ' of every '
											+ CAST(sched.freq_recurrence_factor AS VARCHAR(10))
											+ ' months'
									WHEN sched.freq_type = 32
									THEN CASE	WHEN sched.freq_relative_interval = 1
												THEN 'First'
												WHEN sched.freq_relative_interval = 2
												THEN 'Second'
												WHEN sched.freq_relative_interval = 4
												THEN 'Third'
												WHEN sched.freq_relative_interval = 8
												THEN 'Fourth'
												WHEN sched.freq_relative_interval = 16
												THEN 'Last'
											END
											+ CASE	WHEN sched.freq_interval = 1
													THEN ' Sunday'
													WHEN sched.freq_interval = 2
													THEN ' Monday'
													WHEN sched.freq_interval = 3
													THEN ' Tuesday'
													WHEN sched.freq_interval = 4
													THEN ' Wednesday'
													WHEN sched.freq_interval = 5
													THEN ' Thursday'
													WHEN sched.freq_interval = 6
													THEN ' Friday'
													WHEN sched.freq_interval = 7
													THEN ' Saturday'
													WHEN sched.freq_interval = 8
													THEN ' Day'
													WHEN sched.freq_interval = 9
													THEN ' Weekday'
													WHEN sched.freq_interval = 10
													THEN ' Weekend'
												END
											+ CASE	WHEN sched.freq_recurrence_factor <> 0
															AND sched.freq_recurrence_factor = 1
													THEN '; monthly'
													WHEN sched.freq_recurrence_factor <> 0
													THEN '; every '
															+ CAST(sched.freq_recurrence_factor AS VARCHAR(10))
															+ ' months'
												END
									WHEN sched.freq_type = 64 THEN 'StartUp'
									WHEN sched.freq_type = 128 THEN 'Idle'
								END AS Frequency
							, ISNULL('Every '
										+ CAST(sched.freq_subday_interval AS VARCHAR(10))
										+ CASE	WHEN sched.freq_subday_type = 2
												THEN ' seconds'
												WHEN sched.freq_subday_type = 4
												THEN ' minutes'
												WHEN sched.freq_subday_type = 8
												THEN ' hours'
											END, 'Once') AS SubFrequency
							, REPLICATE('0', 6 - LEN(sched.active_start_time))
							+ CAST(sched.active_start_time AS VARCHAR(6)) AS start_time
							, REPLICATE('0', 6 - LEN(sched.active_end_time))
							+ CAST(sched.active_end_time AS VARCHAR(6)) AS end_time
							, REPLICATE('0', 6 - LEN(jobsched.next_run_time))
							+ CAST(jobsched.next_run_time AS VARCHAR(6)) AS next_run_time
							, CAST(jobsched.next_run_date AS CHAR(8)) AS next_run_date
						FROM msdb.dbo.sysschedules AS sched
							INNER JOIN msdb.dbo.sysjobschedules AS jobsched
								ON sched.schedule_id = jobsched.schedule_id
						WHERE sched.enabled = 1
				)
	SELECT j.name AS JobName
			, j.enabled
			, j.category_id
			, sp.name AS JobOwner
			, c.name
			, c.category_class
			, js.step_id
			, js.step_name
			, js.subsystem
			, js.command
			, js.database_name
			, js.database_user_name
			, ct.next_run_date
			, ct.next_run_time
			, ct.start_time
			, ct.end_time
			, ct.Frequency
			, ct.SubFrequency
			, ct.scheduleName AS ScheduleName
			, ct.SchedEnabled--, ss.
		FROM dbo.sysjobs j
			INNER JOIN dbo.sysjobsteps js
				ON j.job_id = js.job_id
			INNER JOIN dbo.syscategories c
				ON j.category_id = c.category_id
			INNER JOIN sys.server_principals sp
				ON j.owner_sid = sp.sid
			INNER JOIN myCTE ct
				ON ct.job_id = j.job_id;

This script will only retrieve the SQL Agent Job schedules that are not SSRS related and that are not SSIS maintenance package related. The SSRS stuff has already been introduced, and a quick eval of those blog posts will show the differences between grabbing SSRS information vs. grabbing just SQL Agent Jobs.

Stay tuned. I will post a script for the SSIS related scheduling information.

Now, go forth and audit your agent jobs. Document the schedules in an easy to read format. Give that report / document to the auditors and your manager and look like a rock star for the day!

As a part of this maintenance you have scripted solutions to intelligently manage and maintain the fragmentation levels for each of the indexes within each database on each instance.

To further show how diligent you are as a DBA, the outcomes of each maintenance run are logged and you review the logs each morning. This routine helps keep you on top of everything that is happening within the environment.

For months, maybe even years, things are running smoothly. Never a failure. Never an error. Just routine log review day after day. Then one day it happens – there is an error. The index maintenance script failed one night.

Index Cannot be Reorganized…

You receive the error message similar to the following:

Msg 2552, Level 16, State 1, Line 1 The index “blah” (partition 1) on table “blah_blah_blah” cannot be reorganized because page level locking is disabled

Immediately, you start double-checking yourself and verifying that it worked the previous night. You even go so far as to confirm that the same index was previously reorganized. How is it possible that it is failing now on this index. What has changed? Has something changed?

Time for a little digging and investigating, so the dirty work begins. On one side of the coin you are relieved to be able to do something different. On the other side of that coin, you are rather annoyed that something seems to have changed. These feelings are perfectly normal!

First things first – you investigate the indexes in question to confirm what the error is saying. This is easily done with a query such as the following:

SELECT i.name AS IdxName
		, SCHEMA_NAME(o.schema_id) AS SchName
		, OBJECT_NAME(i.object_id) AS ObjName
		, i.allow_page_locks
	FROM sys.indexes i
		INNER JOIN sys.objects o
			ON o.object_id = i.object_id
	WHERE OBJECTPROPERTY(i.object_id,'IsMSShipped') = 0;

Scrolling through the results, you notice (eventually) that the IX_SpecialOfferProduct_ProductID in the AdventureWorks2014 database has page locks disabled. You are absolutely certain that this index was created allowing page locks. Pondering the problem for a moment, you recall having read about the default trace (there are several articles on the default trace – here) and the thought occurs to try and see if there is a breadcrumb there about the change. Using the query from that default trace article, a picture starts to unscramble. Here is that query reposted and a snippet of the results:

DECLARE @DBName sysname = 'AdventureWorks2014'
		,@d1 DATETIME
		,@diff INT;

SELECT ObjectName
		  , ObjectID
		  , DatabaseName
		  , StartTime
		  , EventClass
		  , EventSubClass
		  , T.TextData AS SQLStatement
		  , ObjectType
		  , ServerName
		  , LoginName
		  , NTUserName
		  , ApplicationName
		  , CASE EventClass
				WHEN 46
					THEN 'CREATE'
				WHEN 47
					THEN 'DROP'
				WHEN 164
					THEN 'ALTER'
			END AS DDLOperation
		INTO #temp_trace  
	FROM sys.fn_trace_gettable(CONVERT(VARCHAR(150), 
			( SELECT REVERSE(SUBSTRING(REVERSE(path),
					CHARINDEX('\',REVERSE(path)),256)) + 'log.trc'
				FROM    sys.traces
				WHERE   is_default = 1)), DEFAULT) T  
  WHERE EventClass in (46,47,164) 
			AND EventSubclass = 0
			AND ObjectType <> 21587-- don't bother with auto-statistics as it generates too much noise   
			AND DatabaseName = @DBName;


SELECT @d1 = MIN(StartTime) 
	FROM #temp_trace;

SET @diff= DATEDIFF(hh,@d1,GETDATE());


SELECT @diff AS HrsSinceFirstChange    
		, @d1 AS FirstChangeDate 
		, tt.StartTime AS LastChange   
		, sv.name AS obj_type_desc
		, tt.ObjectType
		, tt.DDLOperation
		, tt.SQLStatement
		, tt.DatabaseName,tt.ObjectName
		, tt.EventClass,tt.EventSubClass
		, tt.ServerName,tt.LoginName, tt.NTUserName
		, tt.ApplicationName
		, (dense_rank() OVER (ORDER BY ObjectName,ObjectType ) )%2 AS l1     
		, (dense_rank() OVER (ORDER BY ObjectName,ObjectType,StartTime ))%2 AS l2    
	FROM #temp_trace tt
		INNER JOIN sys.trace_events AS te 
			ON tt.EventClass = te.trace_event_id
		INNER JOIN sys.trace_subclass_values tsv
			ON tt.EventClass = tsv.trace_event_id
			AND tt.ObjectType = tsv.subclass_value
		INNER JOIN master.dbo.spt_values sv 
			ON tsv.subclass_value = sv.number
			AND sv.type = 'EOD'
	ORDER BY StartTime DESC;

DROP TABLE #temp_trace;

This is a great start. Not seen in the results is the timestamp showing when it was done – which was due solely to snipping. Also not shown is the text of the statement that was run for those three events. So close, yet so far away. This is not quite enough to have the smoking gun evidence to show Jason (me) that I did something wrong and unauthorized. What to do now?

All is not lost yet! Your stunning memory kicks in and you recall several articles about using Extended Events to audit server and database changes. Better yet, you recall that you deployed an XE session to the server where this error occurred. And yes, you are slightly embarrassed that you failed to remove the XE session after fiddling with it. We won’t tell anybody that you deployed a test XE session to a production server as if it were your sandbox. The session currently deployed is trapping all object changes unlike the following session that has it filtered down to just objects that are indexes.

USE master;
GO
-- Create the Event Session
IF EXISTS ( SELECT *
				FROM sys.server_event_sessions
				WHERE name = 'ObjChangeAudit' )
	DROP EVENT SESSION ObjChangeAudit 
    ON SERVER;
GO

EXECUTE xp_create_subdir 'C:\Database\XE';
GO

CREATE EVENT SESSION ObjChangeAudit ON SERVER
ADD EVENT sqlserver.object_altered ( SET collect_database_name = ( 1 )
	ACTION ( sqlserver.sql_text,sqlserver.nt_username,sqlserver.server_principal_name,sqlserver.client_hostname,
	package0.collect_system_time,package0.event_sequence ) 
	WHERE database_name = 'AdventureWorks2014'
		AND object_type = 'INDEX'
			OR object_type = 'OBJ'
	)
ADD TARGET package0.event_file ( SET filename = N'C:\Database\XE\ObjChangeAudit.xel' )
WITH (STARTUP_STATE = OFF
	,TRACK_CAUSALITY = ON);

/* start the session */
ALTER EVENT SESSION ObjChangeAudit 
ON SERVER 
STATE = START;
GO

You query the trace file with a query like this:

SELECT CAST ([t2].[event_data] AS XML) AS event_data, t2.file_offset,t2.file_name, cte1.event_session_id--, '' AS event_predicate
	INTO #xmlprocess
	FROM ( SELECT REPLACE(CONVERT(NVARCHAR(128),sesf.value),'.xel','*.xel') AS targetvalue, ses.event_session_id
				FROM sys.server_event_sessions ses
					INNER JOIN sys.server_event_session_fields sesf
						ON ses.event_session_id = sesf.event_session_id
					--INNER JOIN sys.server_event_session_events sese
					--	ON ses.event_session_id = sese.event_session_id
				WHERE sesf.name = 'filename'
					AND ses.name = 'ObjChangeAudit'
					) cte1
				OUTER APPLY sys.fn_xe_file_target_read_file(cte1.targetvalue,	NULL, NULL, NULL) t2
		;

SELECT event_data.value('(event/@name)[1]', 'varchar(50)') AS event_name
		, event_data.value('(event/data[@name="database_name"]/value)[1]',
							'varchar(max)') AS DBName
		, event_data.value('(event/data[@name="ddl_phase"]/text)[1]',
							'varchar(max)') AS DDLPhase
		, event_data.value('(event/data[@name="object_type"]/text)[1]',
							'varchar(max)') AS object_type
		, event_data.value('(event/@timestamp)[1]', 'varchar(50)') AS [TIMESTAMP]
		, event_data.value('(event/action[@name="collect_system_time"]/value)[1]',
							'varchar(max)') AS SystemTime
		, event_data.value('(event/action[@name="client_hostname"]/value)[1]',
							'varchar(max)') AS ClientHostName
		, event_data.value('(event/action[@name="server_principal_name"]/value)[1]',
							'varchar(max)') AS ServerPrincipalName
		, event_data.value('(event/action[@name="nt_username"]/value)[1]',
							'varchar(max)') AS nt_username
		, event_data.value('(event/action[@name="sql_text"]/value)[1]',
							'varchar(max)') AS sql_text
	FROM #xmlprocess x
		LEFT OUTER JOIN sys.server_event_session_events sese
			ON x.event_data.value('(event/@name)[1]', 'varchar(50)') = sese.name
			AND x.event_session_id = sese.event_session_id
	ORDER BY timestamp,event_data.value('(event/action[@name="event_sequence"]/value)[1]',
							'varchar(max)');

DROP TABLE #xmlprocess;

Wow! Look at the results! There is a smoking gun finally.

Two results happen to pin the root cause of the change squarely on my shoulders. Yup, Jason changed that index to disallow page locks. I highlighted three different areas of interest in the results. The yellow and green indicate the DDL phase. One row for the start of the statement, and another row for the commit of that statement. The red highlight shows me the exact change that was made to this index. This is all very good info!

What Now?

It really is great to have the smoking gun. If something is broke and it worked previously, it is essential to find the root cause. With a root cause under the belt, what needs to be done to fix the failure? That is a little bit easier that finding the root cause. That is, unless there is a technical reason for the index to no longer allow page locks (maybe that smoking gun is less of a smoking gun and more like baby spittle after all).

Here is how you fix the problem:

ALTER INDEX IX_SpecialOfferProduct_ProductID ON Sales.SpecialOfferProduct SET (ALLOW_PAGE_LOCKS = ON) 
GO

But, But, But…

The Extended Events session would be very noisy and capture every alter index statement, right? It should capture statements like the following, right?

ALTER INDEX IX_SpecialOfferProduct_ProductID ON Sales.SpecialOfferProduct REBUILD
GO

ALTER INDEX IX_SpecialOfferProduct_ProductID ON Sales.SpecialOfferProduct REORGANIZE
GO

ALTER INDEX IX_SpecialOfferProduct_ProductID ON Sales.SpecialOfferProduct REBUILD WITH (DATA_COMPRESSION = PAGE)
GO

The answer to that question is: Yes, Yes, Yes. These statements are all captured due to the use of the ALTER statement. Here are the results of the XE session with all of these scripts having been executed:

If you want to audit when the indexes are changing and how they are changing, this will do just the trick. If there are a ton of changes, then be prepared for a deluge of information.

With less than a week before the Richmond SQLSaturday event, I am finally getting this post done – maybe.

This is not just a one day event for me. No no! I will be presenting for three consecutive days on many things SQL Server related. The Virginia Tour 2016 really is more of a Richmond Tour 2016 (March 16-19) with all three days being presentations in the Richmond area.

The tour starts off with a visit to the Richmond User Group. You can find details on the evening and event on their website – here.

Here is a quick recap of the NSFW (Naked Sql: Fundamentals to Wisdom) session:

Ever feel like there are just too many layers with SQL Server? You peel off a layer and behold there is yet another layer. SQL Server is rather complex. The complexities can be rather frustrating from time to time.

Join this NSFW (not safe for whining) session as we strip SQL down to the bare bones. I will help you undress the complexities of a few of the basic SQL Server concepts. These concepts will include locks, principals, query directives and deadlocks.

In this session I will disrobe misunderstandings related to the aforementioned topics. I will show the different types of principals in SQL Server. I will also show various scenarios involving locks and deadlocks along with some of their effects with and without query directives. The goal is to better understand these topics and to hopefully stop using and perpetuating common mis-uses of terminology and common misconceptions about the technology.

This should be a light-hearted and fun session with a bit of NSFW learning. Just hope the censors don’t get to the slide deck first!

After the user group presentation to kick off the week, there will be an all-day encore on Friday. Friday is officially a part of the SQLSaturday event. You are welcome to come and participate in my all day workshop – Monitoring SQL Server with Extended Events. There is an event page with details for registering that you can visit – here!

Here is some of what I will be covering in day 2 of this Richmond tour:

Every good enterprising DBA has the prime directive of monitoring the SQL Servers under his/her jurisdiction. The last thing a DBA wants to hear is that there is a problem with a database under his/her purview from an end-user.

SQL Server has progressed over time to give better and better information to the DBA.  Much of this information can be used in a home grown solution to monitor SQL Server. With the constant increase in surface area from Extended Events, the power to build a home grown solution becomes more and more feasible while being limited only by the imagination of the enterprising DBA (or developer or data professional). This full day workshop will introduce you to an Extended Events based solution to achieve that home-grown monitoring goal.

This full day session on Extended Events will help prepare you to put this tool to immediate use as you walk back to your daily duties.  This workshop will teach you about Extended Events starting with the basics and moving through how to create XE sessions that will get the right data for you, while doing so with minimal impact.

You will be exposed to advanced troubleshooting techniques as we work through complex issues that are made easier through the use of XE.  Take advantage of this opportunity to dive into the world of Extended Events and learn how you can make best use of this tool in your SQL 2008+ environment.

This will be a fun day if for nothing else than it is about Extended Events and how to glean great information from your SQL Servers. That and we will keep it light. Just remember, the more you are involved in the workshop the better it will be!

After a brief break to grab some zzzz’s, I will be wrapping up this tour with the grand finale – SQL Saturday 486!!

Not only will I be presenting at this event, I have also volunteered.

I will have a short introductory style presentation about Extended Events. I intend to show a little profiler and a little Extended Events. As usual, I intend to keep the session light, fun and engaging. Here is a little more info on that session:

Extended Events were introduced in SQL Server 2008. With SQL 2014, we have seen a significant upgrade to this feature. Join me for a little adventure into the realm of extended events. We will discuss how to use extended events to aid in performance tuning and in day to day administration. We will also explore some background and the architecture of extended events. This is a fast paced session and will be a lot of fun. Attendees will be able to take back several sample XE sessions to help troubleshoot different issues and get to more intimately know their data/environment.

Then to cap the day, I will be helping to assist with the Wheel of Doom.

The name of the session sounds much worse than it really is. The premise is simple. Speakers and attendees are invited to submit a lightening talk style session with slide deck. Then attendees of the session can volunteer to spin the wheel and present whichever session pops up.

This session is to help people get over the nerve of presenting. It will be very light-hearted and entertaining. No judgement, no shaming, just have fun with it!

Consider the occasional need to change a trace. Maybe you added the wrong event to the trace, or maybe you neglected to include the event in the first place. There is also the possibility that all of the events are correct but the actions or predicates may not be completely ideal for your trace session. What if it is just as simple as a missing target or adding another target to the session? All of these are potential changes to an XEvent session that might be needed.

Today, I will cover how easy it is to modify these sessions without dropping the entire session as many examples on the internet show. Each of the configuration changes just mentioned can be handled through some pretty simple scripts (and yes through the GUI as well).

Altered States

There is no getting around it. To demonstrate how to change event sessions, an event session must first exist. Let’s use the following as the demo session.

CREATE EVENT SESSION [AlteredState] ON SERVER
ADD EVENT sqlserver.auto_stats ( SET collect_database_name = ( 1 )
	ACTION ( package0.event_sequence, sqlos.cpu_id, sqlserver.database_id,
	sqlserver.database_name )
	WHERE ( [database_name] = N'AdventureWorks2014' ) )
ADD TARGET package0.ring_buffer ( SET max_events_limit = ( 666 )
									, max_memory = ( 65536 )
									, occurrence_number = ( 3 ) )
WITH ( MAX_MEMORY = 4096 KB
		, EVENT_RETENTION_MODE = ALLOW_MULTIPLE_EVENT_LOSS
		, MAX_DISPATCH_LATENCY = 5 SECONDS
		, MAX_EVENT_SIZE = 2048 KB
		, MEMORY_PARTITION_MODE = PER_NODE
		, TRACK_CAUSALITY = ON
		, STARTUP_STATE = ON );
GO

This session starts with a ring_buffer target. There is a single event in the session. This event has a few actions applied to it along with a predicate on the database_name field. I also have several of the settings for this session defined (whether at the custom field level or at the session level).

After deploying this session, I quickly realize that I flubbed it up a bit. I forgot a few things. Since those items are missing, I need to get them added without negatively impacting the already gathered data. Let’s start with the missing target. I don’t know how I missed it, but I completely forgot to add a file target to this session. The file target is a must have, so let’s get it added.

/* target change */
ALTER EVENT SESSION [AlteredState] ON SERVER
ADD TARGET package0.event_file ( SET filename = N'AlteredState'
	, max_file_size = ( 50 )
	, max_rollover_files = ( 6 ) );
GO

Phew, that problem has been easily rectified. Further review of the session shows the next problem. My manager requires that the dispatch latency be no more than 4 seconds and that the event memory not exceed 4090 kb. My manager is a little intense about some of these settings. To ensure I adhere to his requirements, I need to change my session settings now.

/* set operations change */
ALTER EVENT SESSION [AlteredState] ON SERVER
WITH ( MAX_MEMORY = 4090 KB
		, MAX_DISPATCH_LATENCY = 4 SECONDS
 );
GO

Good! We are rolling right along here. Fixed a couple of problems real quick with that session and I can continue on with other tasks for the day. Shortly after lunch the manager stops in and asks who changed various database settings. Looking into this session that was just deployed to try and capture some of those events, I get that sinking feeling when I realize that I completely missed that requirement. D’oh! That’s ok for now because I hadn’t turned off the default trace, but I better get the proper event added to the session.

/* add event */
ALTER EVENT SESSION [AlteredState] ON SERVER
ADD EVENT sqlserver.object_altered (
	ACTION ( package0.event_sequence, sqlos.cpu_id, sqlserver.database_id,
	sqlserver.database_name )
	)
;
GO

The next time something changes with the database, it will now be captured. Sadly, too many things are crossing over the desk and I am trying to move too quickly. I forgot to enable the custom field to collect the database name, I better make that adjustment.

/* event custom field change */
ALTER EVENT SESSION [AlteredState] ON SERVER
DROP EVENT sqlserver.object_altered;
ALTER EVENT SESSION [AlteredState] ON SERVER
ADD EVENT sqlserver.object_altered (SET collect_database_name=(1)
	ACTION ( package0.event_sequence, sqlos.cpu_id, sqlserver.database_id,
	sqlserver.database_name )
	)
;
GO

Nice, I have that adjustment made. Unfortunately this is where it does become a bit more complicated. Notice that I had to drop the event from the session first before making that change to the custom field setting for that event? This should be minimal in overall impact since it does not remove the already captured events from the session target. But it is an extra step that must be remembered when making a change that affects an Event within the session.

So far so good. I can capture the events that relate to a database change. I can figure out who made the change and the statement made for the change, right? Nope. I missed that during the setup. I guess I better add that in so I have better information for auditing the changes made to the database. I can start capturing that with the addition of the correct actions.

/* oops wrong actions for the new event */
ALTER EVENT SESSION [AlteredState] ON SERVER
DROP EVENT sqlserver.object_altered;
ALTER EVENT SESSION [AlteredState] ON SERVER
ADD EVENT sqlserver.object_altered (SET collect_database_name=(1)
	ACTION ( package0.event_sequence, sqlserver.client_hostname,sqlserver.nt_username,sqlserver.server_principal_name,
	sqlserver.database_name, sqlserver.sql_text )
	)
;
GO

Everything is now settled in with that session. I go about my merry way for a couple of weeks. Then, one day, I find I need to review the logs to determine who has been changing database settings again. When looking at the log data I discover there are all sorts of object changes being logged to the event session log files. In review of the session definition I figure out the problem. I missed a predicate limiting the captured events to only those that are database type changes. I might as well get that added in.

/* oops, missed that predicate */
ALTER EVENT SESSION [AlteredState] ON SERVER
DROP EVENT sqlserver.object_altered;
ALTER EVENT SESSION [AlteredState] ON SERVER
ADD EVENT sqlserver.object_altered (SET collect_database_name=(1)
	ACTION ( package0.event_sequence, sqlos.cpu_id, sqlserver.database_id,
	sqlserver.database_name, sqlserver.sql_text )
	WHERE ([object_type]='DATABASE')
	)
;
GO

This will now change the event session so the object_altered event will only capture “database” type events. These are the types of events that include settings changes at the database level.

Making changes to an event session without needing to drop and recreate the entire session is a pretty powerful tool. If I script that session as it is now configured, I would get the following:

CREATE EVENT SESSION [AlteredState] ON SERVER
ADD EVENT sqlserver.auto_stats ( SET collect_database_name = ( 1 )
	ACTION ( package0.event_sequence, sqlos.cpu_id, sqlserver.database_id,
	sqlserver.database_name )
	WHERE ( [database_name] = N'AdventureWorks2014' ) ),
ADD EVENT sqlserver.object_altered ( SET collect_database_name = ( 1 )
	ACTION ( package0.event_sequence, sqlos.cpu_id, sqlserver.database_id,
	sqlserver.database_name, sqlserver.sql_text )
	WHERE ( [object_type] = 'DATABASE' ) )
ADD TARGET package0.event_file ( SET filename = N'AlteredState'
									, max_file_size = ( 50 )
									, max_rollover_files = ( 6 ) ),
ADD TARGET package0.ring_buffer ( SET max_events_limit = ( 666 )
									, max_memory = ( 65536 )
									, occurrence_number = ( 3 ) )
WITH ( MAX_MEMORY = 4090 KB
		, EVENT_RETENTION_MODE = ALLOW_MULTIPLE_EVENT_LOSS
		, MAX_DISPATCH_LATENCY = 4 SECONDS
		, MAX_EVENT_SIZE = 2048 KB
		, MEMORY_PARTITION_MODE = PER_NODE
		, TRACK_CAUSALITY = ON
		, STARTUP_STATE = ON );
GO

Compare this to the starting session, and it is apparent that all of the changes implemented are now intact.

Altering an Extended Event session is not necessarily a difficult task, but it is a very useful ability. Having this basic skill in your wheelhouse can be very positive and useful for you!

This has been another article in the 60 Days of XE series. If you have missed any of the articles, or just want a refresher, check out the TOC.

One of the more useful troubleshooting tools (granted when used properly and not with a knee jerk approach) is waits. There are waits in SQL Server that are very specific to Extended Events. Not all waits are bad. Some are innocuous. But with a shoot from the hip approach, these waits can cause many DBAs to focus on the wrong thing.

In this article, I will show one particular wait for Extended Events. As a matter of fact, if you were paying attention to the last article, you will have already seen this wait in passing. To get a quick glimpse or to recall what was discussed, please read the article about the live stream target here.

Patience Padowan

The first thing I want to do is clear my wait stats. Here is a quicky on doing that. Understand that this clears out the wait stats and resets the counters to 0. If you track your waits on a regular basis, this may cause a raised eyebrow by your team-mates.

DBCC SQLPERF("sys.dm_os_wait_stats",CLEAR);

After clearing my waits, I can check for a baseline. When checking for this baseline it is important to note that I have nothing ready from an extended event target currently. I will start that after getting my baseline. Here is what my waits look like prior to working with the target data from any XEvent Session.

This is pretty vanilla prior to working with the targets. That is a good thing for now. This gives me a good sense that the baseline is a good starting point. Now, similar to what was shown in the live stream article previously mentioned, I am going to open a live stream viewer for the system_health session. At this point, you could wait for a minute or three and then re-query the waits. This additional step would be to help show that the XE wait has not yet introduced itself.

Perfect. Now I have a live stream viewer open for the system_health session. I have a good baseline. Now I just need to watch the viewer for a bit. I am doing this to ensure enough time has passed by that my waits have incremented. After a few events pop into the system_health session, I will re-query my waits.

Look at how that wait has zoomed clear to the top! This wait is huge! This wait does not appear until the “Watch Live Data” option is being used to tap into the streaming target (really should be anything that is tapping into the live stream target via the GUI or via some other program). An example of “some other program” could be as simple as somebody querying the sys.fn_MSxe_read_event_stream function from management studio and trying to return the live stream data (as was discussed in the previously mentioned article).

Not understanding what causes the XE_LIVE_TARGET_TVF wait type can cause a data professional, or two, to chase their tail on something that may not be an issue overall. I have seen this happen on more than one occasion where somebody has spent hours trying to chase down the problem that this wait indicates. It doesn’t necessarily indicate a problem (unless you are a shoot from the hip gun-slinging troubleshooter type). It just means that the process accessing the live stream is waiting for more data to come through. That said, if this wait is high, maybe it is time to look into who might be tapping into the Live stream target.

Pretty straight forward and short today. I hope this helps avoid some time-waste for something that can be ignored most of the time.

This has been another article in the 60 Days of XE series. If you have missed any of the articles, or just want a refresher, check out the TOC.

While it may be a quick traipse today, it won’t be the last article on the subject. It is merely an interesting bird walk into a corner of the Extended Events garden that is seldom visited. Today I will explore the little function called sys.fn_MSxe_read_event_stream.

This function, as the name implies, is used to read the event stream and is most commonly seen when watching the live stream of events for a session. I will show a little bit of that.

Gradually Descending into the Stream

First, let’s set the stage so you can confirm that I am not entirely off my rocker here.

By selecting “Watch Live Data” from the context menu after right clicking on the “system_health” session, I will be given a window just like the window discussed in the past few articles when discussing the GUI. After launching that window, a quick query to your favorite requests and sessions DMVs will reveal a little bit about this function that we are diving into today.

If I click on the query text in that previous result set, I can see the following to be the query that is executing for the “live data” stream.

(@source nvarchar(256),@sourceopt int)
SELECT type, data 
	FROM sys.fn_MSxe_read_event_stream (@source, @sourceopt)

Cleaning it up a bit and giving some useful values to the parameters, I might have something like this:

/* read from livestream */
DECLARE @source NVARCHAR(256) = 'system_health'
	, @sourceopt INT = 0;
SELECT type
		, data
	FROM sys.fn_MSxe_read_event_stream(@source, @sourceopt);

Running that particular query from a management studio window would be rather pointless. It doesn’t ever return unless you cancel the query. The key to this one though is the second parameter. The second parameter tells us what kind of source we want to use for the stream of data. There are two values (that I have been able to find) that can be used: 0 and 1. A value of 0 pulls from the live stream target. A value of 1 pulls from the file target. If using a value of 1, then the first parameter needs to be adjusted accordingly as well. If the two parameters do not match, then an error is thrown.

/* read from file */
DECLARE @source NVARCHAR(256) = 'system_health_*.xel'
	, @sourceopt INT = 1;
SELECT type
		, data
	FROM sys.fn_MSxe_read_event_stream(@source, @sourceopt)
	;
GO

As it happens, I have several log files in the default directory for the system_health and the HKEngine sessions that are deployed and running on my instance of SQL Server. Here is a sample of those files:

So far so good, right? Let’s go ahead and execute that second query (that reads from the file target) and see what happens:

This to me is far from useful as of yet. But there are a couple of rabbit holes to dig into from here. The first being the different types that we can see here. Let’s refashion the query to restrict the types being returned and see what happens:

First, in blue we see that type 2 is exactly the same every single time. In my environment I have exactly four of that type. If I look on the file system, I can see that I have four files as well. This seems to indicate (not yet validated in my plucking at it) that this represents a file. Then type 1 is identical up until the point where I have it highlighted in orange. Just an interesting side note there is all.

If I perform the same thing for the HKEngine session, I see something like the following:

Notice the difference here? Now I have two type 1 entries for each file that has been created. In addition, one of the type 1 entries is exactly the same for all files created.

But without being able to translate the data returned, this is really just a fun exercise into the live stream. That said, I give you one last interesting tidbit for today while leaving the second rabbit hole (translating the data into human readable text) for another time (besides just using the event file function to read it).

DBCC OUTPUTBUFFER(53)

Recall from an earlier result that my spid for the live data viewer was spid 53. If run an output buffer for that spid, I will see something like the following:

Not the friendliest of outputs to peruse and try to figure out, but it does give you a sense of what is going on there. If you know how to convert the data output from sys.fn_MSxe_read_event_stream, I would be interested in seeing what you do for it.

This has been another article in the 60 Days of XE series. If you have missed any of the articles, or just want a refresher, check out the TOC.

Today, I have a few gems that are available to you in the GUI. These gems should help solidify your understanding of the tools available through the GUI to help work with Extended Events. So far I haven’t hid my preference for using a script over the GUI. That said, the GUI can be useful for a thing or two. I will explain in better detail further into the article.

Playing in the Mud

To show these gems, a little setup is required first. So I am going to start with the following sample session (started from the GUI):

The setup thus far is rather simple, I have selected two optional settings – “start the event session immediately after session creation” and “track how events are related to one another.” The next thing to do within the GUI is to make my way through the events and select the events I need. The events I want to select need to help me track information related to the SQL process stopping (exiting) and any info related to when a memory dump is created.

To try and find the appropriate events, I will go to the Events tab, click the drop down shown in the green box and select “Event names and descriptions” from the menu.

After making that selection, then I can type a keyword within the text box under “Event library”. Typing in the term “dump” will produce no results. This means either I have no events that will meet my requirements or I need to try a different term, right?

Well, just to verify the results I desire to confirm what I have seen by using a script. Executing the following script:

SELECT c.OBJECT_NAME AS EventName
		,p.name AS PackageName
		,o.description AS EventDescription
		,ch.Channel
	FROM sys.dm_xe_objects o
		INNER JOIN sys.dm_xe_object_columns c 
			ON o.name = c.OBJECT_NAME
			and o.package_guid = c.object_package_guid
		INNER JOIN sys.dm_xe_packages p
			ON o.package_guid = p.guid
		INNER JOIN (SELECT c.object_name AS EventName,c.object_package_guid AS PkgGuid, v.map_value AS Channel
					FROM sys.dm_xe_object_columns c
						INNER JOIN sys.dm_xe_map_values v
							ON c.type_name = v.name
								AND c.column_value = CAST(v.map_key AS NVARCHAR)
					WHERE c.name = 'channel') ch
			ON ch.EventName = c.object_name
			AND ch.PkgGuid = c.object_package_guid
	WHERE object_type='event'
		AND c.name = 'channel'
		AND (c.OBJECT_NAME like '%dump%'
			or o.description like '%dump%'
			--OR o.description LIKE '%grant%'
			)
	ORDER BY o.package_guid;

This confirms my suspicion. The script returns several results with the term “dump” when querying both the descriptions and event names just as I did with the GUI. There is an interesting development however. All of the results show as being in the debug channel.

So why do these results not display from the GUI? Well that is one of the hidden gems. The debug channel is not shown in the GUI results by default. This will prevent any events from that channel from being displayed. This can be changed by checking the box displayed in the following image:

Consider here the name of the Channel for a moment. These events may be obfuscated from search results by default for a good reason. Some of these events can have a significant performance impact on the instance. That said, there is the occasional good reason to need to use the events from this channel in order to troubleshoot specific issues on the server. You just have to dig harder at it.

Having resolved that issue, I have resolved on using the stack_trace and sql_exit_invoked events. So I select both of the events and then move on to the Data Storage tab so I can configure the target. The target I have chosen to use is the event_file target. Now that I have selected the events and configured a target, I can slick the script button. I recommend always using the script button rather than committing the session direct from the GUI. Clicking the script button here will yield the next gem.

I have scripted the session I configured in the GUI. Overlaying the general page of the session with the produced script shows the gem. While the track_causality setting does get scripted, the option to start the session after creating the session does not properly script. This is an interesting problem. While this produces a small negative impact, it is one to bear in mind. If you need to have the session started, then make sure to manually start it or script the start of the session to confirm it has been done.

A third gem is one that I have not shown here but one to play with in your free time. If you change the name of the session and then script it, see what happens.

These are just a few more gems that I have shown in the GUI tools for Extended Events. I recommend using a script where possible (which is just about everywhere), and I have not hid that fact. While useful to a small degree, I do not recommend using the GUI for most tasks (again with a recommendation to use a script instead). Using a script has numerous benefits above and beyond the use of a GUI.

If for some reason you are not using Extended Events simply because you wanted to use a graphical interface, I recommend reconsidering. The series of posts I have on Extended Events has plenty of useful scripts that will provide the functionality and ease of use for Extended Events to mitigate that internal lust for the GUI.

This has been another article in the 60 Days of XE series. If you have missed any of the articles, or just want a refresher, check out the TOC.

Today, I am going to explore how to further customize the view of the GUI for working with Extended Event Log files or XEL files. This further customization does include working with filters. Prior to proceeding, it would be beneficial to review the aforementioned articles – they do outline some building blocks for this article.

Sifting through it all

Even with a well tuned XE session, sometimes there is just a deluge of data. A deluge of data can make it more difficult from time to time to find exactly what is the most likely candidate to be the cause of the problem currently at hand (whatever that may be).

When dealing with a large quantity of data, it becomes necessary to filter that data down. Filtering it down will make it significantly easier to handle – when an appropriate filter is utilized. The GUI permits the implementation of filters in a couple of different ways to help with these dives into the XE logged data.

When convinced to use the GUI to peruse the log data, there are a few possibilities in how to create a filter (and yes it would be better to do it with a script). The first of these methods is actually quite simple. Let’s start with a grid display of the same session used in the previous couple of articles:

From here, if I right click one of the cells as shown below, I will be prompted with a context menu:

By right clicking a value in the grid, an option to “Filter by this value” will be given in the context menu. Selecting that option will open a new window allowing for further configuration of the filter or to just accept the filter as-is.

From the “Filters” window, you can see there is the option to set a time based filter, to create a filter on values for the fields within the trace log or to combine the two. This is pretty straight forward here. That said, recall that filters configured here can be saved in the viewsetting file discussed in the prior article (mentioned at the beginning of this article).

Further Analysis

After filtering down the results, what if you wish to now perform more complex analysis? Maybe the analysis is to be done for trending purposes, or maybe something else. Regardless of the purpose, the need is there and you wish to know how to do it (but for some reason have a serious allergen against doing the work via a script). Luckily there just happens to be a magic button within the GUI that helps to perform this task.

Within the GUI, we are given the opportunity to group and aggregate data. The aggregations cannot be performed without first performing a grouping (very similar to the requirements via script). The grouping can contain multiple fields or just a single field. Let’s take a quick peek.

Clicking on the Grouping button, the following window opens, permitting the configuration of the groups.

This looks pretty familiar and standard. It is rather simple in design and function. Move columns from left to right to add to the grouping. To remove from the grouping, move from right to left.

With the group established, then comes the fun part to help with analysis. This is where aggregations comes into play. Within an aggregation; we can perform counts, max, min, avg and sum. Depending on the field, the aggregation that can be performed may be restricted. Clicking the aggregation button, one will see the following window:

Given the session being evaluated and the columns that have been chosen previously, I am restricted to the columns shown in the preceding image. Aggregations can only be performed on columns selected in the “Column Chooser”. The rest of the configuration of the aggregation is merely an exercise in plug and play to determine which fields or aggregation will serve your needs the best.

For this exercise and the chosen grouping, I am going to go with Average on the duration field and then sort the aggregation by that same field in a descending order. For your purposes, feel free to choose something more meaningful or appropriate to your needs.

Cautionary Tale

If there happens to be a rather large number of events within the session file(s) being evaluated, don’t expect this to work without a hitch or three. Trying to load a 1GB trace file (read that as a default size on an XE trace file) with over 600,000 events, SSMS regularly crashed on me while trying to apply a filter, a group, and even the aggregation. None of this has been an issue while trying to perform similar types of groupings and aggregations with large trace files via script.

If you are going to attempt to manipulate aggregations, groupings, filters etc. via the GUI with large trace files, understand that you may run into errors and crashes when using the graphical interface rather than a script. The choice is yours!

This has been another article in the 60 Days of XE series. If you have missed any of the articles, or just want a refresher, check out the TOC.

Today, I am going to explore how to configure the GUI with filters and different column sets or orders. To proceed, it will require a refresher course via the prior article. If you need a primer in how to merge the XEL files, or to open the XEL files from within the GUI, I recommend taking the time now to read that prior article.

Getting Sticky Again

This should be a rather quick foray into the world of the GUI and how to customize. Just bear in mind that it is a small building block.

Taking up from the prior article, I will continue to use the same sample session and sample data.

Let’s start with a screenshot where we left off the last time:

By itself, this is not the prettiest of layouts. Sure there is good information, but the layout is rather simple and leaves one wanting more. The “more” comes easily enough after some familiarity with the GUI. Suppose you desire to change the order of the columns presented in this view, or to even select different columns, there are two methods to achieve that goal. The first option would be to do as shown here:

By right clicking an empty space in the title bar, a context menu will be made available. There are several options here that can be a bit fun to peruse. I will just focus on the “Choose Columns” option for now.

The alternative for this method is to look in the toolbar. There will be an XE specific toolbar available when looking at an XE log file. In this toolbar there is a “Choose Columns” button. This method will get us to the same end result as the first.

Selecting the “Choose Columns” from either method just presented will open a new window such as the following:

This kind of screen should be fairly familiar and easily discernible. One the left is a set of available columns that can be added in the display. On the right is the list of columns currently being displayed in tabular format. In between there is a set of buttons to add, remove, add all, or remove all columns. In the top right is a means of re-ordering the columns as presented in the display table. Pick a few and sort the columns to a suitable display mode and you are all set.

Recall from the image showing method one that there is an option to remove the column. This same function can be performed from the context menu or from this “Choose Columns” window.

For the order of the columns, there is another fancy means to do that as well other than through the “Choose Columns” window. Columns can be dragged and dropped from the initial screen to change the presentation order easily enough. Sometimes it is far easier to use the “Choose Columns” window to do that. Pick your poison here and run with it.

Super Stickey

This is fantastic if the manipulation of the display settings is a one-off adventure. What if you need to do this multiple times? Maybe, just maybe, this needs to be done for 100 servers. The process I have just shown can become exceedingly tedious. In addition, every time an XEL file is opened, it defaults back to the same settings I have already shown. What do you do in cases like that?

Beyond using a script (the recommended method by the way), there is an alternative.

The alternative is found in the toolbar discussed earlier in this article. The option is “Display Settings.” After picking the columns and the order and getting the display dialed in just right, the view can be saved. The view settings will be saved as an XML file with the extension of viewsetting. This view can then be used to immediately apply to a freshly opened log file. Sure it is still an extra step or two to open that view and can still become a bit tedious, but it is far better than resetting the view every time. And a bonus is that the saved viewsetting file will also save merged columns, groupings, aggregations, column order, and filters defined in the view.

Then again, all of this is easily achieved through a tsql script which is far more scalable when dealing with multiple servers or the need to review the data more than once. The choice is yours when using XE.

Though I mentioned the creation of a filter for the displayed log data, I will not be delving into that topic today. It would be a very suitable exercise for the reader to figure out how to apply a filter through this tool. Or wait until the next time when that topic will be covered along with aggregations and groups. In the meantime, I recommend trying to figure it out with a little picking and plucking through the GUI-ness.

This has been another article in the 60 Days of XE series. If you have missed any of the articles, or just want a refresher, check out the TOC.

As was the case throughout the course of the series, I intend over the next few articles to continue to produce building blocks that may be helpful for later articles, or concepts. With that in mind, the chosen topic today is very basic in nature. While basic, it is a rather important topic.

While working with Extended Events, or with any sort of logging or tracing utility, it becomes critical to consume the data from disparate sources or even to consume the data from a source server on a different server. The question becomes: How to consume that data?

Getting Sticky

When it comes to working with log files from different sources, or most tasks within SQL Server for that matter, there are two general methods. The first method is of the type frequently frowned upon by the more senior of database professionals, sysadmins and so forth. This first method is to use the graphical user interface or GUI. This is more of a point and click method. This is also a method that I will generally shy away from for most tasks. That said, this method is easily performed through the GUI and is reasonably safe.

The first step is illustrated with the following image:

From within Management Studio, navigate the file menu as shown in the preceding image. After clicking on “Merge Extended Event Files”, a dialog will open as shown (in the background) of the next image:

Clicking add from the “Merge Extended Event Files” window will open a dialog that permits you to navigate to the location of the XEL files that need to be opened and merged. Once located, a single file or multiple files can be selected. One should proceed carefully here to ensure against opening too many files or files that may be too large. Opening too many files or files that are too large can cause various unwanted effects.

Once the file(s) is(are) open, a grid like window will be open in SSMS that looks like the following:

From here, one can manipulate the display to something more suitable to his/her display preferences (e.g. filtering or column layout). I will not cover that here but will be saving it (the display customization tutorial) for another time. That aside, I have just shown how easy it is to look into an Extended Event Log file through the use of the GUI.

Not So Sticky

For the less sticky  (less GUI) method, I have shown a similar routine in the past that lays the groundwork for how to do this via script. You can read a little about that here. Building on that method, there is a tweak to be made – we must specifically name the path to the XEL files to be merged rather than try to build it dynamically.

Using the same session files as shown in the GUI examples, I will show how to proceed with the merge via script.

/* xel and xem are required for 2008 and r2 even if using sql 2012 or later to interpret the data
files copied down from remote server 
*/
INSERT into DBA.dbo.SlowQueriesXE
SELECT CAST ([t2].[event_data] AS XML) AS event_data, t2.file_offset,t2.file_name,''
	FROM sys.fn_xe_file_target_read_file(
		--xel path xel = extended event log
		'G:\Database\XE\XE\SlowQueries*.xel'
		--mdpath or xem path xem = extended event metadata
		,	'G:\Database\XE\XE\SlowQueries*.xem'
		--initial file name
		, NULL
		--initial file offset
		, NULL) t2
GO

This script lays out a rather simple process. That process being to load the log data into a table (in a database of your choosing) and then once loaded, query the data to better understand what has been captured. In the preceding script I left a few notes. It is important to understand that if trying to pull in the logs from a SQL 2008 or R2 instance then the XEM file must be included (as I have done for this particular example).

To parse the data into a friendly format that I can use for analysis, I would do something like the following:

SELECT pars.*, map.map_value
	FROM (
		SELECT x.event_data.value('(event/@name)[1]', 'varchar(50)') AS event_name
				--, xe.event_data.value('(@package)[1]', 'varchar(50)') AS package_name
				, DATEADD(hh, DATEDIFF(hh, GETUTCDATE(), CURRENT_TIMESTAMP),
							x.event_data.value('(event/@timestamp)[1]', 'datetime2')) AS [timestamp]
				, x.event_data.value('(event/action[@name="sql_text"]/value)[1]',
									'varchar(max)') AS sql_text
				, x.event_data.value('(event/action[@name="session_id"]/value)[1]',
									'varchar(max)') AS session_id		
				, x.file_offset,x.predicate--,x.file_name
				,event_data.value('(event/data[@name="cpu_time"]/value)[1]','bigint') AS cpu_time
				,event_data.value('(event/data[@name="duration"]/value)[1]','bigint') AS duration
				,event_data.value('(event/data[@name="physical_reads"]/value)[1]','bigint') AS physical_reads
				,event_data.value('(event/data[@name="logical_reads"]/value)[1]','bigint') AS logical_reads
				,event_data.value('(event/data[@name="writes"]/value)[1]','bigint') AS writes
				,event_data.value('(event/data[@name="row_count"]/value)[1]','bigint') AS row_count
				,event_data.value('(event/data[@name="result"]/value)[1]','varchar(max)') AS result
				, event_data.value('(action[@name="tsql_stack"]/value/frames/frame/@handle)[1]','varchar(max)') AS stackhandle
				, event_data.value('(action[@name="tsql_stack"]/value/frames/frame/@offsetStart)[1]','bigint') AS stackoffset
				, event_data.value('(action[@name="tsql_stack"]/value/frames/frame/@offsetEnd)[1]','bigint') AS stackoffsetend
				--, x.event_data
			FROM DBA.dbo.SlowQueriesXE x) pars
		LEFT OUTER JOIN (SELECT oc.object_name as EventName,oc.name as ColName,mv.name as MapName, map_key, map_value 
							FROM sys.dm_xe_map_values mv
								Inner Join sys.dm_xe_object_columns oc
									on mv.name = oc.type_name
									AND mv.object_package_guid = oc.object_package_guid
							WHERE oc.object_name = 'sql_batch_completed'
								AND oc.column_type <> 'readonly'
							) map
			ON pars.result = map.map_key;

From here it is really easy to add/remove columns or re-order the columns into a more friendly format for the person reviewing the data. This is incredibly easy – especially given the appropriate recipe / script / process. I would also venture that this method will require fewer resources and lead to less chance of error.

Enjoy working with evaluating and merging these XEL files from different sources. It should be fun!

This has been another article in the 60 Days of XE series. If you have missed any of the articles, or just want a refresher, check out the TOC.

With the proximity to Halloween, it’s not hard (knowing me) to figure out why I originally decided to terminate this series on the Eve of Halloween day. Then again, as it turns out, I decided to bring the series back to life on the other side of Halloween. In short, it was to be killed on Halloween Eve and then implicitly converted to some living dead creature. You pick whether it is a zombie (which is far better suited to SQL Server) or a vampire.

If you are interested in the previous Halloween posts, here is a list of a few of them:

XE Related

Ghosts in your Database I

Ghosts in your Database II

Ghosts – an eXtrasensory Experience

DB and Fun Related

All Halloween posts

That list is my Halloween treat this year. Now for the trick with a very strong warning. Because of this warning, I am not posting any code showing how to perform the trick.

Warning

DO NOT ATTEMPT THIS ON ANY PRODUCTION SERVER. I BEAR NO RESPONSIBILITY FOR ANY PRODUCTION SERVER CHANGES DUE TO THE READING OF THIS ARTICLE (OR ANY OTHER ARTICLE I HAVE WRITTEN). SOLE RESPONSIBILITY OF CHANGES OR ACTIONS TAKEN BELONG TO THE PERSON PERFORMING THE ACT OR CHANGE. IN OTHER WORDS, YOU ARE SOLELY RESPONSIBLE FOR BREAKING YOUR PRODUCTION ENVIRONMENT IF YOU IMPLEMENT THIS IN PRODUCTION!!!

I have debated seriously over even publishing this “trick” for Halloween because of how deliciously evil it is. I will try and paint the picture in broad strokes. I will leave it up to you to connect the dots.

Problem

A third party vendor has provided an application along with some database requirements. Among these requirements is that the application account must use the ‘sa’ login. You have a strict policy that ‘sa’ must never be used for any applications or by the end-users. This is an extremely protected account by policy. The dilemma you have been presented is that the CEO insists that this application must be used (never happens right?) and the vendor insists the application will not function properly without the use of ‘sa’ (I can hear you chortle at that).

Now you, as the DBA, are stuck between a rock and an even harder place. Being smart (and somewhat smart and sadistic), you insist that the use of the ‘sa’ login should not be performed because it will break SQL Server (in reality we know the login does not break SQL Server, but something inevitably will break due to a mis-step by the person using it a) when they shouldn’t, and b) because they lack proper training). Inside you are grinning from ear to ear because you have some devilish ideas, some of which you learned by attending a session by Rob Volk (twitter) where he shows some devilish methods to deter the use of ‘sa’.

For the sake of the scenario, let’s just say you have a policy preventing the implementation of logon triggers (as suggested by Rob) due to a couple of mishaps a few months back. Somebody implemented a logon trigger that wasn’t properly configured and it happened to prevent all users from accessing the server (including the dba’s). Ruh roh!

And then…

Later in the week, after reading about the power of Extended Events, you decide to play around and do a little discovery. You come across this blog post that shows you how to find all of the possible events within the XEvent Engine. So you run the script that you found and shown here:

SELECT xo.name AS EventName, xo.description
	FROM sys.dm_xe_objects xo
	WHERE xo.object_type = 'event'
		AND (xo.capabilities_desc <> 'private'
		OR xo.capabilities_desc IS NULL)
	ORDER BY xo.name;

And while looking through the events and descriptions you discover that there is an event to log all of the successful logins. Not thinking anything of the third party vendor (because it just makes your blood boil) you begin to dream of the potential for auditing all successful logins (established connections) for all users and documenting who may be doing what and when.

After taking a few notes about the potential for the login event and getting access and logins under better control, you continue along your journey through Extended Events by reading this article about Actions. Feeling comfortable enough from the article, you do what any good data professional, trying to learn a new topic, would do – you start exploring using the provided query:

SELECT xp.name AS PackageName
		, xo.name AS ActionName
		, xo.description AS ObjDescription
		, xo.capabilities_desc
	FROM sys.dm_xe_packages AS xp
		INNER JOIN sys.dm_xe_objects AS xo
			ON xp.guid = xo.package_guid
	WHERE ( xp.capabilities IS NULL
			OR xp.capabilities & 1 = 0
			)
		AND ( xo.capabilities IS NULL
				OR xo.capabilities & 1 = 0
			)
		AND xo.object_type = 'action'
	ORDER BY ActionName, PackageName;

While looking through the available actions, you see one in particular that causes you to raise an eyebrow (not both, just one). There is an action called “debug_break”. You squirm and ponder for quite a bit at the name and definition of this particular object. You wonder out loud “why would anybody ever want that in there?”

Your co-worker interrupts with “Do what?”

To which you reply “Oh nothing important. It was just something I read.” After which you dismiss it, realize the time of day, pack up and head home for the evening. Hours later after the work day has long since been finished, you awaken in a cold sweat with an “Aha!” that startles your slumbering neighbors dog. Immediately you pull out your laptop, connect to your local sandbox instance and get to work with a wry smile and devious giggle.

Upon returning to work the next day, you call the vendor and work out a solution to build them a sandbox server to provide a proof of concept. You grant them the use of sa to use for their application login. Sitting down together and connected to the new sandbox, the vendor attempts to login and the following happens:

Slack-jawed and speechless the vendor pales in complexion. Deep down inside you giggle like an elementary grade school girl – content with yourself. BATTLE WON!

After the vendor departs with a promise to rework the security requirements, you restart the service on the sandbox and go about the rest of your work week with a silly grin from ear to ear and nobody the wiser.

That concludes the trick. In the tradition of Halloween, you knocked on my virtual door of Extended Events and I, in accordance with the traditions of Halloween, gave you a trick and a treat.

I hope this helps you to understand the power of Extended Events. Thinking through a problem and reaching a solution is what XEvents can help to provide. I hope this Halloween XE article was enjoyable to you. Stay tuned as the XE 60 Day series goes full Zombie starting next week.

This has been another article in the 60 Days of XE series. If you have missed any of the articles, or just want a refresher, check out the TOC.