Learn more about SQL Server tools

mssqltips logo
giveaway
 

Tutorials          DBA          Dev          BI          Career          Categories          Webcasts          Whitepapers          Today's Tip          Join

Tutorials      DBA      Dev      BI      Categories      Webcasts

DBA    Dev    BI    Categories

 

Managing Temporal Table History in SQL Server 2016


By:   |   Read Comments   |   Related Tips: 1 | 2 | 3 | 4 | More > SQL Server 2016

Problem

SQL Server 2016 introduced a new feature, Temporal Tables, which allow you to keep a historical record of all of the versions of each row in a table. As rows get introduced, changed, and deleted over time, you can always see what the table looked like during a certain time period or at a specific point in time.

You may have heard Temporal Tables referred to as system-versioned tables. What happens is that the historical versions of rows, according to the system time of their last modification, are stored in a separate table, while the current version resides in the original table.

For tables that don't change very often, this works fantastic - queries against the base table know, based on the filter criteria, whether to get the data from the base table or the history table. For tables with a high volume of insert/delete activity, however, or with rows that get updated frequently, the history table can quickly grow out of control. Imagine a table with 100 rows, and you update those 100 rows 100 times each - you now have 100 rows in the base table, and 9,900 rows in the history table (99 versions of each row).

While there are definitely going to be regulatory/auditing exceptions, in many cases, you won't want or need to keep every single version of every single row for all of time.

Solution

The MSDN article, Manage Retention of Historical Data in System-Versioned Temporal Tables, provides a few options:

The way these solutions are explained, though, lead you to make a blanket choice about retaining historical data only based on a specific point in time (say, archive everything from before January 1, 2017) or fixed windows (once a month, switch out the oldest month). This may be perfectly adequate for your requirements, and that's okay.

When I considered these solutions, I immediately envisioned a scenario that they wouldn't cover: what if I want to keep only the last three versions of a row, regardless of when those modifications took place? Or all previous versions from the past two weeks or the current calendar year, plus one additional version before that? If I archive or delete based only on a point in time, then I might keep too many versions of some rows, and no historical versions of other rows. If I want to keep the three previous versions, I can't possibly enforce that based on a point in time.

Any criteria can be accomplished, of course, if we put a little more thought into the "custom cleanup script" solution. The procedure demonstrated in the documentation accepts a specific datetime value, and deletes all historical data before that point. I'd like to demonstrate how to accomplish a selective delete (or archiving into yet another historical location) using a different set of criteria.

First, we need a base table, and a few rows:

CREATE TABLE dbo.Employees
(
  EmployeeID int PRIMARY KEY,
  FirstName  nvarchar(64),
  LastName   nvarchar(64),
  Salary     int,
  ValidFrom  datetime2(7) GENERATED ALWAYS AS ROW START NOT NULL,
  ValidTo    datetime2(7) GENERATED ALWAYS AS ROW END   NOT NULL,
             PERIOD FOR SYSTEM_TIME (ValidFrom, ValidTo)
) WITH (SYSTEM_VERSIONING = OFF);

INSERT dbo.Employees(EmployeeID, FirstName, LastName, Salary)
  VALUES (1, N'Bobby',  N'Orr',     25000),
         (2, N'Milt',   N'Schmidt', 25000),
         (3, N'Eddie',  N'Shore',   25000);

Now, even though SYSTEM_VERSIONING is OFF, the ValidFrom and ValidTo values are populated with the time of the insert and the end of the day on 9999-12-31, respectively. If you update the data in this table right now, the ValidFrom value will update the current time, but no historical version of the row will be stored anywhere.

We can then create a history table. The columns and data types must match, but the history table can't have constraints. So we're going to create a clustered index on EmployeeID, ValidFrom, ValidTo:

CREATE TABLE dbo.Employees_History
(
  EmployeeID int NOT NULL,
  FirstName  nvarchar(64),
  LastName   nvarchar(64),
  Salary     int,
  ValidFrom  datetime2(7) NOT NULL,
  ValidTo    datetime2(7) NOT NULL
);  

CREATE CLUSTERED INDEX EmployeeID_From_To 
  ON dbo.Employees_History(EmployeeID, ValidFrom, ValidTo);

Next, we'll fictitiously populate it with some historical versions of these rows, just as if I had set this up a couple of years ago (this is *absolutely not* a demonstration of how Temporal Tables should work, nor a recommendation to ever do it this way; we're just trying to set up some dummy data):

-- a historical version representing when we updated salary:
INSERT dbo.Employees_History
(
  EmployeeID, FirstName, LastName, Salary, ValidFrom, ValidTo
)
SELECT EmployeeID, FirstName, LastName, 20000, DATEADD(YEAR, -1, ValidFrom), ValidFrom
FROM dbo.Employees;

INSERT dbo.Employees_History
(
  EmployeeID, FirstName, LastName, Salary, ValidFrom, ValidTo
)
-- then another salary update from a year before:
SELECT EmployeeID, FirstName, LastName, 15000, DATEADD(YEAR, -1, ValidFrom), ValidFrom
FROM dbo.Employees_History
-- and a row that has been "deleted" from the primary table
UNION ALL 
SELECT 4, N'Phil', N'Esposito', 24500, '20150101', '20161231';

-- then, finally, let's add a new row that doesn't exist in history:
INSERT dbo.Employees(EmployeeID, FirstName, LastName, Salary)
  VALUES(5, N'Brad', N'Marchand', 22750);

If we take a quick look, we have 4 rows in the base table, and 7 rows in history:

SELECT * FROM dbo.Employees;
SELECT * FROM dbo.Employees_History;

Base table and history table rows (click to enlarge)

Base table and history table rows

Now, we can turn system versioning for the table ON:

ALTER TABLE dbo.Employees SET (SYSTEM_VERSIONING = ON
(
  HISTORY_TABLE = dbo.Employees_History,
  DATA_CONSISTENCY_CHECK = ON
));

And just for kicks, let's update one row:

UPDATE dbo.Employees SET FirstName = N'Milton' 
WHERE EmployeeID = 2;

What this does, effectively, is moves the existing row from the base table to the history table, updates the ValidTo value to the current time, then creates a new row in the base table with the updated column and a new ValidFrom value. (That is what happens logically, but not necessarily what happens physically.) Now the two tables look like this - up top, I've highlighted the changed value in the base table, and below, the row that now appears in the history table (click to enlarge):

Highlighting changes after an update

Highlighting changes after an update (click to enlarge)

This should demonstrate the purpose of this article: As you update more rows in the base table, the history table can very quickly ramp up and take over your disk, especially if the rows are a lot wider than this simple example.

Identifying Rows to Clean up

Depending on the rules you want to use to determine which history rows to keep or throw away, it should be easy in this case to visually identify those rows, and then build the proper query.

First, let's look at the total set of rows we have in our base table and the history table together:

SELECT *, src = 'Base table' FROM dbo.Employees
UNION ALL
SELECT *, 'History table' FROM dbo.Employees_History
ORDER BY EmployeeID, ValidFrom;

That looks like this (click to enlarge):

Union of base table and history table data

Union of base table and history table data (click to enlarge)

Again, the rules you want to use to dictate which rows to keep and which rows to delete or archive will have bearing on this. There are a variety of things you may want to do, for example you may want to keep a certain number of history versions of a row regardless of time, or you may always want to keep only a certain window, or you may want to use some combination. You may even want to have different retention policies applied to different products, departments, teams, or whatever data you're storing.

Assuming we want to start simple and only keep the previous version of any given row (including the latest version for any key that only exists in the history table), we can first apply shading visually to the above image to highlight the history rows we're going to delete (click to enlarge):

Highlighting history rows to delete

Highlighting history rows to delete (click to enlarge)

My suggestion would be to use a common table expression (CTE) to help identify those rows correctly, as you can then easily change the outer query from a SELECT to a DELETE. So we can start with this:

;WITH h AS 
(
  SELECT *, rn = ROW_NUMBER() OVER 
    (PARTITION BY EmployeeID ORDER BY ValidFrom DESC)
  FROM dbo.Employees_History
)
SELECT * FROM h WHERE rn > 1
-- DELETE h WHERE rn > 1
;

And sure enough, here are the rows we've identified using the SELECT query (click to enlarge):

Proving we've identified the right rows

Proving we've identified the right rows (click to enlarge)

If we want something more flexible than just keeping the latest history row, like keeping *all* of history for the past month in addition to the latest versioned row before that, it just makes our query a little more complex. In this case it only changes the visual by a single row, marked here as "Now keep!" (click to enlarge):

Highlighting rows to delete based on a different retention policy

Highlighting rows to delete based on a different retention policy (click to enlarge)

And the code just uses an additional CTE to first eliminate all of the updates in the past month before determining which rows represent the most recent updates:

;WITH only_history_over_a_month_old AS
(
  -- select only the rows more than a month old (maybe use SYSUTCDATETIME()):
  SELECT * FROM dbo.Employees_History
    WHERE ValidFrom < DATEADD(MONTH, -1, SYSDATETIME())
),
h AS 
(
  SELECT *, rn = ROW_NUMBER() OVER 
    (PARTITION BY EmployeeID ORDER BY ValidFrom DESC)
  FROM only_history_over_a_month_old
)
SELECT * FROM h WHERE rn > 1
-- DELETE h WHERE rn > 1
;

Easier said than done?

The CTEs above use a SELECT as the final query, and have the DELETE commented out. However, you can't just swap the comments to perform the delete. The trick with actually deleting rows from the history table is that system versioning needs to be turned off (true for any DML you want to manually perform). If you try to delete from the table right now, you will get the following error:

Msg 13560, Level 16, State 1
Cannot delete rows from a temporal history table 'SampleDB.dbo.Employees_History'.

You will need to stop versioning the base table, either during a maintenance window, or by implementing the changes within a serializable transaction (or both, to be safe). In either case, you likely want to minimize the amount of time that the base table is not being versioned - by using a serializable transaction, it ensures that any potential writers will be blocked while the changes are taking place.

The sequence, at a high level, would look like this:

  1. Begin a serializable transaction
  2. Turn versioning off on the base table
  3. Purge or move the rows
  4. Turn versioning back on
  5. Commit the transaction

And the code would look like this:

SET TRANSACTION ISOLATION LEVEL SERIALIZABLE;

BEGIN TRANSACTION;

ALTER TABLE dbo.Employees SET (SYSTEM_VERSIONING = OFF);

-- assuming the "delete all but the most recent history version" variation:
;WITH h AS 
(
  SELECT *, rn = ROW_NUMBER() OVER 
    (PARTITION BY EmployeeID ORDER BY ValidFrom DESC)
  FROM dbo.Employees_History
)
DELETE h WHERE rn > 1;

SELECT @@ROWCOUNT; -- maybe only execute up to this point
                   -- if it doesn't look right, ROLLBACK!

ALTER TABLE dbo.Employees SET (SYSTEM_VERSIONING = ON
(
  HISTORY_TABLE = dbo.Employees_History,
  DATA_CONSISTENCY_CHECK = ON
));

COMMIT TRANSACTION;
-- ROLLBACK TRANSACTION;

That CTE + DELETE in the middle would be the only part that needs to change, depending on the retention policy you want to enforce. I should add the disclaimer that you should check @@ROWCOUNT after the DELETE and make sure that not only the right number of rows, but also the right rows, have been deleted (I've left out error handling and detailed validation for brevity).

If you are deleting a large number of history rows, you will likely want to add some batching logic to this, rather than delete all the rows at once, in order to minimize the impact on the transaction log and, more importantly, blocking. (I describe a technique for this in Break large delete operations into chunks. However, since you're relying on CTEs here, you may need to dump those rows to a #temp table in order to process them x rows at a time, or use a cursor to cycle through a certain number of key values at a time. This is so that you don't inadvertently delete the "most recent" row for any key more than once.)

Another Space-Saving Idea

Since system versioning is all-or-nothing, and you don't necessarily need to maintain a historical record of every change to every column in a table, another thing you can consider doing is removing the data from columns that don't need to be audited. If you had an additional column that stored notes or other unstructured data for each employee, where only the current version really matters, you could use the same process as above to set that column to NULL in the history table.

Summary

Temporal tables provide a rich and powerful way to maintain historical versions of rows, with minimal effort. When history grows to the point that maintenance is required, there are many options for cleaning up, and you aren't restricted to naïve, hard-coded, time-based windows. I hope I have shown here how you can accomplish much more flexible retention policies, allowing you to keep exactly the history you want.

Next Steps


Last Update:






About the author
MSSQLTips author Aaron Bertrand Aaron Bertrand is a Senior Consultant at SQL Sentry, Inc., and has been contributing to the community for about two decades, first earning the Microsoft MVP award in 1997.

View all my tips





More SQL Server Solutions











Post a comment or let the author know this tip helped.

All comments are reviewed, so stay on subject or we may delete your comment. Note: your email address is not published. Required fields are marked with an asterisk (*).

*Name    *Email    Notify for updates 


Get free SQL tips:

*Enter Code refresh code     



Learn more about SQL Server tools