By: Arshad Ali | Comments (10) | Related: 1 | 2 | 3 | 4 | 5 | > Fragmentation and Index Maintenance
Problem
In my last tip, Index Fragmentation Report in SQL Server 2005 & 2008, I discussed what fragmentation is, its different types, its performance impact and what are different methods available in SQL Server 2005 and 2008 to identify fragmentation levels. So now once you have identified the high fragmentation level in your database, which could be a bottleneck in your SQL Server performance, what is the next step of fixing this high fragmentation. In this tip, I am going to discuss the different methods and its feasibility to fix the identified high fragmentation levels.
Solution
Fragmented indexes can be de-fragmented in two ways depending on the level of fragmentation and size of the tables in terms of rows.
Reorganize Index
The reorganize operation doesn't take a lot of system resources and can be done while users are accessing the table that the index exists on, that's the reason it is an "online" operation. The reorganize process reorganizes the leaf nodes of the index physically to match it with the logical order, this physical order matching the logical order improves the performance of index scans. The indexes can be reorganized using ALTER INDEX...REORGANIZE command (more details on this command and its usage are given below). This command is the same as DBCC INDEXDEFRAG of previous versions (though it still available for backward compatibility), but there is one difference, it has some additional features like large objects compaction (LOB_COMPACTION) and it is an online operation. Also, reorganize does NOT update statistics.
Script : Index Reorganize
ALTER INDEX { index_name | ALL }
ON <object>
REORGANIZE
[ PARTITION = partition_number ]
[ WITH ( LOB_COMPACTION = { ON | OFF } ) ]
-- index_name | ALL -> name of index or all index on given object to be reorganized
-- <object> -> Name of the object on which the index(es) exist(s)
-- partition_number -> can be specified only if the index_name is a partitioned
--index and specifies the partition which you need to reorganize
-- LOB_COMPACTION -> Specifies that all pages that contain large object (LOB)
--data are compacted. The default is ON.
Rebuild Index
Rebuilding an index means dropping an existing index and creating a new one altogether. Any fragmentation that was in the older index is gone with the drop and in the new index the logical order matches the physical order. Because the existing index is removed and created afresh, users would not be able to access the table and performance may degrade while the index is being built. (If the index you are rebuilding is a cluster index, then your tables will be unavailable to the user during the rebuild process though if it is non-clustered index only a shared lock will be placed on the table so users will be able to access the table, but would not be allowed to do modifications on it). In other words, users will not be able to lock the table during the time the index rebuild is occurring. This could be a major performance bottleneck when rebuilding indexes. Though beginning with SQL Server 2005 you can rebuild your indexes in an online fashion, so that other users are able to access the table and index while the rebuild is being performed. Since you can access the indexes during the rebuild operation, you are not limited to only rebuilding indexes during off-peak hours, though there is some trade-off needed as discussed in the section below.
You might be wondering how SQL Server makes available the index while it is being rebuilt. Well, it uses Row versioning behind the scenes. When you create an index online, it does not drop the existing index and create a new one, but rather it keeps the original index (source) to be used by the users and creates an index (target) parallel to it. The target index resembles the source index which is now a write-only index. Any modifications are applied to the target as well as on the source index. But during the rebuild only the source index will be available for use by the users. Once the index rebuild operation is complete, the original (source) index will be dropped and the meta-data will be updated to make the newly created (target) available for read and write operations. Of course when you plan to rebuild your indexes online, you need to consider its trade-off of having almost double the storage size for indexes to maintain two indexes (source and target) simultaneously (also additional temporary disk might be required for sorting operations).
Indexes can be rebuilt using the following two techniques.
CREATE INDEX Command
One way is to simply drop the index using a DROP INDEX statement followed by a CREATE INDEX statement. Though you can combine these two commands by using the DROP_EXISTING clause of CREATE INDEX command as given below in the script table. You can use the DROP_EXISTING clause to rebuild the index, add or drop columns, modify options, modify column sort order, or change the partition scheme or filegroup. DROP_EXISTING enhances performance when you re-create a clustered index, with either the same or different set of keys, on a table that also has non-clustered indexes. DROP_EXISTING replaces the execution of a DROP INDEX statement on the old clustered index followed by the execution of a CREATE INDEX statement for the new clustered index. The benefit it gives, the non-clustered indexes are rebuilt only once, and even then only if the index definition has changed. With this command you can rebuild the index online.
Script : Index Rebuild using CREATE INDEX Command
ALTER INDEX { index_name | ALL }
CREATE [ UNIQUE ] [ CLUSTERED | NONCLUSTERED ] INDEX index_name
ON <object>
WITH (DROP_EXISTING = { ON | OFF });
-- index_name -> name of index on a given object to be rebuild
-- <object> -> Name of the object on which the index(es) exist(s)
-- DROP_EXISTING -> Specifies that the named, preexisting clustered, or nonclustered is
--dropped and rebuilt. The default is OFF.
-- Note -> Along with DROP_EXISTING clause you can use several other clauses with
--CREATE INDEX command, for more details refer to link provided below.
ALTER INDEX command
This statement replaces the DBCC DBREINDEX statement. The ALTER INDEX statement allows for the rebuilding of the clustered and non-clustered indexes on the table. The drawback with this statement is that you cannot change the index definition the way you can do with the CREATE INDEX command. Though with this command you can rebuild the index online. ALTER INDEX cannot be used to repartition an index or move it to a different filegroup. This statement cannot be used to modify the index definition at all, such as adding or deleting columns or changing the column order. Use CREATE INDEX with the DROP_EXISTING clause to perform these operations as stated above.
Script : Index Rebuild using ALTER INDEX Command
ALTER INDEX { index_name | ALL }
ON <object>
REBUILD WITH (
{
| FILLFACTOR = fillfactor
| SORT_IN_TEMPDB = { ON | OFF }
| STATISTICS_NORECOMPUTE = { ON | OFF }
| ONLINE = { ON | OFF }
| MAXDOP = max_degree_of_parallelism
| DATA_COMPRESSION = { NONE | ROW | PAGE }
[ ON PARTITIONS ( { <partition_number_expression> | <range> }
[ , ...n ] ) ]
}
-- FILLFACTOR -> how full the leaf level of each index page would be during
--index creation or alteration.
-- SORT_IN_TEMPDB -> Specifies whether to store the sort results in tempdb. The default is OFF.
-- STATISTICS_NORECOMPUTE -> Specifies whether distribution statistics are recomputed.
--The default is OFF which means Automatic statistics updating are enabled.
-- ONLINE -> Specifies whether underlying tables and associated indexes are available
--for queries and data modification during the index operation. Only available in
--SQL Server Enterprise edition, has default value OFF.
-- MAXDOP -> Overrides the max degree of parallelism configuration option for the duration
--of the index operation. To learn more about MAXDOP option click here.
-- DATA_COMPRESSION -> Available only in SQL Server 2008, specifies the data compression
--option for the specified index, partition number, or range of partitions.
--Note -> There are some more clauses which you can use with ALTER INDEX
--command, for more details refer to link provided below.
Notes
- Rebuilding the Clustered Index (CI) does not rebuild the Non-Clustered Indexes (NCI) unless the keyword ALL is specified, you will have do it separately.
- If you rebuild your indexes online, you must ensure that you have enough available disk space to hold the index that is being created along with the pre-existing index (source and target indexes simultaneously). After the rebuild operation, the old index will be dropped though. Also, rebuilding indexes online takes a significant amount more time and resources than just rebuilding the index. This is usually a considerable tradeoff since the table will remain available during the rebuild operation.
- Also the underlying table cannot be altered, truncated, or dropped while an online index operation is in process.
- For partitioned indexes built on a partition scheme, you can use either of these methods (Reorganize and Rebuild) on a complete index or on a single partition of an index.
- In general, fragmentation on small indexes is often not controllable, in other words rebuilding or reorganizing small indexes often does not reduce fragmentation. That is because the pages of small indexes are stored on mixed extents. Mixed extents are shared by up to eight different objects, so the fragmentation in a small index might not be reduced after reorganizing or rebuilding the index.
- Index rebuilding can be either online or offline.
- Index reorganizing is always executed online.
- The create index operation can be minimally logged if the database recovery model is set to either bulk-logged or simple.
- An index cannot be reorganized or rebuilt if the filegroup in which it resides is offline or set to read-only.
Example Script to Rebuild / Reorganize Fragmented Indexes
Let's see an example, the script provided below reorganizes and rebuilds the indexes if the fragmentation level is higher than the given threshold. You can define the threshold for reorganizing as well as for rebuilding and the script will work accordingly. Please note this script may take several minutes to several hours depending on the level of fragmentation and size of the tables in terms of rows. Kindly test it on your dev or test environment before executing on your production environment.
--Description : This script reorganizes and rebuilds the index if the fragmentation level is higher the given threshold
-- You can define the threshold for reorganize as well as for rebuild and script will work accordingly
-- INPUTS : @fillfactor - While rebuilding index what would be FILLFACTOR for new index
-- @FragmentationThresholdForReorganizeTableLowerLimit - Fragmentation Level lower threshold to check for reorganizing the table, if the fragmentation is higher than this level, it will be considered for reorganize
-- @@FragmentationThresholdForRebuildTableLowerLimit - Fragmentation Level lower threshold to check for rebuilding the table, if the fragmentation is higher than this level, it will be considered for rebuild
-- NOTES : PRINT statements are all queued up and don't show up until the entire script is printed. However, there is an alternative to PRINTing messages.
-- You can raise an error that isn't really an error (code of 0) and you'll get the same effect--message will be printed immediately.
DECLARE @cmd NVARCHAR(1000)
DECLARE @Table VARCHAR(255)
DECLARE @SchemaName VARCHAR(255)
DECLARE @IndexName VARCHAR(255)
DECLARE @AvgFragmentationInPercent DECIMAL
DECLARE @fillfactor INT
DECLARE @FragmentationThresholdForReorganizeTableLowerLimit VARCHAR(10)
DECLARE @FragmentationThresholdForRebuildTableLowerLimit VARCHAR(10)
DECLARE @Message VARCHAR(1000)
SET NOCOUNT ON
--You can specify your customized value for reorganize and rebuild indexes, the default values
--of 10 and 30 means index will be reorgnized if the fragmentation level is more than equal to 10
--and less than 30, if the fragmentation level is more than equal to 30 then index will be rebuilt
SET @fillfactor = 90
SET @FragmentationThresholdForReorganizeTableLowerLimit = '10.0' -- Percent
SET @FragmentationThresholdForRebuildTableLowerLimit = '30.0' -- Percent
BEGIN TRY
-- ensure the temporary table does not exist
IF (SELECT OBJECT_ID('tempdb..#FramentedTableList')) IS NOT NULL
DROP TABLE #FramentedTableList;
SET @Message = 'DATE : ' + CONVERT(VARCHAR, GETDATE()) + ' - Retrieving indexes with high fragmentation from ' + DB_NAME() + ' database.'
RAISERROR(@Message, 0, 1) WITH NOWAIT
SELECT OBJECT_NAME(IPS.OBJECT_ID) AS [TableName], avg_fragmentation_in_percent, SI.name [IndexName],
schema_name(ST.schema_id) AS [SchemaName], 0 AS IsProcessed INTO #FramentedTableList
FROM sys.dm_db_index_physical_stats(DB_ID(), NULL, NULL, NULL , NULL) IPS
JOIN sys.tables ST WITH (nolock) ON IPS.OBJECT_ID = ST.OBJECT_ID
JOIN sys.indexes SI WITH (nolock) ON IPS.OBJECT_ID = SI.OBJECT_ID AND IPS.index_id = SI.index_id
WHERE ST.is_ms_shipped = 0 AND SI.name IS NOT NULL
AND avg_fragmentation_in_percent >= CONVERT(DECIMAL, @FragmentationThresholdForReorganizeTableLowerLimit)
ORDER BY avg_fragmentation_in_percent DESC
SET @Message = 'DATE : ' + CONVERT(VARCHAR, GETDATE()) + ' - Retrieved indexes with high fragmentation from ' + DB_NAME() + ' database.'
RAISERROR(@Message, 0, 1) WITH NOWAIT
RAISERROR('', 0, 1) WITH NOWAIT
WHILE EXISTS ( SELECT 1 FROM #FramentedTableList WHERE IsProcessed = 0 )
BEGIN
SELECT TOP 1 @Table = TableName, @AvgFragmentationInPercent = avg_fragmentation_in_percent,
@SchemaName = SchemaName, @IndexName = IndexName
FROM #FramentedTableList
WHERE IsProcessed = 0
--Reorganizing the index
IF((@AvgFragmentationInPercent >= @FragmentationThresholdForReorganizeTableLowerLimit) AND (@AvgFragmentationInPercent < @FragmentationThresholdForRebuildTableLowerLimit))
BEGIN
SET @Message = 'DATE : ' + CONVERT(VARCHAR, GETDATE()) + ' - Reorganizing Index for [' + @Table + '] which has avg_fragmentation_in_percent = ' + CONVERT(VARCHAR, @AvgFragmentationInPercent) + '.'
RAISERROR(@Message, 0, 1) WITH NOWAIT
SET @cmd = 'ALTER INDEX ' + @IndexName + ' ON [' + RTRIM(LTRIM(@SchemaName)) + '].[' + RTRIM(LTRIM(@Table)) + '] REORGANIZE'
EXEC (@cmd)
--PRINT @cmd
SET @Message = 'DATE : ' + CONVERT(VARCHAR, GETDATE()) + ' - Reorganize Index completed successfully for [' + @Table + '].'
RAISERROR(@Message, 0, 1) WITH NOWAIT
RAISERROR('', 0, 1) WITH NOWAIT
END
--Rebuilding the index
ELSE IF (@AvgFragmentationInPercent >= @FragmentationThresholdForRebuildTableLowerLimit )
BEGIN
SET @Message = 'DATE : ' + CONVERT(VARCHAR, GETDATE()) + ' - Rebuilding Index for [' + @Table + '] which has avg_fragmentation_in_percent = ' + CONVERT(VARCHAR, @AvgFragmentationInPercent) + '.'
RAISERROR(@Message, 0, 1) WITH NOWAIT
SET @cmd = 'ALTER INDEX ' + @IndexName + ' ON [' + RTRIM(LTRIM(@SchemaName)) + '].[' + RTRIM(LTRIM(@Table)) + '] REBUILD WITH (FILLFACTOR = ' + CONVERT(VARCHAR(3),@fillfactor) + ', STATISTICS_NORECOMPUTE = OFF)'
EXEC (@cmd)
--PRINT @cmd
SET @Message = 'DATE : ' + CONVERT(VARCHAR, GETDATE()) + ' - Rebuild Index completed successfully for [' + @Table + '].'
RAISERROR(@Message, 0, 1) WITH NOWAIT
RAISERROR('', 0, 1) WITH NOWAIT
END
UPDATE #FramentedTableList
SET IsProcessed = 1
WHERE TableName = @Table
AND IndexName = @IndexName
END
DROP TABLE #FramentedTableList
END TRY
BEGIN CATCH
PRINT 'DATE : ' + CONVERT(VARCHAR, GETDATE()) + ' There is some run time exception.'
PRINT 'ERROR CODE : ' + CONVERT(VARCHAR, ERROR_NUMBER())
PRINT 'ERROR MESSAGE : ' + ERROR_MESSAGE()
END CATCH
Next Steps
- Review Index Fragmentation Report in SQL Server 2005 and 2008 tip.
- Review SQL Server script to rebuild all indexes for all tables and all databases tip.
- Review Using SQL Server DMVs to Identify Missing Indexes tip.
- Review Deeper insight into unused indexes for SQL Server tip.
- Review ALTER INDEX command on MSDN.
- Review CREATE INDEX command on MSDN.
- Review How Online Index Operations Work on MSDN.
- Review Disk Space Requirements for Index DDL Operations on MSDN
- Some of this information is from SQL Server Books Online
About the author
Arshad Ali is a SQL and BI Developer focusing on Data Warehousing projects for Microsoft.This author pledges the content of this article is based on professional experience and not AI generated.
View all my tips
