solving sql server problems for millions of dbas and developers since 2006



SQL Server DBA Tips SQL Server Developer Tips SQL Server Business Intelligence Tips SQL Server Career Tips SQL Server Tip Categories SQL Server Tutorials SQL Server Webcasts SQL Server Whitepapers SQL Server Tools SQL Server Questions and Answers MSSQLTips Authors About MSSQLTips SQL Server User Groups MSSLQTips Giveaways MSSQLTips Advertising Options

MSSQLTips Facebook Page MSSQLTips LinkedIn Page MSSQLTips RSS Feed MSSQLTips Twitter Page MSSQLTips Google+ Page








Delete duplicate rows with no primary key on a SQL Server table

By: | Read Comments (24) | Print

Greg is the President of Edgewood Solutions and a co-founder of MSSQLTips.com.

Related Tips: More

Problem
Every once in awhile a table gets created without a primary key and duplicate records get entered.  The problem gets even worse when you have two identical rows in the table and there is no way to distinguish between the two rows.  So how do you delete the duplicate record?

Solution
One option that SQL Server gives you is the ability to set ROWCOUNT which limits the numbers of records affected by a command.  The default value is 0 which means all records, but this value can be set prior to running a command.  So let's create a table and add 4 records with one duplicate record.

Create a table called duplicateTest and add 4 records.

CREATE TABLE dbo.duplicateTest
(
[ID] [int] ,
[FirstName] [varchar](25),
[LastName] [varchar](25
ON [PRIMARY]

INSERT INTO dbo.duplicateTest VALUES(1'Bob','Smith')
INSERT INTO dbo.duplicateTest VALUES(2'Dave','Jones')
INSERT INTO dbo.duplicateTest VALUES(3'Karen','White')
INSERT INTO dbo.duplicateTest VALUES(1'Bob','Smith')

If we select all data we get the following:

SELECT FROM dbo.duplicateTest

ID FirstName LastName
1 Bob Smith
2 Dave Jones
3 Karen White
1 Bob Smith

If we try to select the record for Bob Smith will all of the available values such as the following query:

SELECT FROM dbo.duplicateTest WHERE ID AND FirstName 'Bob' AND LastName 'Smith'

We still get 2 rows of data:

ID FirstName LastName
1 Bob Smith
1 Bob Smith

So to delete the duplicate record with SQL Server 2000 and 2005 we can use the SET ROWCOUNT command to limit the number of rows affected by a query.  By setting it to 1 we can just delete one of these rows in the table.  Note: the select commands are just used to show the data prior and after the delete occurs.

SELECT FROM dbo.duplicateTest

SET ROWCOUNT 1
DELETE FROM dbo.duplicateTest WHERE ID 1
SET ROWCOUNT 0

SELECT FROM dbo.duplicateTest

With SQL Server 2005 we can also use the TOP command when we issue the delete, such as the following. Note: the select commands are just used to show the data prior and after the delete occurs.

SELECT FROM dbo.duplicateTest
DELETE TOP(1FROM dbo.duplicateTest WHERE ID 1
SELECT FROM dbo.duplicateTest

So as you can see with SQL Server 2005 there are two options to allow you to delete duplicate identical rows of data in your tables.

Here is one note from Microsoft about using SET ROWCOUNT:

Using SET ROWCOUNT will not affect DELETE, INSERT, and UPDATE statements in the next release of SQL Server. Avoid using SET ROWCOUNT together with DELETE, INSERT, and UPDATE statements in new development work, and plan to modify applications that currently use it. Also, for DELETE, INSERT, and UPDATE statements that currently use SET ROWCOUNT, we recommend that you rewrite them to use the TOP syntax.

Next Steps

  • Take a look how the ROWCOUNT command can be used to affect the results of your query
  • Also take a look at the TOP command and changes that have been implemented with SQL Server 2005
  • Start using TOP instead of ROWCOUNT for SQL Server 2005 and later

Change History

Release History
May 24, 2007 Here is an updated way to delete a variable set of duplicate rows using the TOP command.  The data below represents a sample data set where the same value appears multiple times, but there is no primary key to only delete all but one record.

EmpID
Aufaq
Aufaq
Aufaq
Aufaq
Aufaq
Basharat
Basharat
Basharat
Basharat
John
John
John
John
John
John

If you use the following command this will get a count of how many rows there are and delete this minus one record.

DELETE TOP (SELECT COUNT(*) -FROM dbo.Emptest WHERE EmpID 'Basharat'
FROM dbo.Emptest 
WHERE EmpID 'Basharat'

Thanks goes out to Basharat Bhat for this update to this tip.

Basharat Bhat
BQE Software Inc, Kashmir, INDIA
Software Developer (Microsoft Technologies)



Related Tips: More | Become a paid author


Last Update: 11/10/2006

Share: Share 






Comments and Feedback:

Tuesday, March 18, 2008 - 2:29:33 AM - Bals Read The Tip

The corelated query in TOP clause is not working .. Basharat Bhat has given only for single dup entry


Friday, April 04, 2008 - 7:38:20 AM - admin Read The Tip

Bals,

Can you please post the code you are having an issue with?

Thank you,
The MSSQLTips.com Team


Tuesday, May 06, 2008 - 1:16:11 PM - Gadi35 Read The Tip

There is another very good way to do this, in SQL Server 2005, if you do not have the key set up, but want to effectively use a column or more as your primary key.

 ;with DelDup as (select row_number() over (partition by
FirstName, LastName
order by FirstName, LastName) as RowNofrom duplicateTest)
Delete from DelDup where RowNo> 1

--alternate--

 ;with DelDup as (select row_number() over (partition by
ID
order by ID) as RowNofrom duplicateTest)
Delete from DelDup where RowNo> 1

You can easily adjust this to use one column or as many as you wish, as long as you update the column(s) after the partition by and after the order by.


Friday, May 09, 2008 - 2:29:02 PM - admin Read The Tip

Thank you both for the feedback.

Thank you,
The MSSQLTips.com Team


Thursday, May 15, 2008 - 7:31:44 AM - Prakash M Read The Tip

Hi Friend Pls test the following one, dis s a very simple method

Delete Top (Select Count(*)-1 From EmpTest Where EmpId=A.EmpId)
From EmpTest As A

 

 


Sunday, May 18, 2008 - 11:57:57 PM - balakumar.sk Read The Tip

It didnt work Prakash,

gave me the follwoing error

Msg 156, Level 15, State 1, Line 2

Incorrect syntax near the keyword 'As'."

 


Friday, May 29, 2009 - 2:22:18 AM - kanagarajum Read The Tip

Hai,  can u try this code?

 

DELETE FROM TestTable WHERE
EXISTS
(SELECT Sid FROM TestTable T WHERE T.Tname = TestTable.Tname AND I.SId < TestTable.SId)

 

 

 


Wednesday, February 22, 2012 - 12:30:20 AM - karthick Read The Tip

hi,

im facing one problem in ur example you knows the duplicate data's key value so you can delete easily.

What my question in there is 1000's records with few duplicate (in composit key one is different) then how to extract the that particular duplicate  records and delete it

 


Friday, March 02, 2012 - 1:49:40 AM - manolitobsj Read The Tip

This post have been posted for more than two year but it helped me alot. Thanks.


Friday, March 09, 2012 - 6:24:41 AM - gopi Read The Tip

There are duplicate records in Fact and Facterror table(both table have same records in them some 100 records).i need to delete the records from facterror table and insert it into Fact table.


Friday, March 09, 2012 - 8:24:37 AM - Greg Robidoux Read The Tip

Gopi, not sure what you are asking.

If the records already exist in both the FACT table and FACTERROR then can you just delete the records from the FACTERROR table?


Wednesday, March 14, 2012 - 1:42:54 PM - erni Read The Tip

how to delete duplicate using phpmyadmin?is working when insert data,data can not read..that he publish 1234567 are duplicate primary key and when insert 3456789,he also publish 1234567?

how i can i do?plz


Wednesday, March 14, 2012 - 1:48:47 PM - Greg Robidoux Read The Tip

Hi Erni, not exactly sure what you need to do.

If you are trying to eliminate duplicate primary key values in a table, make sure the table has a PRIMARY KEY setup so there is not a chance to get duplicate records.  The front end application would need to handle the error message from SQL Server, but this would ensure you don't have duplicates.


Tuesday, April 10, 2012 - 6:50:11 AM - kisan Read The Tip

I want To delete Secound Record  how i can delete:

ID FirstName LastName
1 Bob Smith
1 Bob Smith

 


Tuesday, April 10, 2012 - 7:31:14 AM - Greg Robidoux Read The Tip

Hi Kisan,

if the records are identical it doesn't really matter which record you are deleting also you have no control over what record SQL Server will present to you.

If there are other columns in the table that have different values then you can add that to the where clause if not you won't have any real control.

You can also look the post above from Gadi35.


Wednesday, April 11, 2012 - 8:13:02 AM - kisan Read The Tip

I got your answer 

 if i  delete record it will delete both  beczz their is no. primary key use it...

if i m taking any column so i cant delete but thier is redudency  it will delete both record......

 

so telll me sir what synatx should i use in delete query to deleted secound record..


Wednesday, April 11, 2012 - 8:49:12 AM - Greg Robidoux Read The Tip

Kisan - if you run this query this will delete only one of these records. Since they are identical it doesn't matter which one you delete.

DELETE TOP(1) FROM dbo.duplicateTest WHERE ID = 1

 


Wednesday, April 11, 2012 - 12:14:13 PM - Gadi35 Read The Tip

Greg, that last code will cancel anything that exists and only leave the duplicates.


Wednesday, April 11, 2012 - 12:55:19 PM - Greg Robidoux Read The Tip

Gadi35 - yes you are correct that it will only delete one record.  I was referring to the example that "kisan" listed that just had two records, so this would delete one of those two records, but I agree if there were more than two duplicates only one record would be deleted.

 


Wednesday, April 11, 2012 - 12:56:01 PM - kisan Read The Tip

thanku...  sir.......

 

this  i try this query it will work fine....... thanku muchhhhhh..


Thursday, April 12, 2012 - 3:00:31 AM - kisan Read The Tip

 

sir I have One More Question

Suppose i create auto id like

create table demo

(

 id int identity(1,1) primary key

)

if i inserted  5 records and i delte  record Number 5

so agin if i inserted  record so it will give id no:6

but i want  id 5

so how will be do sir....


Thursday, April 12, 2012 - 8:59:47 AM - Greg Robidoux Read The Tip

Kisan - if you delete records from a table that uses an identity column you are going to have gaps in the numbering.

If there were only 5 records in the table 1,2,3,4,5 and you delete record number 5 the next identity value will still be 6.  You could use the RESEED option to fix this and make the next number 5 again.  Take a look at this tip: http://www.mssqltips.com/sqlservertip/1123/managing-and-maintaining-sql-server-identity-values/

But if you have records with values 1,2,3,4,5 and you delete record number 3 you will still have a row without a number 3 value.  To get around this you could use SET IDENTITY INSERT.  Take a look at this tip: http://www.mssqltips.com/sqlservertip/1061/using-identity-insert-to-keep-sql-server-table-keys-in-sync/

 


Friday, April 27, 2012 - 8:45:00 AM - Jeff Read The Tip

Seems like you could also use select top 1 


Friday, April 27, 2012 - 4:57:45 PM - varshini Read The Tip

 

Nice Job

 



Post a Comment or Question

Keep it clean and stay on the subject or we may delete your comment.
Your email address is not published. Required fields are marked with an asterisk (*)

*Name   *Email   Notify for updates
Comments
*Enter Code refresh code


 
Sponsor Information
"SQL diagnostic manager delivers response in minutes, not hours!"

SQL Monitor – For database professionals who need results on Day One. Try it online.

Need SQL Server help and not sure where to turn? Reach out to expert consultants for a Health Check.

Get SQL Server Tips Straight from Kevin Kline.

Join the over million SQL Server Professionals who get their issues resolved daily.

Free web casts for DBAs and Developers on Performance Tuning, Development, Administration and more....


Copyright (c) 2006-2012 Edgewood Solutions, LLC All rights reserved
privacy | disclaimer | copyright | advertise | about
authors | contribute | feedback | giveaways | user groups
Some names and products listed are the registered trademarks of their respective owners.


Edgewood Solutions LLC | MSSharePointTips.com | MSSQLTips.com