sql: How to remove duplicate rows (content the same, but sequence is different)

Question

You see that the SKU1 has 2 rows, but actually the content of these 2 rows are the same, just the sequence of "b" and "c" makes difference.

What if I want to remove the duplicate rows as shown in the 2nd picture?

In Oracle there is a LEAST/GREATEST function that can realize it, but I used SQL Server, therefore it doesn't work following the instruction of the below post:

How to remove duplicate rows in SQL

Answer 1

Please Use Max() and Min Function instead of least and greatest of oracle if used the follwoing steps and got the same result.

Create Table Transactions (Name varchar(255),Quantity1 int,Quantity2 int)
   Insert Into Transactions values
   ('SKU1',10,20),
      ('SKU1',20,10),
   ('SKU2',10,20),
   ('SKU2',10,20)

Now I used the query below to find the solution of your answer

    Select T1.Name,MAX(T1.Quantity1),MIN(T2.Quantity2) From Transactions   T1
    join Transactions T2
    on T1.Name=T2.Name
    group by T1.Name

Please Reply

Answer 2

If it's only 2 columns where the order should not matter for the group by?

Then you could use IIF (or a CASE WHEN) to calculate the maximum and minimum values.
And use those calculated values in the GROUP BY.

For example:

select Name, 
MAX(Val1) as Val1,
MIN(Val2) as Val2
from Table1
GROUP BY Name,
IIF(Val2 is null or Val1 < Val2, Val1, Val2),
IIF(Val1 is null or Val1 < Val2, Val2, Val1);

For the example records that would give the result:

Name Val1 Val2
SKU1 20   10
SKU2 20   10

Or if you want to use a fancy XML trick :

select Name, max(Val1) as Val1, min(Val2) as Val2
from (
  select *, 
  cast(
    convert(XML, 
      concat('<n>',Val1,'</n><n>',Val2,'</n>')
    ).query('for $n in /n order by $n return string($n)'
  ) as varchar(6)) as SortedValues
  from Table1
) q
group by Name, SortedValues;

The last method could be more usefull when there are more columns involved.

To actually remove the duplicates?
Here's an example that uses a table variable to demonstrate:

declare @Table1 TABLE (Id int, Name varchar(20), Val1 int, Val2 int);

Insert Into @Table1 values
(1,'SKU1',10,20),
(2,'SKU1',20,10),
(3,'SKU1',12,15),
(4,'SKU2',10,null),
(5,'SKU2',null,10),
(6,'SKU2',10,20);

delete from @Table1
where Id in (
    select Id
    from (
    select Id, 
    row_number() over (partition by Name, 
         IIF(Val2 is null or Val1 < Val2, Val1, Val2),
         IIF(Val1 is null or Val1 < Val2, Val2, Val1)
       order by Val1 desc, Val2 desc
    ) as rn
    from @Table1
    ) q
    where rn > 1
);

select * from @Table1;

Answer 3

greatest() can be simulated using a CASE expression

greatest(b,c) is the same as:

case 
   when b > c then b
   else c
end

You can use this together with a distinct to remove your duplicates:

select distinct 
          a, 
          case when b > c then b else c end as x
from the_table
order by a;

Answer 4

Try %%physloc%%. It is the equivalent of oracle's RowId.

1. Find it

   select *, %%physloc%% from [MyTable] where ...

2. Delete what you want

   delete from [MyTable] where %%physloc%% = 0xDEADBEEF -- (your address)

3. Consider adding a Unique / Primary Key to prevent future occurrences.

Answer 5

SELECT * FROM abc where A='SKU1'and B=20 || A='SKU2'and B=10 

 a      b    c  
SKU1    20  10
SKU2    10  20

Answer 6

From your question it is not clear whether you want to filter duplicates by row or duplicates by column. Let me describe both to make sure your question is addressed completely.

---

In Example 1, you can see that we have duplicate rows:

To filter them, just add the keyword DISTINCT to your query, as follows:

SELECT DISTINCT * FROM myTable;

It filters the duplicate rows and returns:

Hence, you don't need a least or greatest function in this case.

---

In Example 2, you can see that we have duplicates in the columns:

Here, SELECT DISTINCT * from abc will still return all 4 rows. If we regard only the first column in the filtering, it can be achieved by the following query:

select distinct t.Col1,
    (select top 1 Col2 from myTable ts where t.Col1=ts.Col1) Col2,
    (select top 1 Col3 from myTable ts where t.Col1=ts.Col1) Col3
from myTable t

It will pick the first matching value in each column, so the result of the query will be:

The difference between Example 1 and this example is that it eliminated just the duplicate occurances of the values in Col1 of myTable and then returned the related values of the other columns - hence the results in Col1 and Col2 are different.

Note:

• In this case you can't just join the table myTable because then you would be forced to list the columns in the select distinct which would return more rows then you want to have. Unfortunately, T-SQL does not offer something like SELECT DISTINCT ON(fieldname) i.e. you can't directly specify a distinct (single) fieldname.
• You might have thought "why not use GROUP BY?" The answer of that question is here: With GROUP BY you are forced to either specify all columns which is a technical DISTINCT equivalent, or you need to use aggregate functions like MIN or MAX which aren't returning what you want either.

A more advanced query (you might have seen is once before!) which has the same result is:

SELECT Col1, Col2, Col3 
FROM    (
        SELECT *, ROW_NUMBER() OVER (PARTITION BY Col1 ORDER BY Col1) AS RowNumber 
        FROM myTable
        ) t
WHERE RowNumber=1

This statement numbers each occurance of a value on Col1 in the subquery and then takes the first of each duplicate rows - which effectively is a grouping by Col1 (but without the disadvantages of GROUP BY).

---

N.B. In the examples above, I am assuming a table definition like:

CREATE TABLE [dbo].[myTable](
    [Col1] [nvarchar](max) NULL,
    [Col2] [int] NULL,
    [Col3] [int] NULL
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]

For the examples above, we don't need to declare a primary key column. But generally spoken you'll need a primary key in database tables, to be able to reference rows efficiently.

If you want to permanently delete rows not needed, you should introduce a primary key, because then you can delete the rows not displayed easily as follows (i.e. it is the inverse filter of the advanced query mentioned above):

DELETE FROM [dbo].[myTable]
WHERE  myPK NOT IN
    (SELECT myPK  
        FROM    (
            SELECT *, ROW_NUMBER() OVER (PARTITION BY Col1 ORDER BY Col1) AS RowNumber 
            FROM [dbo].[myTable]
        ) t
    WHERE RowNumber=1 and myPK=t.myPK)

This assumes you have added an integer primary key myPK which auto-increments (you can do that via the SQL Management Studio easily by using the designer).

Or you can execute the following query to add it to the existing table:

BEGIN TRANSACTION
GO
ALTER TABLE dbo.myTable ADD
    myPK int NOT NULL IDENTITY (1, 1)
GO
ALTER TABLE dbo.myTable ADD CONSTRAINT
    PK_myTable PRIMARY KEY CLUSTERED (myPK) 
    WITH (STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, 
            ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) 
    ON [PRIMARY]

GO
ALTER TABLE dbo.myTable SET (LOCK_ESCALATION = TABLE)
GO
COMMIT

---

You can find some more examples here at MSDN.

Answer 7

This may seem a little complicated at first, but we can also use PIVOT/UNPIVOT to obtain the results

Below is the query

select * 
from 
    (
    select 
        *, 
        'quantity'+ cast(row_number() over (partition by name order by data) as nvarchar) cols  
    from
        (
            select 
                distinct name, data 
            from 
                (select * from transactions)s
            unpivot
            (
                data for cols in (quantity1,quantity2)
            )u
        )s
    )s
pivot
(
    max(data) for cols in (quantity1,quantity2)
)p