Code works (mostly). But failure to remove duplicates

Question

Summary:

Hey I've got a filter for some data. It's filtering fine but it's not removing duplicate objects. Arguments taken in order:

1. datasource
2. filters (remove all matching)
3. unique: (whether to remove duplicate items)
4. prop ( object properties - used for Listing a an attribute of unfiltered items - not in use))

Data:

var moduleData = [
     {manufacturer: "SunPower Corp.", productNo: "SPR-M475-COM-MLSD", watts: 475, cells: 72, warranty: 25, degradation: 0, volume: 0, smartModule: false, adder: 0, available: true},
     {manufacturer: "TestCo", productNo: "TE-STc001", watts: 475, cells: 72, warranty: 25, degredation: 0, volume: 0, smartModule: false, adder: 0, available: true},
     {manufacturer: "TestCo", productNo: "TE-STc001", watts: 475, cells: 72, warranty: 25, degredation: 0, volume: 0, smartModule: false, adder: 0, available: true},
     {manufacturer: "TestCo", productNo: "TE-STc002", watts: 475, cells: 72, warranty: 25, degredation: 0, volume: 0, smartModule: false, adder: 0, available: true},
     {manufacturer: "TestCo", productNo: "TE-STc002.5", watts: 475, cells: 72, warranty: 25, degredation: 0, volume: 0, smartModule: false, adder: 0, available: true},
     {manufacturer: "TestCoDuplicate", productNo: "TE-STc002.5", watts: 430, cells: 71, warranty: 23, degredation: 2, volume: 1, smartModule: true, adder: 5, available: true},
     {manufacturer: "TestCo", productNo: "TE-STc003", watts: 475, cells: 72, warranty: 25, degredation: 0, volume: 0, smartModule: false, adder: 0, available: true},
    ]

Function:

const filters = {}

const getFilteredArray = ((data, filters, isItUnique, prop  ) => {
//Arrays

    // filtered Array =======================================
    if ((prop.length > 0 || prop != undefined || prop != null) && (prop.length == 0 || prop == undefined || prop == null)){ 
        let nonUniqueArray = data.filter(p => 
            filters.every(f=> Object.keys(f).every(k => p[k] === f[k]))
        )
        console.log('filtered Array - (unique, no prop) ')
        
        var unique = []
        //XXXXXXXXXXXXX make array unique XXXXXXXXXXXXXXXXX
        
            const uniqueArray = nonUniqueArray.filter(element => {
                const isDuplicate = unique.includes(element);

                if (!isDuplicate) {
                    unique.push(element)
            
                    return true
                }
            })

        return uniqueArray
    }
})


filters.available = true
filters.manufacturer = "TestCo"


//console.log(filters)
console.log(getFilteredArray(moduleData, [filters], "unique", "" ))

So I'm looking for products with a manufacturer of TestCo, that's available and looking to remove duplicates.

ITS RETURNING:

Console:[Object, Object, Object, Object, Object] (5)<br>
0 {manufacturer: "TestCo", productNo: "TE-STc001", watts: 475, cells: 72, warranty: 25, …}<br>
1 {manufacturer: "TestCo", productNo: "TE-STc001", watts: 475, cells: 72, warranty: 25, …}<br>
2 {manufacturer: "TestCo", productNo: "TE-STc002", watts: 475, cells: 72, warranty: 25, …}<br>
3 {manufacturer: "TestCo", productNo: "TE-STc002.5", watts: 475, cells: 72, warranty: 25, …}<br>
4 {manufacturer: "TestCo", productNo: "TE-STc003", watts: 475, cells: 72, warranty: 25, …}<br>

Its clear you can see prod No TE-STc001 twice. (not removed). Being a complete copy, I would think one of them wouldn't be added to the uniqueArray. Why is that and how can I fix my code? Thanks in advance!

Speaking of duplicates, does this answer your question? [Get all unique values in a JavaScript array (remove duplicates)](https://stackoverflow.com/questions/1960473/get-all-unique-values-in-a-javascript-array-remove-duplicates) — Yogi, Apr 20 '22 at 22:49
Rather than embedding a `.every()` inside a .filter() with another `.every()` inside of the first `.every()` (which is N x N x N iterations), I prefer to build a Map object of the "unique" key and let the Map tell you what is and isn't unique and then you can output the unique Map to your final form. — jfriend00, Apr 20 '22 at 22:58
@Yogi so that link gave me this example" // usage example: var myArray = ['a', 1, 'a', 2, '1']; var unique = myArray.filter((v, i, a) => a.indexOf(v) === i); console.log(unique); // unique is ['a', 1, 2, '1']. This is what I replaced it with: function onlyUnique(value, index, self) { return self.indexOf(value) === index; } var uniqueArray = nonUniqueArray.filter(onlyUnique); still same result. — Michael Martell, Apr 20 '22 at 23:04
@ yogi. I tried this as well: same output. var uniqueArray = nonUniqueArray.filter((v, i, a) => a.indexOf(v) === i); — Michael Martell, Apr 20 '22 at 23:10
Based on the data it seems like the `productNo` property can be used to determine duplicates. If that's the case you can just do `return nonUniqueArray.filter((v,i,a)=>a.findIndex(v2=>(v2.productNo===v.productNo))===i)` which was taken from the top answer here https://stackoverflow.com/questions/2218999/how-to-remove-all-duplicates-from-an-array-of-objects — CaseyC, Apr 20 '22 at 23:38
@CaseyC Believe it or not I have 2 manufacturers with the same product number and I need access to both items separately. — Michael Martell, Apr 21 '22 at 00:15
You can extend this to filter by more than one property. `arr.filter((v,i,a)=>a.findIndex(v2=>['productNo','manufacturer'].every(k=>v2[k] ===v[k]))===i)` — CaseyC, Apr 21 '22 at 16:47

score 1 · Accepted Answer · answered Apr 22 '22 at 23:25

Depending on your use case you can remove duplicate objects from your array in several ways.

1.) Filter duplicates by one object property

const uniqueData = moduleData.filter((v,i,a) => a.findIndex(v2 => (v2.productNo === v.productNo)) === i)

2.) Filter by several properties

const uniqueData = moduleData.filter((v,i,a) => a.findIndex(v2 => ['productNo','manufacturer'].every(k => v2[k] === v[k])) === i)

3.) Check all properties and filter duplicates (if you know the quality of your data this is likely overkill)

const uniqueData = moduleData.filter((v,i,a) => a.findIndex(v2 => [...Object.keys(moduleData[0])].every(k => v2[k] === v[k])) === i)

In the second half of your solution instead of looking again for duplicates within the list after you create an array of object properties for your dropdown values the way you are, you can easily filter out duplicates by simply using the Set constructor and the spread syntax

doubleUniquePropList = [...new Set(uniquePropList)]

score 0 · Answer 2 · answered Apr 21 '22 at 00:13

So, I found the solution:

It seemed to make the difference to stringify the object and then compare. And I took the advice of jFriend00 and removed duplicates first before filtering.

The solution:

// filtered Array =======================================
    if ((prop.length > 0 || prop != undefined || prop != null) && (prop.length == 0 || prop == undefined || prop == null)){ 
        console.log('filtered Array - (unique, no prop) ')
        
        //Make data unique
        const uniqueArray = data.filter((object,index) => index === data.findIndex(obj => JSON.stringify(obj) === JSON.stringify(object)));
            
        return uniqueArray
    }

... and that made it not all that difficult then to provide filtered Lists as well. (for dropdowns.)

// filtered List =======================================
    if ((prop.length > 0 || prop != undefined || prop != null) && (prop.length !== 0 || prop.length !== undefined || prop.length !== null)){ 
        console.log('filtered List - (unique, prop) ')

        //Make data unique
        const uniqueArray = data.filter((object,index) => index === data.findIndex(obj => JSON.stringify(obj) === JSON.stringify(object)));

        //filter the list for matches
        let UniqueArray = uniqueArray.filter(p => 
            filters.every(f=> Object.keys(f).every(k => p[k] === f[k]))
        )

        //map Array into the desired list
        let uniquePropList = UniqueArray.map(item => item[prop])

        //look again for duplicates within the list
        var unique = []
        let doubleUniquePropList = uniquePropList.filter(element => {
            const isDuplicate = unique.includes(element);

            if (!isDuplicate) {
                unique.push(element)
        
                return true
            }
        })

        return doubleUniquePropList
    }

Now I can use this function for many different use cases! Thank you guys for your help! and I hope this can help someone else!

This is a fragile solution. `JSON.stringify({a: 1, b: 2}) === JSON.stringify({b: 2, a: 1})` is false. So this will fail if the object keys are ever in a different order. You might want to sort the keys first to make sure they are in the same order. — CaseyC, Apr 21 '22 at 16:45
@CaseyC Thank you for the tip! I spent the morning trying to get this to work. I think one got it but I wanted to run it by you to make sure this solves the issue: — Michael Martell, Apr 21 '22 at 20:00
```const orderedPropArray = [] data.forEach(o => { orderedPropArray.push(Object.keys(o).sort().reduce( function(obj, key) { obj[key] = o[key] return obj }, {} )) }) //Make data unique const uniqueArray = orderedPropArray.filter((object,index) => index === orderedPropArray.findIndex(obj => JSON.stringify(obj) === JSON.stringify(object))); return uniqueArray }``` @CaseyC — Michael Martell, Apr 21 '22 at 20:03
I suppose that if a duplicate object has 1 extra key and value than it's duplicate, that both would be considered !===? @CaseyC — Michael Martell, Apr 21 '22 at 20:06
That looks like it would work. I'd still probably just use what I suggested above. You can extend this to filter duplicates by more than one property as needed. Just add the properties you'd like to compare in the array. In this case it's checking `productNo` and `manufacturer` `arr.filter((v,i,a)=>a.findIndex(v2=>['productNo','manufacturer'].every(k=>v2[k] ===v[k]))===i)` You could get really crazy and check all the keys if you want by using `Object.entries()` but it's likely overkill. `arr.filter((v,i,a)=>a.findIndex(v2=>[...Object.keys(array[0])].every(k=>v2[k] ===v[k]))===i)` — CaseyC, Apr 22 '22 at 23:03

Code works (mostly). But failure to remove duplicates

2 Answers2