How to check for duplicate data sets in JSON payload using JavaScript?

Question

I would like to find duplicate data sets from below payload using combination of 'NAME', 'ID'. If a set exist more than 3 times, I need to return NAME, ID of duplicated data set.

{
"Test": "1",
"value": [
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "EFGH",
        "ID": "5678",
        "ACTIVE": "true"
    },
    {
        "NAME": "EFGH",
        "ID": "5678",
        "ACTIVE": "true"
    },
    {
        "NAME": "EFGH",
        "ID": "5678",
        "ACTIVE": "true"
    },
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "IJKL",
        "ID": "91011",
        "ACTIVE": "true"
        }
    ]
}

Expected output:

["ABCD:1234", "EFGH:5678"]

It would probably be good to include an example of what you expect the final result to be. — Adam Harte, Oct 01 '21 at 04:32

userDuR · Answer 1 · 2021-10-02T00:58:02.880

Try this. You can improve this further by performance wise, if you work around a bit.

  var data = {
 "Test": "1",
 "value": [
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "EFGH",
        "ID": "5678",
        "ACTIVE": "true"
    },
    {
        "NAME": "IJKL",
        "ID": "91011",
        "ACTIVE": "true"
    }
   ]
 };
 var objArray = data.value;
 var duplicates = []; // duplicates will be stored here.
 for(var i=0, iLen = objArray.length; i<iLen;i++){
    var obj = objArray[i];
        var filtered = objArray.filter(function(arrVal) {
        return arrVal.NAME === obj.NAME && arrVal.ID === obj.ID ;
    });
    var dup = obj.NAME + ":" + obj.ID;
    if(filtered.length>=3 && duplicates.indexOf(dup) < 0) {
        duplicates.push(dup);
    }

 }

thanks for the snippet. i have update the original question, is there is way to achieve it by making changes to original snippet from you? — Rick, Oct 01 '21 at 16:15

score 0 · Answer 2 · answered Oct 01 '21 at 04:41

0

this questuon was answered multiple times on SO. You create array from the json and compare the elements of the array.

How to remove all duplicates from an array of objects?

answered Oct 01 '21 at 04:41

fra

186
1
12

My goal is to find the duplicate set. – Rick Oct 01 '21 at 04:42
yes, so you create array of objects from the json and compare the objects of that array to find the duplicates. There are millions of ways to do that and on that link you find at least 10 of them. – fra Oct 01 '21 at 04:47
I am not good with coding. Looking if someone provide me a snippet. – Rick Oct 01 '21 at 04:59

score 0 · Answer 3 · answered Oct 02 '21 at 01:24

According to your sample output I believe it's "at least 3 times" rather than "more than 3 times".

Below snippet can product the expected output with the sample data.

const data = {
"Test": "1",
"value": [
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "EFGH",
        "ID": "5678",
        "ACTIVE": "true"
    },
    {
        "NAME": "EFGH",
        "ID": "5678",
        "ACTIVE": "true"
    },
    {
        "NAME": "EFGH",
        "ID": "5678",
        "ACTIVE": "true"
    },
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "ABCD",
        "ID": "1234",
        "ACTIVE": "true"
    },
    {
        "NAME": "IJKL",
        "ID": "91011",
        "ACTIVE": "true"
        }
    ]
};

const occurrence = {}; // key: count
const result = [];
for (const item of data.value) {
  const key = `${item.NAME}:${item.ID}`;
  occurrence[key] = (occurrence[key] || 0) + 1;
  if (occurrence[key] >= 3) {
    result.push(key);
  }
}

console.log(result);

How to check for duplicate data sets in JSON payload using JavaScript?

3 Answers3