Remove Duplicates from ArrayList of ArrayLists

Question

I have a problem that I am sure the solution is so easy but I am not able to find it. I have an ArrayList of smaller ArrayLists. Those lists contain elements of type String. I want to merge the smaller lists to one and then remove the duplicates. Let me be clear.

I have this:

[[USA, Maine], [USA, Maine, Kennebunk], [USA, Maine, North Berwick], 
[USA, New Hampshire], [USA, Keene, New Hampshire], [USA, Keene, New 
Hampshire, Main Street], [USA, New Hampshire, Swanzey]].

This is my main list that has smaller lists inside. I want to have a final ArrayList which is the merge of the smaller ones and get the duplicates deleted.

What I want is:

[USA, Maine, Kennebunk, North Berwick, New Hampshire , Keene, Main Street, Swanzey]

Any help is appreciated. Thank you

Answer 1

This is a concise solution using the Stream class:

listOfLists.stream().flatMap(List::stream).collect(Collectors.toSet())

Note that the result is of type Set. This takes care of removing the duplicates.

If you need a List, you can use this:

listOfLists.stream()
           .flatMap(List::stream)
           .distinct()
           .collect(Collectors.toList())

Note that this even guarantees the order of elements to be stable, i.e. [["foo","bar"],["bar","abc","foo"]] will always result in ["foo","bar","abc"] in this order. Most solutions using Set do not guarantee this because most of them are not sorted.

Answer 2

It's easy to perform with help of Sets (Set does not allow duplicate values)

public List<String> merge(List<List<String>> list) {
    Set<String> uniques = new HashSet<>();
    for(List<String> sublist : list) {
        uniques.addAll(sublist);
    }
    return new ArrayList<>(uniques);
}

p.s. when you want that you merged list will be sorted change HashSet to TreeSet like this: Set<String> uniques = new TreeSet<>();

Answer 3

Traditional solution:

Set<String> result = new LinkedHashSet<>();
for (List<String> innerList : filmingLocations) result.addAll(innerList);

As result is a LinkedHashSet, it preserves insertion order, so the order of the elements will be as in the inner lists.

You can also use an equivalent Java 8 solution:

Set<String> result = new LinkedHashSet<>();
filmingLocations.forEach(result::addAll);

Or even a Java 8 stream-based solution:

Set<String> result = filmingLocations.stream()
    .flatMap(List::stream)
    .collect(Collectors.toCollection(LinkedHashSet::new));

Answer 4

If you are targeting < Java 8, you could create instance of your final ArrayList, let's call it "resultList". Then iterate over each of your inner ArrayLists and add only these Strings for which contains() method returns false. This is the solution only if you have to use ArrayList as your final Collection. Otherwise, you should consider using HashSet, which automatically holds unique values inside and gets rid of any repeated objects. The following code might help you a bit, if you need to use ArrayList as your result Collection:

ArrayList<ArrayList<String>> sourceList = new ArrayList<>();
        // Adding sample ArrayLists ("a" and "b") of Strings to sourceList:
        ArrayList<String> a = new ArrayList<>();
        a.add("USA");
        a.add("Maine");
        sourceList.add(a);
        ArrayList<String> b = new ArrayList<>();
        b.add("USA");
        b.add("Maine");
        b.add("Kennebunk");
        sourceList.add(b);
        ArrayList<String> resultList = new ArrayList<>();
        for(ArrayList<String> outerList : sourceList) {
            for(String str : outerList) {
                // If resultList doesn't contain currently checked string...
                if(!(resultList.contains(str))) {
                    // Add this string to resultList...
                    resultList.add(str);
                }
            }
        }
        System.out.println(resultList.toString());

Output you get: [USA, Maine, Kennebunk]

Answer 5

Solution:
Loop trough ever String in you ArrayList of ArrayLists and add the string to another ArrayList if it is not already on this list using the .contains() method of ArrayList

Code:

  public ArrayList<String> merge(ArrayList<ArrayList<String>> startArrayList) {
    ArrayList<String> finalArrayList = new ArrayList<String>();
    //Iterate over each element
    for (ArrayList<String> innerList:startArrayList) {
      for (String value:innerList) {
        //add the String if it is missing
        if (!finalArrayList.contains(value))
          finalArrayList.add(value);
      }
    }
    return finalArrayList;
  }

Answer 6

I saw this post and had to answer, Berwick/Kennebunk are towns I lived in lol. Are you local?

Anyways The easiest way to do this is with set operations as mentioned above. This gurenttees some O(log n) search .

public List<String> mergeTowns (List<List<String>> list) {
    Set<String> uniques = new HashSet<>();
    for(List<String> sublist : list) {
        uniques.addAll(sublist);
    }
    return new ArrayList<>(uniques);
}

---

If your looking for a little more dynamic data structure use a map where country is you key and towns are your values. This way if you decide to build up a big data base of towns by who have different countries you can and then search the map by country to show the towns . Maybe use State instead of country as your key.

The resulting data structure will give up a map like this. when printed.

[USA = [berwick, kennebunk, north berwick, wells], CANADA = [berwick, kennebunk, north berwick, wells], MEXICO = [berwick, kennebunk, north berwick, wells]]

The way the data structure is built up prevents duplicate town entries in the same country/state.

public class Merge {


    private static ArrayList<String> mergeMap(HashMap<String, Set> map) {
        ArrayList<String> data = new ArrayList();
        for(Entry<String, Set> entries : map.entrySet()){
            String country = entries.getKey();
            Set<String> towns = entries.getValue();
            data.add(country+" = "+towns);
        }
        return data;
    }



    public static void main(String[] args) {
        //Mock data
        String[] countrys = {"USA", "CANADA", "MEXICO"};

        //Try this way of building your data structure instead of an array list of array list. 
        HashMap<String,Set> map = new HashMap<String,Set>();
        TreeSet<String> towns = new TreeSet<String>();

        // Add a couple towns to your set of towns
        towns.add("berwick");
        towns.add("north berwick");
        towns.add("kennebunk");
        towns.add("kennebunk");
        towns.add("kennebunk");
        towns.add("kennebunk");
        towns.add("wells");
        towns.add("wells");

        //With a map you could push a different set of towns to different countries
        for(String country: countrys){
            map.put(country, towns);
        }

        //Pass in your map<Country, Towns>
        ArrayList<String> mergedValues = mergeMap(map);
    }
}