Exclude duplicates( by content) in wallking trough directory - java

Question

Here is the code I wrote, with some help of course. But there is some bugs in the logic part I cant spot. I am pretty new to programming, and i little help wouldn't mind

public class Directories {

 public static void main(String[] args) {
    Path currentDir = Paths.get("/root"); // some directory
    displayDirectoryContents(currentDir);
}

public static void displayDirectoryContents(Path dir) {

    final List<Path> duplicates = new ArrayList<Path>();
    final List<Path> uniqueFiles = new ArrayList<Path>();   
    try {   
        final DirectoryStream<Path> stream = Files.newDirectoryStream(dir);
        for(Path entry : stream){
            if(Files.isDirectory(entry)){
                displayDirectoryContents(entry);
            } else {
                for(final Path alreadySeen : uniqueFiles){
                    if(isDuplicated(entry, alreadySeen)){
                        duplicates.add(entry);

                    } else {
                        uniqueFiles.add(entry);

                    }   
                }
            }
        }

    } catch (Exception e) {
        e.printStackTrace();
    }
}
private static final boolean isDuplicated(final Path first, final Path second){
    try{
        return Files.size(first) == Files.size(second) && 
                Arrays.equals(Files.readAllBytes(first), Files.readAllBytes(second));
    } catch (IOException e) {
        e.printStackTrace();
    }
    return false;

}

}

I would really appreciate some help. Thank you

Answer 1

Here is a solution based on Java 8, using Files.find():

public static void listDups(final Path baseDir)
{
    final BiPredicate<Path, BasicFileAttributes> filesOnly
        = (path, attrs) -> attrs.isRegularFile();

    final List<Path> uniqueFiles = new ArrayList<>();
    final List<Path> dups = new ArrayList<>();

    try (
        final Stream<Path> stream = Files.find(baseDir, filesOnly);
    ) {
        stream.forEach(path -> {
            final boolean alreadyFound = uniqueFiles.stream()
                .anyMatch(found -> sameContent(path, found));
            final List<Path> list = alreadyFound ? dups : uniqueFiles;
            list.add(path);
        }
    }

    return dups;
}

private static final sameContent(final Path first, final Path second)
    throws IOException
{
    return Files.size(first) == Files.size(second)
        && Arrays.equals(Files.readAllBytes(first), Files.readAllBytes(second));
}

Not ideal however; you might want to replace the Arrays.equals() with sequential reading from input streams from both files.

But that is a proof of concept.

Answer 2

The questions should be arisen are:

1. How do you want to check for "duplicating by content"? Do you mean comparing them byte-by-byte? What if there are two files of 10Gb length? What if there are plenty of such files?
2. Suppose two files are equal by content. Which one do you want to include to the list?

In this answer I supposed the following details:

1. md5 is used to check files for similarity. In this question you can see how one can get it in java.

2. you don't care what the file is missed; you just need to exclude any duplicate. (it's done by (f1, f2) -> f1) merge function in code)

   static byte[] md5(Path file) {
       try {
           MessageDigest digest = MessageDigest.getInstance("MD5");
           int read;
           byte[] buffer = new byte[4096];
           try (InputStream is = new FileInputStream(file.toFile())) {
               while ((read = is.read(buffer)) > 0) {
                   digest.update(buffer, 0, read);
               }
           }
           return digest.digest();
       } catch (IOException | NoSuchAlgorithmException ex) {
           //handle it or
           throw new RuntimeException(ex);
       }
   }
   public static void main(String[] args) throws IOException {
       System.out.println("first attempt:");
       Files.list(Paths.get("/tmp/t")).forEach(System.out::println);
       System.out.println("second attempt:");
       Files.list(Paths.get("/tmp/t"))
           .collect(Collectors.toMap(f -> new BigInteger(md5(f)), f -> f, (f1, f2) -> f1))
           .values()
           .forEach(System.out::println);
   }
   

Description: lets list all files we want to check and calculate md5 sum for each of them. Then put all pairs (md5, file) into map. Map will only keep one value(file) for one key(md5) by map definition. md5 values of files are equal if files are equal by content. Situation where two different(by content) files will have the same md5 values is almost impossible. So resulting map values would be unique files.

I created folder /tmp/t/ and files in it: 1 and 3 are equal but 2 is different. Output:

first attempt:
/tmp/t/2
/tmp/t/1
/tmp/t/3
second attempt:
/tmp/t/1
/tmp/t/2

The code I posted here only lists single directory contents. You can extend it to your use case using Files.walkFileTree or similar approach.