Best way to get path all files, subfolder in large directory?

Question

Below is my code to get all files in large directory (Framework 3.5). It run ~ 1 hour, but cannot finish, memory in Task Manager upto 1,600,000K

void ApplyAllFiles(DirectoryInfo dir)
{
    String space = "          ";
    foreach (FileInfo f in dir.GetFiles())
    {
        try
        {            
            int size = Convert.ToInt32(f.Length / 1024);
            if (size > filesize && isFileType(f.FullName) == true)
            {
                pw.WriteLine(f.LastWriteTime + space + size + space + f.FullName);
            }
        }
        catch
        { }
    }
    foreach (DirectoryInfo d in dir.GetDirectories())
    {
        try
        {
            ApplyAllFiles(d);
        }
        catch
        {

        }
    }
}

If I use Java code, just 5 min for finish and memory always < 100,000 K. I think C# code maybe has problem (getFiles & getDirectories is not good for large Directory). I hope someone can help.

I SOLVED IT BY TAKE FOREACH LOOP INSIDE TRY { }.

Did you try `Directory.GetFiles(dirPath, "*", SearchOption.AllDirectories)` ? — KozhevnikovDmitry, Sep 03 '18 at 01:30
You'll likely want to leverage `DirectoryInfo.EnumerateDirectories` and `DirectoryInfo.EnumerateFiles` here. — Parrish Husband, Sep 03 '18 at 01:33
@John even if you are going to ignore an exception it's usually a specific exception type and it's a good practice to add a comment as to why you are doing it. — Selman Genç, Sep 03 '18 at 01:34
@ParrishHusband I've used blind try/catches in WebApi code where I'm logging data for statistical purposes - if the logging fails, I don't want the web request to fail, and since it's for our analysis and doesn't affect user data - in short: it doesn't matter if it fails. — ProgrammingLlama, Sep 03 '18 at 01:34
You also probably don't need recursion for this. Look into using an explicit stack. — Parrish Husband, Sep 03 '18 at 01:43
@KozhevnikovDmitry use SearchOption.AllDirectories will get Exception when File/Folder is denied — kien vu, Sep 03 '18 at 01:43
@kienvu GetFiles will throw Out of memory if there are so many files. Use `EnumerateFiles` instead and try running the program as admin if u are getting access error. — Selman Genç, Sep 03 '18 at 01:45
@kienvu, I see. Take look on similar question here(https://stackoverflow.com/questions/14305581/method-to-get-all-files-within-folder-and-subfolders-that-will-return-a-list) and use `StringBuilder` instead of string concatenation. Enumerating of files and directories is also good advice. — KozhevnikovDmitry, Sep 03 '18 at 01:47
I use framework 3.5, Method Directory.EnumerateFiles/Directories does not exitst — kien vu, Sep 03 '18 at 09:24
I SOLVED IT BY TAKE FOREACH LOOP INSIDE TRY { }. Thank for all your comment — kien vu, Sep 03 '18 at 15:37

Parrish Husband · Answer 1 · 2018-09-03T13:29:47.717

You can create a non-recursive extension method:

public static IEnumerable<FileInfo> GetFilesDepthFirst(this DirectoryInfo root,
    string dirPattern = "*", string filePattern = "*.*")
{
    var stack = new Stack<DirectoryInfo>();
    stack.Push(root);
    while (stack.Count > 0)
    {
        var current = stack.Pop();
        IEnumerable<FileInfo> files = Enumerable.Empty<FileInfo>();
        IEnumerable<DirectoryInfo> dirs = Enumerable.Empty<DirectoryInfo>();

        try
        {
#if NET35
            dirs = current.GetDirectories(searchPattern: dirPattern);
            files = current.GetFiles(searchPattern: filePattern);
#else
            dirs = current.EnumerateDirectories(searchPattern: dirPattern);
            files = current.EnumerateFiles(searchPattern: filePattern);                  
#endif
        }
        catch (UnauthorizedAccessException) { }
        catch (PathTooLongException) { }

        foreach (FileInfo file in files)
            yield return file;

        foreach (DirectoryInfo dir in dirs)
            stack.Push(dir);
    }
}

Which you can call like:

var dInfo = new DirectoryInfo("C:\\");
var allFiles = dInfo.GetFilesDepthFirst(filePattern: "*.txt");
foreach(FileInfo file in allFiles)
{
    // build output string
}

Thank for your help, but I use framework 3.5, method Directory.EnumerateDirectories/Files does not exitst — kien vu, Sep 03 '18 at 09:26
@kienvu you can still use `.GetFiles` and `GetDirectories` in what I posted earlier, it just may be slightly less efficient. I'll update to show this option.. — Parrish Husband, Sep 03 '18 at 13:31
I solved it by take foreach loop inside try{ }. Maybe I cannot get some files path, but the code run well — kien vu, Sep 03 '18 at 15:29

score -1 · Answer 2 · answered Sep 03 '18 at 03:07

-1

As other users have stated, Directory.GetFiles is terrible. Directory.EnumerateFiles is an alternative.

Reference from here

answered Sep 03 '18 at 03:07

Gauravsa

6,330
2
21
30

Best way to get path all files, subfolder in large directory?

2 Answers2