I have heard that file system performance (on a NTFS partition) can start decreasing if the number of files in a single directory becomes very huge (eg : >= 10.000.000 items). Is it true ?
If true, what is the recommended maximum number of files in a single directory ?
EDIT:
About performance: I'm thinking about file operations inside that folder (read, write, create, delete) that could possibly get slow.
I answer my own question : Yes, it's definitely slower.
I wrote a C# Console Application that creates many empty files in a folder and then randomly access them.
Here is the results :
10 files in a folder : ~26000 operation/sec
1.000.000 files a in folder : ~6000 operation/sec
Here is source code :
List<string> files = new List<string>();
Console.WriteLine("creating files...");
for (int i = 0; i < 1000 * 1000; i++)
{
string filename = @"C:\test\" + Guid.NewGuid().ToString();
using (File.Create(filename));
files.Add(filename);
}
Console.WriteLine("benchmark...");
Random r = new Random();
Stopwatch sw = new Stopwatch();
sw.Start();
int count = 0;
while (sw.ElapsedMilliseconds < 5000)
{
string filename = files[r.Next(files.Count)];
string text = System.IO.File.ReadAllText(filename);
count++;
}
Console.WriteLine("{0} operation/sec ", count / 5);
If you read this, then you should get a pretty good understanding of how NTFS works indexing files and folders.
Locally it shouldn't be much of a hazel indexing files and folders, if you follow the guidelines in the link above, but it will need alot of maintenance with that many files.
On a network it will be another story. It will be slow, this is from my own expirience at work, where we have folders with thousand of folders and that takes some time to index over a network.
Another thing to probably increase with that many files is to disable short-naming:, which will stop windows from creating a second file directory entry which will follow the 8.3 convention (MS-DOS file-naming convention) and decrease the time for folders to enumerate, because it doesn't have to look up the short-names associated with their long-names when enumerating.
Run in the Start menucmd and when you see the command-prompt, then right-click on it and select Run as administratorIf you want to enable it again, type fsutil behavior set disable8dot3 0
