Save 50 million records to CSV files, saving 0.2 million per file

Question

I have function that generates numbers and stores them to List<int>.
Now I must store those results into files as fast as possible.

Here is my code so far:

private void Save_Click(object sender, EventArgs e)
{
    //this is just for tests
    List<int> myResults = Enumerable.Range(1, 50000000).ToList();
    const string dir = @"D:\TESTS";

    int fileCount = 1;
    var file = Path.Combine(dir, string.Format("{0}.csv", fileCount));
    var sw = new StreamWriter(file, false);
    int i = 0;

    Stopwatch stopwatch = new Stopwatch();
    stopwatch.Start();

    foreach (int res in myResults.Shuffle())
    {
        sw.WriteLine(res);
        i++;
        if (i%200000 != 0) continue;
        fileCount++;
        sw.Close();
        file = Path.Combine(dir, string.Format("{0}.csv", fileCount));
        sw = new StreamWriter(file, false);
    }

    sw.Close();
    stopwatch.Stop();

    label3.Text = string.Format("Save time(s): {0:0.##}", stopwatch.Elapsed.TotalSeconds);
}

Shuffle is extension method taken from this answer.

public static class Extensions
{
    public static IEnumerable<T> Shuffle<T>(this IEnumerable<T> source, Random rng = null)
    {
        if (rng == null)
            rng = new Random();

        T[] elements = source.ToArray();
        for (int i = elements.Length - 1; i > 0; i--)
        {
            int swapIndex = rng.Next(i + 1);
            yield return elements[swapIndex];
            elements[swapIndex] = elements[i];
        }
        yield return elements[0];
    }
}

My problem is that save takes about 5-7 minutes on my PC and when I increase number of results to 100 millions I get OutOfMemoryException.

How can I speed thing up and eliminate that error?

Answer 1

The most problematic lines in your code are:

List<int> myResults = Enumerable.Range(1, 50000000).ToList();

and:

foreach (int res in myResults.Shuffle())

Try to avoid at all cost to create 100m objects on the heap. Instead, generate the data continuously and immediately write it to disk without keeping it in memory. Otherwise the memory management and garbage collection become the bottleneck.

And move the shuffling outside of the timed code. I'm pretty sure considerable time is eaten by the shuffling.

So currently you measure the efficiency of .NET garbage collection and the shuffling algorithm instead of what your really want to measure, namely how long it takes to write the CSV files.

Answer 2

I ran this code on my notebook, without the shuffle method, and it took 22 seconds. So I think most of the time is probably going into that method.

I'd suggest that you also don't create the data before you use it, because that will use up a lot of memory. Create an enumerating method and yield return the data row by row.

You are also doing a lot of very small IO operations. Rather do fewer larger writes, so try batch up the writes to disk. Use a StringBuilder or something similar to create larger chunks of data to write. You could also look into the BufferedWriter class.