Read x number of lines of a file at a time C#

Question

I want to read and process 10+ lines at a time for GB files, but haven't found a solution to spit out 10 lines until the end.

My last attempt was :

        int n = 10;
        foreach (var line in File.ReadLines("path")
            .AsParallel().WithDegreeOfParallelism(n))
        {
            System.Console.WriteLine(line);
            Thread.Sleep(1000);
        }

I've seen solutions that use buffer sizes but I want to read in the entire row.

Answer 1

The Default behavour is to read all the Line in one shot, if you want to read less than that you need to dig a little deeper into how it reads them and get a StreamReader which will then let you control the reading process

        using (StreamReader sr = new StreamReader(path)) 
        {
            while (sr.Peek() >= 0) 
            {
                Console.WriteLine(sr.ReadLine());
            }
        }

it also has a ReadLineAsync method that will return a task

if you contain these tasks in an ConcurrentBag you can very easily keep the processing running on 10 lines at a time.

var bag =new ConCurrentBag<Task>();
using (StreamReader sr = new StreamReader(path))
{
    while(sr.Peek() >=0)
    {
        if(bag.Count < 10)
        {
            Task processing = sr.ReadLineAsync().ContinueWith( (read) => {
                string s = read.Result;//EDIT Removed await to reflect Scots comment
                //process line
            });
            bag.Add(processing);
        }
        else
        {
            Task.WaitAny(bag.ToArray())
            //remove competed tasks from bag
        }
    }
}

note this code is for guidance only not to be used as is;

if all you want is the last ten lines then you can get that with the solution here How to read a text file reversely with iterator in C#

Answer 2

This method would create "pages" of lines from your file.

public static IEnumerable<string[]> ReadFileAsLinesSets(string fileName, int setLen = 10)
{
    using (var reader = new StreamReader(fileName))
        while (!reader.EndOfStream)
        {
            var set = new List<string>();
            for (var i = 0; i < setLen && !reader.EndOfStream; i++)
            {
                set.Add(reader.ReadLine());
            }
            yield return set.ToArray();
        }
}

... More fun version...

class Example
{
    static void Main(string[] args)
    {
        "YourFile.txt".ReadAsLines()
                      .AsPaged(10)
                      .Select(a=>a.ToArray()) //required or else you will get random data since "WrappedEnumerator" is not thread safe
                      .AsParallel()
                      .WithDegreeOfParallelism(10)
                      .ForAll(a =>
        {
            //Do your work here.
            Console.WriteLine(a.Aggregate(new StringBuilder(), 
                                          (sb, v) => sb.AppendFormat("{0:000000} ", v), 
                                          sb => sb.ToString()));
        });
    }
}

public static class ToolsEx
{

    public static IEnumerable<IEnumerable<T>> AsPaged<T>(this IEnumerable<T> items,
                                                              int pageLength = 10)
    {
        using (var enumerator = new WrappedEnumerator<T>(items.GetEnumerator()))
            while (!enumerator.IsDone)
                yield return enumerator.GetNextPage(pageLength);
    }

    public static IEnumerable<T> GetNextPage<T>(this IEnumerator<T> enumerator,
                                                     int pageLength = 10)
    {
        for (var i = 0; i < pageLength && enumerator.MoveNext(); i++)
            yield return enumerator.Current;
    }

    public static IEnumerable<string> ReadAsLines(this string fileName)
    {
        using (var reader = new StreamReader(fileName))
            while (!reader.EndOfStream)
                yield return reader.ReadLine();
    }
}

internal class WrappedEnumerator<T> : IEnumerator<T>
{
    public WrappedEnumerator(IEnumerator<T> enumerator)
    {
        this.InnerEnumerator = enumerator;
        this.IsDone = false;
    }

    public IEnumerator<T> InnerEnumerator { get; private set; }
    public bool IsDone { get; private set; }

    public T Current { get { return this.InnerEnumerator.Current; } }
    object System.Collections.IEnumerator.Current { get { return this.Current; } }

    public void Dispose()
    {
        this.InnerEnumerator.Dispose();
        this.IsDone = true;
    }

    public bool MoveNext()
    {
        var next = this.InnerEnumerator.MoveNext();
        this.IsDone = !next;
        return next;
    }

    public void Reset()
    {
        this.IsDone = false;
        this.InnerEnumerator.Reset();
    }
}