Depth First Recursion with IAsyncEnumerable

Question

The topic of recursion with an IEnumerable was covered in this 2010 SO Post. I'm curious how the story has changed with the advent of IAsyncEnumerable<T> in .Net Core 3.X? Specifically:

1. Should I still be concerned about the resource cost of creating iterators recursively, for deep trees, as Jon Skeet warned?
2. In that same SO post, someone provided a "generic non-recursive extension method" to achieve depth first iteration without recursion. If that same approach should be used today, what does an updated version of that RecursiveSelect() extension look like for IAsyncEnumerable<T>?

My interest is performing a depth first search of a (remote) file system, where each iteration causes one of the remote files to be downloaded locally. To that end, I intend to create an asynchronous "generator" (an iterator whose length is unknown) which I can iterate with await foreach.

As a test case, I'm first simply trying to do a depth-first-search of a local directory...copying the files I find to some other arbitrary folder.

I want to be sure I'm using IAsyncEnumerable effectively.

Note: Even if creating vast numbers of IAsyncEnumerable iterators is not expensive for a deep tree, it still makes sense to use the non-recursive approach because the non-recursive approach is probably capable of "check-pointing" its progress and can "resume" if something caused the application to crash. I'm not sure the same could be done for the recursive approach.

Answer 1

The same considerations/solutions should apply to IAsyncEnumerable, you can check at sharplab.io - state machine and iterators created per method invocation with extra code for asynchronous handling (compare to the "simple" IEnumerable with yield return):

This method:

public static async IAsyncEnumerable<int> AEn()
{
    for (int i = 1; i <= 10; i++)
    {
        await Task.Delay(1000);//Simulate waiting for data to come through. 
        yield return i;
    }
}

Is translated by compiler to:

[AsyncIteratorStateMachine(typeof(<AEn>d__0))]
public static IAsyncEnumerable<int> AEn()
{
    return new <AEn>d__0(-2);
}

Where <AEn>d__0 is compiler generated implementation of IAsyncEnumerable.

So you should use the stack implementation for your depth-first-search.