C# process large list of items in a loop and retry if fails

Question

I have a large database and I access it using Entity Framework. I get all items as a List<Item>. When I process them in a loop:

List<Item> lstItems = UnitOfWork.Items.GetAllAsync(); // repository pattern and Unit Of Work patterns
foreach (Item item in lstItems)
{
  //do something with the item
}

What I need is if one of the items fail processing in the loop I want to be able to retry that action 3 times before ignoring it in this loop. Also I don't want the loop to brake, so no matter if an item processing is success or fail the loop should continue until the last item in the list.

Any idea of a pattern I can use for this purpose?

I was thinking about using try catch and if processing fails then in the catch section add it to a new List<Item> called processAgain and once the main list is finished processing then handle the processAgain list.

Answer 1

As you want to achive parallel processing, using a Task and async processing would be a good approach. Therefore, you need to define a method or expression that will be your Task/Action and includes a retry pattern internally:

public const int MAX_RETRY_COUNT = 3;

private void ProcessItemsAsync(Item item, int retryCount)
{        
    // Note: Exceptions thrown here will pop up
    // as AggregateException in Task.WaitAll()
    if (retryCount >= MAX_RETRY_COUNT)
        throw new InvalidOperationException(
            "The maximum amount of retries has been exceeded");

    retryCount++;
    // Either implement try-catch, or use conditional operator.
    try
    {
        // Do stuff with item
    }
    catch(Exception ex)
    {
        // Exception logging relevant? If not, just retry
        ProcessItemsAsync(item, retryCount);
    }
}

Once you defined your Task method, you can process a bulk of tasks at once:

public const int BULK_AMOUNT = 10;

private async void ProcessSqlData()
{
    List<Item> lstItems = await UnitOfWork.Items.GetAllAsync();
    for (var i = 0; i < lstItems.Count; i += BULK_AMOUNT)
    {
        // Running the process parallel with to many items
        // might slow down the whole process, so just take a bulk
        var bulk = lstItems.Skip(i).Take(BULK_AMOUNT).ToArray();
        var tasks = new Task[bulk.Length];
        for (var j = 0; j <= bulk.Length; j++)
        {
            // Create and start tasks, use ProcessItemsAsync as Action
            tasks[j] = Task.Factory.StartNew(() => ProcessItemsAsync(bulk[j], 0));
        }
        // Wait for the bulk to complete
        try
        {
            Task.WaitAll(tasks);
        }
        catch (AggregateException e)
        {
            Log.WriteLine(String.Format(
                "The maximum amount of retries has been exceeded in bulk #{0}. Error message: {1}",
                i,
                e.InnerException != null
                    ? e.InnerException.Message
                    : e.Message));
        }
    }
}

However, if you know that your computer running this has enough performance, you might increase the BULK_AMOUNT. You should test it, to find the optimal amount here.

Answer 2

I would suggest a recursive approach:

    static int errorCounter;

    static void Main(string[] args)
    {
        List<Item> lstItems = new List<Item>();
        foreach (var item in lstItems)
        {
            errorCounter = 0;
            bool succesful = CustomAction(item);
        }
    }

    static bool CustomAction(Item item)
    {
        try
        {
            //your custom actions with the item
        }
        catch (Exception ex)
        {

            if (errorCounter < 3)
            {
                errorCounter++;
                CustomAction(item);
            }
            else
            {
                return false;
            }
        }
        return true;
    }

EDIT: if for any reason you do not want to use a static counter, you can also initialize the counter in the Main method and pass it to CustomAction as a parameter

Answer 3

You can use Producer/Consumer pattern like here

Just create your processing logic (retry for three times) in Consume method before remove item from the queue

Answer 4

You can try to use recursion. Something like that:

List<Item> lstItems = UnitOfWork.Items.GetAllAsync(); // repository pattern and Unit Of Work patterns
foreach (Item item in lstItems)
{

    YourProcessingFunction(yourArgs, int count);
}

And in YourProcessingFunction(yourArgs, int count):

try
{
    if(count<3)
    {
       //Processing
       return someValue;
    }
    else
    {
        return;
    }
}
catch(Exception ex)
{
    return YourProcessingFunction(yourArgs, int count);
}

Answer 5

there is no need of recursion as it will make your logic complex. you can use try catch block as follows

List<Item> lstItems = UnitOfWork.Items.GetAllAsync(); // repository pattern and Unit Of Work patterns

        foreach (Item item in lstItems)
        {
          try
          {
             //do something with the item
          }
          catch(Exception ex)
          {
             for(int i = 1; i <= 3; i++)
             {
                 // do something with the item
             }
             // to continue external loop
             continue;
          }

    }

i hope it will help :)