0

We are planning to create C# load test framework, where we can make concurrent calls to our IO based API (Cache / Database / File), thus test them under load and following are the possible options:

  1. Use Async-Await, with the scheduler hint TaskCreationOptions.LongRunning, thus ensuring that we have background process for each request initiated and thus genuinely test our API under load. Also will help us in aggregating the final results in an efficient manner as we would get the details of each Task returned post Async call, something similar to:

    void Main()
{

   List<Task<long>> taskList = new List<Task<long>>();

   for (int i = 0; i < 500; i++)
   {
      taskList.Add(TestLoadAsync());
   }

   Task.WaitAll(taskList.ToArray());

   long averageTime  = taskList.Average(t => t.Result);

}

public static async Task<long> TestLoadAsync()
{
   // Returns the total time taken using Stop Watch in the same module
      return await Task.Factory.StartNew(() => // Call IO API,
                                       TaskCreationOptions.LongRunning);
}

  2. Second option is using System.Diagnostics Process.Startas shown underneath:

    void Main()
{   

   for (int i = 0; i < 500; i++)
  {
    System.Diagnostics.Process.Start("LoadAPI.exe");
  }
}

"LoadAPI.exe"

public class LoadAPI
{ 
  public static void Main()
  {
     LoadAPI api = new LoadAPI();
     api.TestLoadAPI();

     // Register the details of stop watch time in database, for further           
        analysis             
  }

  public void TestLoadAPI()
  {
      // Call IO API
  }
}

Important set of questions which need to be answered are:

  • Which is a better approach and Why ?
  • Does the standard load testing frameworks use a similar strategy ?

I am posting the question to understand the various viewpoints and have a in depth discussion in my organisation, in my understanding we shall use the:

  • Async-Await as that's the natural mechanism for IO based concurrent calls
  • In using Process.Start we are possibly loading the OS by creating a costly process, which would be starving each other and not the best mechanism for IO calls
  • Standard load testing frameworks use the background processes as created by Async -Await
  • Windows as an OS, which is a Multi threaded architecture, invoking multiple processes using process.start is not the correct way to load test

Please share your view to have a better insight on the topic

Mrinal Kamboj
  • 11,300
  • 5
  • 40
  • 74
  • Personally I would use Parallel.Foreach, with MaxDegrees of Parallelism - http://stackoverflow.com/a/9290531/852806 or Option 1. Op1 1 appears to run one .exe so it would find cross threading issues within your application, as opposed to loading several instances which might not find that and might be more indicative of your machines power. You need to be careful as there is often a physical limit to the number of threads that run at one time per instance (http://stackoverflow.com/a/4828359/852806). You might find trying to run 500 threads at one time is slower than batches of 50. – JsAndDotNet Aug 08 '16 at 09:49
  • Thanks for the details, 500 is just an indicative number, mostly it would be in line with machine's logical cores we will run simultaneously on multiple machines to achieve the target like 500. Reason why Async-Await is chosen, is because API are IO bound, for it foreground threads via Parallel API may not be the best option – Mrinal Kamboj Aug 08 '16 at 09:59
  • Looks like it doesn't matter how you will start your work: with `Task.Run`, `Task.Factory.StartNew`, `Parallel.Foreach` or instantiating new thread or creating a new process. As I understood correctly, the only thing you need to perform is to call some API and then wait for a result. In this case, cost of creating of a new process will be insignificantly small comparing to the time you will spend on waiting for the result. If your tests and the app you are going to test are not on the same machine, you can use any approach, preferably simplest one. – cassandrad Aug 08 '16 at 12:17
  • @cassandrad I would tend to disagree, since Async Await is specially made for Concurrent IO calls, which is not the use case for Parallel API or the exclusive process, being resource heavy, the different is foreground and background threads as i understand. In fact as Async Await use IO completion ports, which is not thread dependant – Mrinal Kamboj Aug 08 '16 at 13:53
  • Async/Await is made to simplify asynchronous programming in general, not only IO. You can use it with CPU-heavy computations as well. What I'm saying is: it doesn't matter how long it will take to start a routine, because the only thing that all the routines do is waiting. You can't affect waiting with only creation of a new process. The difference between foreground and background threads is in termination of the app, again it will not affect your tests. The only reason to use async\await is — it's just simplest way to write code. – cassandrad Aug 08 '16 at 14:04
  • @cassandrad Its not that you cannot use the parallel API for IO operations, but it is a sheer wastage of resources, since IO works on IO completion port and doesn't need a thread. Consider this, both Parallel and Async-Await are meant for concurrency, but they need to do something different for MS to introduce them, else they will not just come up with another technology. What's the difference between Parallelism and Asynchronous operations. There are enough articles by Stephen Cleary, Stephen Toub, which will validate the usage of Async-Await for only IO operations – Mrinal Kamboj Aug 08 '16 at 18:10
  • I once built a load test tool, based on two components - "loader" and "sampler". The loader would fire and forget requests in a constant rate, say 100 r/s. The sampler would then sample the loaded system in a lower rate, and tunes the loader's rate according to the measured times. By that you could easily say that "my system responds within X milliseconds under a max load of Y r/s". You could also measure the saturation of your system that way. – shay__ Aug 10 '16 at 19:42

0 Answers0