1

I have a MailBoxProcessor, which does the following things:

  1. Main loop (type AsyncRunner: https://github.com/kkkmail/ClmFSharp/blob/master/Clm/ContGen/AsyncRun.fs#L257 – the line number may change as I keep updating the code). It generates some "models", compiles each of them into a model specific folder, spawns them as external processes, and then each model uses WCF to "inform" AsyncRunner about its progress by calling updateProgress. A model may take several days to run. Once any of the models is completed, the runner generates / spawns more. It is designed to run at 100% processor load (but with priority: ProcessPriorityClass.BelowNormal), though I can specify a smaller number of logical cores to use (some number between 1 and Environment.ProcessorCount). Currently I "async"-ed almost everything that goes inside MailBoxProcessor by using … |> Async.Start to ensure that I "never ever" block the main loop.

  2. I can "ask" the runner (using WCF) about its state by calling member this.getState () = messageLoop.PostAndReply GetState.

  3. OR I can send some commands to it (again using WCF), e.g. member this.start(), member this.stop(), …

Here is where it gets interesting. Everything works! However, if I run a "monitor", which would ask for a state by effectively calling PostAndReply (exposed as this.getState ()) in an infinite loop, the after a while it sort of hangs up. I mean that it does eventually return, but with some unpredictably large delays (like a few minutes). At that same time, I can issue commands and they do return fast while getState still has not returned.

Is it possible to make it responsive at nearly 100% load? Thanks a lot!

Konstantin Konstantinov
  • 1,363
  • 11
  • 21
  • Why use multiple processes when you have MailBoxProcessor and tasks? A MailBoxProcessor works more or less as a separate process. The problem is that it's rather basic and you have to provide DoP and backpressure yourself – Panagiotis Kanavos Jan 15 '19 at 09:56
  • You should probably look at [TPL Dataflow](https://learn.microsoft.com/en-us/dotnet/standard/parallel-programming/dataflow-task-parallel-library). Each processing block receives messages in its input queue and process them using the function you provide with a configurable degree of parallelism. If you want to process 3 messages concurrently you can set the DOP to 3. Typically it's 1 because dataflow blocks are expected to work in a pipeline of blocks, each one running in a separate thread. That's similar to how a pipeline of shell commands works – Panagiotis Kanavos Jan 15 '19 at 09:59
  • It's also possible to implement backpressure by setting a limit to a block's input buffers. If a target block is full, source blocks will pause processing until they can push their results – Panagiotis Kanavos Jan 15 '19 at 10:02

1 Answers1

0

I would suggest not asyncing anything(other than your spawning of processes) in your main program, since your code creates additional processes. Your main loop is waiting on the loop return to continue before processing the GetState() method.