0

I have an issue when BlockingCollection<MyItem> is taking too much time to wake up from .Take() call.

The scenario is this: I have a thread that pushing data to BlockingCollection very fast (actually in XUnit i did a for loop. I have a 3 Task that just sitting in .Take() call and waiting to items to be added. From output I can see that almost 200 items (up to 1 second or more) was added to collection before the first Task wake up and actually collect data from BlockingCollection.

I have multiple "buffers" based on BlockingCollection organized in pipe-line fashion and all of them suffer from too-much-time-waking on .Take() action.

I tried .TryTake() and .GetConsumingEnumerable() with the same results.

The idea that in the end of this pipeline I have a one threaded, slow function that process items one by one and it could take unknown time to process single item. I just need to make sure that 'item' passed from "buffer" to "buffer" very fast (as soon as it get inserted in the first "buffer")

I just need to make sure that start-up time (in .Take() or .TryTake() etc..) will happen close to time when item added to collection

Alex F
  • 3,180
  • 2
  • 28
  • 40
  • 3
    Post complete, executable repro code. I suspect that you can't and will find the bug in the process. – usr May 25 '15 at 10:03
  • @usr eee.... Where I should look for a bug ? in `.Take()` implementation ? I didn't say that application crash or something and I didn't say that it's `not working`' - it does work and everything passed, processed, etc... ok. I'm asking for design pattern for fast producer - slow consumer – Alex F May 25 '15 at 10:07
  • How many active threads do you (try to) have? – H H May 25 '15 at 10:12
  • 3
    @Jasper my point is that there is no obvious mistake you are making. It is necessary to look at the code to find the mistake. – usr May 25 '15 at 10:14
  • @HenkHolterman Hmmm.. though question. Currently I see about 65 threads on `vs.executionengine.x86` (xunit test running app) - so I guess almost all of them are mine and I guess I need good half of them alive. I'm not sure how to answer.... – Alex F May 25 '15 at 10:22
  • Your requirement is strange... Why do you care about _from "buffer" to "buffer" very fast_ at all? That part shouldn't matter. And with too many threads on too few cores they'll just have to wait their turn. – H H May 25 '15 at 12:57
  • @HenkHolterman Sorry - it was misleading -> I correct the question – Alex F May 25 '15 at 13:58
  • What do you mean by `start-up` time? – VMAtm May 25 '15 at 15:47
  • 65 threads?!?! If you have many more threads alive than lets say the (number of cores) *(2~4) then you have a serious design issue! – AK_ May 25 '15 at 16:34
  • @VMAtm - As I sad in the question. It taking tooo long for `BlockingCollection` to wake-up and actually understand that it does have items in collection and actually `.Take()` them. I'm pushing up to 200 items into collection just before the very first `.Take()` unblock and collect itme – Alex F May 26 '15 at 06:33
  • @AK_ I don't know how many threads from there are mine and what are they status :( The number is just final include-all-from-everywhere number. I don't know how to understand it :( – Alex F May 26 '15 at 06:35
  • 1
    Well i guess your issues are coming from thread contention and context switching. http://stackoverflow.com/questions/1970345/what-is-thread-contention – AK_ May 26 '15 at 11:37
  • @AK_ I guess you right ! Please post it as answer to question so I could 'mark it as answer' – Alex F May 26 '15 at 11:58
  • 1
    Perhaps threading is not the right solution in this situation. Something asynchronous based on Task might be a better approach. Or, since you have a sequence of data that you want to react to immediately, why not consider using the Reactive Extensions for .NET (Rx)? – Tim Long May 26 '15 at 16:21
  • @TimLong You are right. My solution more and more mimic the Rx. I even derive from `IObservable<>` and `IObserver<>` because this is exactly the data-flow that I need but I'm still struggling to understand Rx and actually here I run into problem of creating the streams... May be you have a link to GitHub project that using Rx so I could learn a little bit more about it ? – Alex F May 27 '15 at 07:15
  • 1
    @Jasper I'm not sure if this will help, but I have just pushed this today... https://bitbucket.org/tigranetworks/ta.ascom.reactivecommunications – Tim Long May 27 '15 at 23:15

0 Answers0