2

I have problem with my multithread application. When at one thread executing synchronous popen() command - other application threads are slow down significantly. Thread with popen() execute ffmpeg, that generates high load.

Normally, other threads execution time is 0.0007 ms. And when popen is used, some threads are increasing there execution time up to 14-20 seconds.

How to solve this problem?

System is FreeBSD 6.4

    FILE *pipe;
    char buff[512];
    if ( !(pipe = popen( command.c_str(), "r")) )
    { // if pipe is NULL
        return false;
    }

    while ( fgets(buff, sizeof(buff), pipe) != NULL )
    {
        ptr_output->append(buff);
    }

here is new code of popen can that does NOT help: Correct Code - Non-blocking pipe with popen

Community
  • 1
  • 1
abrahab
  • 2,430
  • 9
  • 39
  • 64

2 Answers2

2

fgets is a blocking read, so while the thread above is waiting for data to be read from the pipe, the other threads are blocked. You will want to use select/poll for with a file descriptor to see if you have data on the pipe before you issue a read. That way, you can preempt this thread, and let other threads run doing useful work.

Specksynder
  • 813
  • 9
  • 20
  • wow, thanks, are you sure that on blocking read by `fgets` all threads are waiting, not only the thread that made the call to `fgets`? this `pipe` code is used only at one thread... can you add some code examples how to solve this? – abrahab Jun 09 '12 at 12:17
  • This questions looks to be the exact problem that you are looking to solve: http://stackoverflow.com/q/149860/282531 . – Specksynder Jun 09 '12 at 12:41
  • why does fgets() block threads other than the current one? It is indeed a blocking call, but only for the current thread, right? – Brady Jun 09 '12 at 14:43
  • @Brady I am also waiting for the answer, but seems its IO call, and its related to system behavior. – abrahab Jun 09 '12 at 14:50
2

What is the relationship between the different threads? If they depend on each other, meaning they send data back and forth, then if one thread goes slower, it makes sense that the others would as well.

Something else to consider is how the thread that is executing ffmpeg affects the rest of the system. For example, if its a single-core CPU and that particular thread is generating a high CPU load, then that would leave less cycles for the rest of the threads, thus slowing them down. Granted, a change from 0.0007 ms to 14 - 20 seconds is indeed extreme!

Are there any other resources shared (stdin, mutexes, etc) between the threads that the high load thread could be abusing (holding/locking too long), thus causing starvation for the other threads?

Additionally, I would suggest profiling the application (or at least some of the threads) to see why it is so much slower. Im almost positive that you'll find out that some threads are blocked waiting for a common resource, like a mutex or something similar.

If this is linux, here are 2 Stack Overflow questions that may help:

Community
  • 1
  • 1
Brady
  • 10,207
  • 2
  • 20
  • 59
  • Thanks for you answer, yes, first of all I guess that problem is with mutexes, but I made code changes for testing purposes and here is only one mutex for 4 workers + 1 main thread that accept() connections and manage query for workers. And other thread with pipe and `ffmpeg` does not use it. And the problem is still there... (2-core cpu, not so good, but not so bad). – abrahab Jun 09 '12 at 15:15
  • idle is near 0-10% when ffmpeg is working, but other applications like nginx, apach and mysql does not slow down. and yes, seems the problem is not at `fgets`, I rewrite it as NONBLOCKING and its not help :( http://stackoverflow.com/questions/10962044/correct-code-non-blocking-pipe-with-popen new code of popen call.. – abrahab Jun 09 '12 at 15:25
  • 1
    @abrahab, ok, then I think the best thing you could do is profile the app and see what those threads are doing. That question referenced in my answer has several good alternatives to do so. – Brady Jun 09 '12 at 15:27
  • Application on the server, I have only unix console, does not know that profiler to use (Shiny from your link seems need desktop ). I stop the application when its slow down all threads with `gdb` and make `info threads`. One thread with `ffmpeg` is always at `read` (see new code, its analog of `fgets`). Two others at `accept()` and `pthread_testcancel()`. Seems, `read` is the problem........... :( – abrahab Jun 09 '12 at 15:53
  • 1
    Try the valgrind drd tool from the question "How to profile pthread mutex in linux". Also where is the ffmpeg thread reading from? – Brady Jun 09 '12 at 16:01
  • bad.. but I can not http://stackoverflow.com/questions/10861384/cpp-gdb-valgrind-memory-usage-statistic-while-app-is-running-and-valgrind-unha ... "problem on problem....". `ffmpeg` thread reading from pipe on the while curcle http://stackoverflow.com/questions/10962044/correct-code-non-blocking-pipe-with-popen and when this code on read - got something like hanged... – abrahab Jun 09 '12 at 16:09
  • 1
    @abrahab, why would the threads be at `pthread_testcancel()`? Did you call that? If so, why? Regarding the Shiny profiler, etc, you could consider dumping the profiling info to a file. – Brady Jun 09 '12 at 20:47
  • no, I does not.. seems, its special thread make this call... I add the thread info to the first post. And interesting, I created only 4 workers + 1 signal thread + 1 maintenance thread + 1 ffmpeg + 1 main (accept) thread = 8 thread but application have 9 – abrahab Jun 09 '12 at 22:03
  • seems, the problem was solved!.. and its really was `fgets-read` block issue that described by Specksynder below! When 5 hour ago, I was trying Non-Blocking version of `fgets`, I miss one line of code and `fgets` continuing block. Check: http://stackoverflow.com/questions/10962044/correct-code-non-blocking-pipe-with-popen and comments of the question. Now, not 100% sure, but seems its help (i can not repeat the bug, but need some more time) – abrahab Jun 09 '12 at 22:14