-5

In our project, I create lots of java threads to do some continuous work.

In most time these threads are sleeping so total cpu cost is low.

with 1168 threads , system overload should low than 0.20

But then I noticed that if I create more than 1000 thread with jvm, then I will get

fork: retry: Resource temporarily unavailable

when i wanna connect vm (I mean VM , not jvm) by ssh , this seem's a serious problem about system resource ..

And I'm very worried about what would happen then in my program..

Audrius Meškauskas
  • 20,936
  • 12
  • 75
  • 93
jackalope
  • 1,554
  • 3
  • 17
  • 37
  • 3
    You are just running out of available threads/processes. *Why on earth do you need over 1000 threads?* – thkala Jan 26 '13 at 16:39
  • Seems to me that you are exceeding linux kernel parameters. Most of them are very tunable. See: http://stackoverflow.com/questions/9361816/maximum-number-of-processes-in-linux – Gray Jan 26 '13 at 16:41
  • @thkala yes I have really lots of seperated work should be done in seperated thread. And as i said , lots of time , these threads is waiting for system IO – jackalope Jan 26 '13 at 16:44
  • @jackalope: What do your logs say? Which resource are you running out of? Process/thread slots? Memory? – thkala Jan 26 '13 at 16:49
  • 1
    @jackalope: Launching a bazillion threads is rarely the answer. About double the number of processors should be enough - you should use asynchronous I/O for everything else... – thkala Jan 26 '13 at 16:50
  • @thkala yes but i really don't think 1000+ thread is a big number.. – jackalope Jan 26 '13 at 16:56
  • @jackalope: depends on the memory requirements of each thread. What are your VM specifications? – thkala Jan 26 '13 at 17:13
  • 1000 threads is definitely A LOT, no matter who you talk to. Having more than about half a dozen threads would most likely suggest a poorly designed system [aside from high performance computing where you want one thread per processor core]. – Mats Petersson Jan 26 '13 at 18:30
  • @jackalope: Another thing that bothers me is that you have not mentioned how the number of threads is actually controlled. You mentioned having 1100 threads - how certain are you that when `SSH` failed you did not have 1,1000 or 110,000 threads? – thkala Jan 26 '13 at 18:36

2 Answers2

4

So many threads do not look like a good design. I propose to receive all events that currently wake up your threads in one loop that could immediately convert them into Futures or Runnables and post to some ExecutorService. This service could manage a thread pool for you. Standard approach with ServerSocket and Socket is implemented very much that way.

Audrius Meškauskas
  • 20,936
  • 12
  • 75
  • 93
  • yes i have lots of `ExecutorService` actually...... – jackalope Jan 26 '13 at 16:49
  • You only one that should handle all tasks. You can select implementation that will allocate a clever number of additional threads as required. – Audrius Meškauskas Jan 26 '13 at 16:50
  • yes that may be a good idea! – jackalope Jan 26 '13 at 16:52
  • @AudriusMeškauskas: won't the thread pool stall if the tasks are I/O bound as the OP mentioned? I am not aware of any standard Java thread pool that performs scheduling and load balancing on its own... – thkala Jan 26 '13 at 17:02
  • [ThreadPoolExecutor](http://docs.oracle.com/cd/E17802_01/j2se/j2se/1.5.0/jcp/beta1/apidiffs/java/util/concurrent/ThreadPoolExecutor.html) – Audrius Meškauskas Jan 26 '13 at 18:22
  • @AudriusMeškauskas: unless I am mistaken, `ThreadPoolExecutor` will just launch new threads when the queue bound is reached, until the thread limit is reached in which case any new task submission will block. It does not have any specific handling for I/O-blocked tasks. It would be no different than creating new threads manually. – thkala Jan 26 '13 at 18:32
  • If your IO is so slow that threads just wait for the most of time, you should redesign the protocol - split traffic into packets that could all be received and serviced in one thread. – Audrius Meškauskas Jan 26 '13 at 18:43
1

As mentioned here, the total number of threads and processes on a Linux system is limited, both directly and indirectly via other resource limits. You are apparently crossing that limit and the fork() call needed to create a new SSH session fails.

Why are you creating so many threads, anyway?

You rarely gain any performance advantage when the number of threads significantly exceeds the number of available processor cores, and each thread does have a minimum set of memory requirements that is not negligible.

If you are dealing with some form of the C10K problem, you should consider combining multiple threads with the use of asynchronous I/O.

Community
  • 1
  • 1
thkala
  • 84,049
  • 23
  • 157
  • 201
  • noticed this value in my vm equals `124413` , any suggestion?? – jackalope Jan 26 '13 at 16:44
  • @jackalope: it's not just that number. The limits imposed by the memory subsystem are probably more relevant. In any case, it seems to me that you need to fix your application - increasing the limits only delays the inevitable. Is there anything in your logs that can help you track down the culprit? – thkala Jan 26 '13 at 16:48
  • sorry but no logs found in /var/log/messages – jackalope Jan 26 '13 at 16:58