outofmemoryexception: threads using too much virtual memory, about 12GB

Question

actually i have to create lots of threads to send pcap file using UDP protocol, when thread completely sends the pcap file it then sleep for some time. when i sleep thread to 420 seconds virtual memory gets full after creating more than 3100 threads and program throws a OutOfMemoryException.

i searched internet about this problem but found that a thread takes only 1MB to create and pcap file is just 60KB, and my 3100 threads are consuming more than 12GB(1.06*3100<12GB). on the other hand physical memory is not used more than 200MB. i have to create more than 5000 threads at the same time

what am i doing wrong? can anyone help me?

thanks

my code:

public static void send_pcap_file_with_single_port()
    {
        string callID = Call_ID;
        try
        {
            //CREATING CONNECTION HERE
            using (FileStream stream = new FileStream("g711a.pcap", FileMode.Open, FileAccess.ReadWrite, FileShare.ReadWrite))
            {


                for (Pos = 0; Pos < (StreamBytes - ChunkSize); Pos += ChunkSize)
                {
                    //creating RTP_header here

                    stream.Read(RTP_payload, 0, ChunkSize);
                    //combining both the byte arrays
                    System.Buffer.BlockCopy(RTP_header, 0, Bytes_to_send, 0, RTP_header.Length);
                    System.Buffer.BlockCopy(RTP_payload, 0, Bytes_to_send, 16, RTP_payload.Length);

                    RTPpacket_queue.Enqueue(Bytes_to_send);
                    //RTP_handler.Send(Bytes_to_send, Bytes_to_send.Length, remote_EP);
                }

                //done processing here
                stream.Close();
                stream.Dispose();

                RTP_count++;
                GC.Collect();
            }
            System.Threading.Thread.Sleep(420000);


        }
        catch (Exception e)
        {
            //using (StreamWriter sw = new StreamWriter(stream_logFile))
            //{
            //    sw.WriteLine(e.ToString());
            //}
            //send_BYE_message_toSIPp(client, "BYE", 5060, 2, callID);
            Console.WriteLine(e.ToString());
        }
    }

creating threads here:

Thread RTP_sender = new Thread(new ThreadStart(send_pcap_file_with_single_port));
                            RTP_sender.Start();

Answer 1

In simple terms you exhaust you garbage collector by creating objects in long term pile (objects that survive longer then few seconds). Fix would be to free and recreate thread when it is needed.

In any case by default i5 has 2 cores, if you have 3 or more threads than they are running them on same cpu. Running 3000+ of them means 1500 each, it is not a problem unless they try to write in same place (in case they start blocking like hell).

Answer 2

To demonstrate you do not need 5000 permanent threads to accomplish something like this, I have created a sample program.

The program does not do much really but what it does do is that it creates 5000 objects, each of which creates a thread when it needs to do its work. There isn't any actual work being done other than simply sleeping for a random interval but still.

Just run the program, leave it running for a while and keep an eye on its memory use. You will see it is very much manageable while still actually doing work on 5000 objects.

You will probably need to actually be creative in applying this approach to your situation but you could do something along the lines of what I am doing.

namespace Test
{
    using System;
    using System.Collections.Generic;
    using System.ComponentModel;
    using System.Diagnostics;
    using System.Threading;

    public static class MainClass
    {
        public static Random sleeper = new Random ();

        public static void Main (string[] args)
        {
            Stopwatch timer = new Stopwatch ();
            List<WorkerClass> workload = new List<WorkerClass> ();

            // Create a workload of 5000 objects
            for (int i = 0; i < 5000; i++) {
                workload.Add (new WorkerClass ());
            }

            int fires = 0;
            // Start processing the workload
            while (true) {
                // We'll measure the time it took to go through the entire workload
                // to illustrate that it does not take all that long.
                timer.Restart ();
                foreach (WorkerClass w in workload) {
                    // for each of the worker objects in the entire workload
                    // we decrease its internal counter by 1.
                    // Because after the loop is done, we sleep for 1 secondd
                    // that amounts to reducing the counter by 1 every second.
                    w.counter--;
                    if (w.counter == 0) {
                        fires++;
                        // Once the counter hits 0, do the work.
                        w.DoWork ();
                    }
                }
                timer.Stop ();
                Console.WriteLine ("Processing the entire workload of {0} objects took {1} milliseconds, {2} workers actually fired.", workload.Count, timer.ElapsedMilliseconds, fires);
                fires = 0;
                Thread.Sleep (1000);
            }
        }
    }

    public class WorkerClass
    {
        public int counter = 0;

        public WorkerClass ()
        {
            // When the worker is created, set its internal counter
            // to a random value between 5 and 10.
            // This is to mimic sleeping it for a random interval.
            // Also see the primary loop in MainClass.Main
            this.counter = MainClass.sleeper.Next (5, 10);
        }

        public void DoWork ()
        {
            // Whenever we do the work, we'll create a background worker thread
            // that actually does the work.
            BackgroundWorker work = new BackgroundWorker ();
            work.RunWorkerCompleted += (object sender, RunWorkerCompletedEventArgs e) => {
                // This simulates going back to sleep for a random interval, see
                // the main loop in MainClass.Main
                this.counter = MainClass.sleeper.Next (5, 10);
            };
            work.DoWork += (object sender, DoWorkEventArgs e) => {
                // Simulate working by sleeping a random interval
                Thread.Sleep (MainClass.sleeper.Next (2000, 5000));
            };
            // And now we actually do the work.
            work.RunWorkerAsync ();
        }
    }
}