0

I'm working on the core dump from the customer. Due to that, I won't be able to share pieces of code which are causing error. Still, I hope someone will have some hints how to proceed further.

I have an application which is performing communication over sockets. It opens number of connections to multiple clients. After some time, it fails with OutOfMemoryException, even that there is plenty memory on the server and process does not exceed limits for 32 bit.

Thread pool status from core dump looks like this:

!ThreadPool
CPU utilization: 17%
Worker Thread: Total: 16 Running: 0 Idle: 16 MaxLimit: 1023 MinLimit: 16
Work Request in Queue: 0
--------------------------------------
Number of Timers: 2
--------------------------------------
Completion Port Thread:Total: 29 Free: 0 MaxFree: 32 CurrentLimit: 29 
MaxLimit: 1000 MinLimit: 16

Most of 29 Completion Port Threads throws OutOfMemoryException.

Part of !Threads output looks like this:

 0    1 1c548 00390680   2026020 Preemptive  00000000:00000000 0038a7f0 0     STA 
   2    2 2b770 0039db70     2b220 Preemptive  00000000:00000000 0038a7f0 0     MTA (Finalizer) 
   6    3 2c080 003ef588   a029220 Preemptive  00000000:00000000 0038a7f0 0     MTA (Threadpool Completion Port) 
   7    4 22ae4 004067e0   202b020 Preemptive  0F0905F0:00000000 0038a7f0 0     MTA 
  11    5 1ba94 03a79358   a029220 Preemptive  00000000:00000000 03a99928 0     MTA (Threadpool Completion Port) 
  12    6 3105c 03a7c138   a029220 Preemptive  00000000:00000000 03a99928 0     MTA (Threadpool Completion Port) System.OutOfMemoryException 016c4bc8
  13    7 250f0 03a7c958   a029220 Preemptive  0F5B50BC:00000000 03a99928 0     MTA (Threadpool Completion Port) System.OutOfMemoryException 00ef70fc
  14    8 32d9c 03a7d578   a029220 Preemptive  00000000:00000000 03a99928 0     MTA (Threadpool Completion Port) System.OutOfMemoryException 016c896c
  15    9 3281c 03a7dd98   a029220 Preemptive  00000000:00000000 03a99928 0     MTA (Threadpool Completion Port) System.OutOfMemoryException 016c2f2c
  16   10 1c360 03a7e7c0   a029220 Preemptive  00000000:00000000 03a99928 0     MTA (Threadpool Completion Port) System.OutOfMemoryException 016bc434
  17   11 240ac 03a815f0   a029220 Preemptive  00000000:00000000 03a99928 0     MTA (Threadpool Completion Port) System.OutOfMemoryException 016bb198
  18   12 250fc 03a8fcd0     21220 Preemptive  00000000:00000000 0038a7f0 0     Ukn 
  19   13 30f1c 03ab4fb0   102a220 Preemptive  00000000:00000000 0038a7f0 0     MTA (Threadpool Worker) 
  20   14 1a034 03ae9210   202b020 Preemptive  0F625BB8:00000000 03a99928 0     MTA 
  21   15 1f3bc 03ae49b8   202b220 Preemptive  00000000:00000000 03a99928 1     MTA 
  22   16 f8d8 03aba8c8   202b020 Preemptive  00000000:00000000 03a99928 0     MTA 
  23   17 32808 03b10118   a029220 Preemptive  00000000:00000000 03a99928 0     MTA (Threadpool Completion Port) System.OutOfMemoryException 016bb760
  24   18 2ee1c 03b0fbd0   a029220 Preemptive  00000000:00000000 03a99928 0     MTA (Threadpool Completion Port) System.OutOfMemoryException 016bc268
  25   19 539c 03b0ebf8   a029220 Preemptive  00000000:00000000 03a99928 0     MTA (Threadpool Completion Port) System.OutOfMemoryException 016c6288
  26   20 31be8 03b0f140   202b220 Preemptive  00000000:00000000 03a99928 1     MTA 
  27   21 2e884 03b10660   202b220 Preemptive  00000000:00000000 03a99928 1     MTA 
  28   22 10934 03b0f688   202b220 Preemptive  00000000:00000000 03a99928 1     MTA

For me it looks like a deadlock, where consumer stopped processing data coming to socket, but this is just a vague theory.

Do you have any idea on this ?

Midi
  • 459
  • 1
  • 5
  • 19
  • Although [this](https://support.microsoft.com/help/2020006/) is specific to ASP.NET, a lot of advice (like heap fragmentation) actually applies generally. What you really need is a full user mode dump (so including everything in the managed heap), and then apply the known techniques for analyzing OOM. You seem to be proficient with a debugger already, so that should not be a huge problem. Here's a nice [flowchart](https://stackoverflow.com/a/26150591/4137916) to get you started. – Jeroen Mostert Feb 12 '18 at 15:31
  • 1
    Also, obviously, do a stack trace on a failing thread first. It's possible a generic "out of resources" or "invalid argument" error code is being translated to "out of memory", even though *memory* isn't actually the resource we're running out of (but some small segment of a kernel pool somewhere, for example, or a pinned handle buffer, or an `OVERLAPPED` pool, or whatever else the designers came up with). Finding the failing call(s) would help with that. – Jeroen Mostert Feb 12 '18 at 15:33
  • 1
    A good start is to run a `!dumpheap -stat` and see how much total space is used by managed memory, **including the free space**. That last bit is important because it helps you evaluate how much memory space is lost in fragmentation (could be an issue if the application is doing a lot of LOH allocations) – Kevin Gosse Feb 13 '18 at 07:20

1 Answers1

0

You mentioned x86. Due to a old decision, x86 .NET Applications are limited to aboug 2 GiB of RAM of User-mode virtual Adress Space: https://msdn.microsoft.com/en-us/library/windows/desktop/aa366778.aspx#memory_limits

There is a switch to increase the Limits to 3 or even 4 GiB, using IMAGE_FILE_LARGE_ADDRESS_AWARE

Also there are other quirks to .NET memory management and measuring you should be aware off:

Kevin Gosse
  • 38,392
  • 3
  • 78
  • 94
Christopher
  • 9,634
  • 2
  • 17
  • 31
  • As I mentioned in the question, application process does not exceed limits for 32 bit (2 GiB). The odd thing is that OutOfMemoryException is thrown only from Completion Port Threads – Midi Feb 12 '18 at 15:29
  • @Midi it does not exceed 2 GiB, becaue it fails **trying** to exceed 2 GiB. That is what the OOM tells you: Either due to Fragmentation or the limit, you would have to exceed 2 GiB (or 3 or 4). Wich does not work. OOM is thrown instead. – Christopher Feb 12 '18 at 15:32