I must confess being pretty good at analyzing memory dumps from a performance or an exception perspective. I have written a few articles and labs about those types of problems:
- Lab 19: Debugging a high CPU hang W3WP process using WinDbg
- Lab 20: Debugging a low CPU hang W3WP process using WinDbg
- Create a memory dump of your App Service when it consumes too much memory
- Memory metrics for an Azure Web App and App Service Plan
- Creating a W3WP Memory dump
And one of my favorites here – “Must use, must know WinDbg commands, my most used”
Although I have a memory consumption lab here – “Lab 21: Debugging a W3WP process with high memory consumption”, I must also admit that these sometimes give me some trouble, I think mostly because I haven’t worked on so many memory consumption issues. I would assume this is because I work primarily with Managed code and the memory is collected and protected by the garbage collection.
So any opportunity I get to look at a memory issue, I jump on it and like always want to share my perspective.
The first thing was to run the commands I always do, I run these commands regardless of what kind of memory dump or in what context the memory dump is taken in. This helps me continue to build my perception of what ‘normal’ is.
Here are the commands I always run when I perform analysis of a memory dump.
- !sos.threadpool
- !mex.us
- !mex.aspxpagesext
- !mex.mthreads / !sos.threads
- !mex.dae
- !sos.finalizequeue
As this was a memory issue, I wanted and needed to run address –summary command. After running this, I saw something I hadn’t seen…or maybe notices before. The fact that the Stack was consuming over 51% of the memory.
--- Usage Summary ---------------- RgnCount ----------- Total Size -------- %ofBusy %ofTotal Free 114 7ff`a0b46000 ( 7.999 Tb) 99.98% Stack 2346 0`30c80000 ( 780.500 Mb) 51.19% 0.01% <unknown> 113 0`21670000 ( 534.438 Mb) 35.05% 0.01% Image 372 0`0a44d000 ( 164.301 Mb) 10.78% 0.00% Heap 72 0`02597000 ( 37.590 Mb) 2.47% 0.00% TEB 782 0`0061c000 ( 6.109 Mb) 0.40% 0.00% Other 8 0`001b9000 ( 1.723 Mb) 0.11% 0.00% PEB 1 0`00001000 ( 4.000 kb) 0.00% 0.00%
That must be one heck of Stack.
A stack is simply a textual representation of what the thread is doing. It contains the class, method and parameter list names which do consume memory, but if the stack is so big I would have expected a Stack Overflow Exception instead of what I saw above… I wrote about a StackOverflowException here – “Capture a StackOverflowException and make a dump 0xc00000fd” . And yes, there are certain types of variables stored on the stack which could consume memory, in contrast to the heap….I got that.
But hey, it is what it is and I have confidence in the output of that command (address –summary), and I found the issue when I was executing !sos.threads, !mex.us and !mex.dae.
0:000> !sos.threads ThreadCount: 4778 UnstartedThread: 0 BackgroundThread: 4777 PendingThread: 0 DeadThread: 0 Hosted Runtime: no
That is a lot of threads and if the stacks are kind of long, then sure, they could be consuming some memory.
Running !mex.us, I saw that all of the threads has the same pattern, they were trying to Dispose an object which no longer existed and the stack up to that point and after that point was indeed pretty long, longer than what I would consider ‘normal’.
Dispose(Boolean)
Then, the output of !mex.dae showed +4000 exceptions.
In Generation: 2 from .NET v4.7.2053.00 HResult: 0x80131622 Type: System.ObjectDisposedException Message: Cannot access a disposed object. Stack Trace:
So, although the issue was raised as a memory issue, it actually turned out to be a coding issue that included many Monitor.Enter() methods, try…finally… code blocks and a Task.Result() call, and most of all the Monitor.Exit() method was within an if{} statement which, in this case was never true…