Lock; literal images 'r' us

One of the functions that immediately caught my eye was LdrLockLoaderLock. I’d previously spent quite a few frustrating hours trying to figure out how to determine whether some code was being executed from DllMain, i.e. while in the loader lock, so I could avoid doing anything dodgy – or indeed, anything at all.

The case I was looking at was some logging library code that was used, amongst other things, to record the fact that DLLs were being unloaded. Unfortunately when this was called from DllMain, it sometimes caused a deadlock, for all the reasons we already know about. The library code was called from lots of DLLs, so it wasn’t feasible to fix all of the call sites, instead I had to make the logging a no-op when it’s not safe.

I’m embarrassed to say that my previous attempt to detect the lock involved some pretty heinous hackery. I worked out the memory address (the offset within ntdll.dll) where the loader lock critical section is located, cast that bit of memory to a CRITICAL_SECTION and tested it. I even had to provide the ability to change the offset based on the version of ntdll being used, in a vain attempt to reduce its fragility. Ouch. It was very nasty, and to be honest although it worked in the cases where I tested it, I was reluctant to release it.

Luckily, along comes LdrLockLoaderLock to save my blushes. It appears to give me exactly the functionality I need; you can pass a flag to tell it to return immediately if the lock’s already been taken, and there’s a status parameter that can be used to tell if you got the lock – whereupon you can call the corresponding LdrUnlockLoaderLock. Nice!

I wonder if this is what’s used by the Visual Studio loader lock managed debugging assistant to determine if your managed code is being run under the loader lock?

What is .NET type equivalence?

Described at a high level here, .NET 4.0 type equivalence essentially gives you a way of indicating that different .NET types represent the same underlying COM type and is most commonly used in COM interop scenarios. One of the reasons for its introduction is to save developers from having to ship large interop DLLs with their software, e.g. the multi-megabyte Microsoft.Office.Interop. Instead the compiler can inline the definition of any types used, and mark them appropriately as representing the original COM types.

The error

We noticed that whenever we built and ran an application that referenced a DLL using .NET 2.0, it worked. Doing the same thing with .NET 4.0 caused a BadImageFormatException.

Unhandled Exception: System.BadImageFormatException: Could not load file or assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies. An attempt was made to load a program with an incorrect format.
at X.Main()


Let’s dig!

So, the BadImageFormatException doesn’t actually tell us much. Let’s break out WinDbg and see what we can find. Running the faulting app we can see several C++ exceptions before the CLR exception is thrown:

(178c.790): C++ EH exception - code e06d7363 (first chance)
...
(178c.790): C++ EH exception - code e06d7363 (first chance)
(178c.790): CLR exception - code e0434352 (first chance)

I changed the exception handling settings to stop on C++ exceptions (sxe eh) then ran again to see where things were going wrong. It stopped here:

0:000> kp
ChildEBP RetAddr
0012d15c 79084c0f KERNEL32!RaiseException+0x53
0012d194 793371be MSVCR100_CLR0400!_CxxThrowException+0x48
0012d5e4 79455cae clr!EEFileLoadException::Throw+0x1a8
0012d634 794558d2 clr!CompareTypeTokens+0x200
0012d6b0 791b5c00 clr!IsTypeDefEquivalent+0x102
0012d6d4 791b2ca8 clr!MethodTableBuilder::CheckForTypeEquivalence+0x94
0012d7ac 791b27c9 clr!MethodTableBuilder::BuildMethodTableThrowing+0x60d
0012d9a4 791a4578 clr!ClassLoader::CreateTypeHandleForTypeDefThrowing+0x88e

Interesting. Notice how the call stack contains some .NET 4.0 specific methods relating to the new type equivalence feature. We’re hitting a new code path, which is consistent with the fact that running against a down-level CLR works.

After a bit more toing-and-froing, I discovered that the C++ exception is thrown when clr!MDInternalRO::IsValidToken returns an error. By disassembling the function we can see it’s just looking at various bits in the token value, and it decides that the value passed (0×02000000) isn’t valid. Looking at the output from ildasm that token doesn’t appear anywhere. And if we add a dump of the value, we can see that it indeed doesn’t look like the other tokens:

0:000> bu clr!MDInternalRO::IsValidToken "dd esp+8 L1; g"
...
0012f5a8  02000001
0012f31c  06000001
0012f2c0  02000002
0012f0f4  02000002
0012ebe4  01000001
0012e944  23000001
...
0012d5f4  02000000
(18ec.1ec8): C++ EH exception - code e06d7363 (first chance)

What’s the culprit?

So it looks pretty conclusive; the DLL contains something that the CLR isn’t expecting. But what? It’s time to break out the oldest tool in the troubleshooting box: the binary chop!

Eventually I got the referenced DLL down to only a single simple construct. Can you guess what it is? A global literal value. A real global value, one that isn’t even part of a type. Crazy huh? In IL it looks like this:

.field public static literal valuetype Test.MyEnum LiteralValue = int32(0x00000001)

It’s a literal value of an enumerated type. That’s important: using a value of a simple type (say int32) does not provoke the error.

Now, I wasn’t even sure that this is a valid IL construct, but according to the ECMA IL spec, specifically partition II, section 15, it is:

The CLI also supports global fields, which are fields declared outside of any type definition. Global fields shall be static.

So it looks like we’re not doing anything illegal, backed up by the fact that the .NET 2.0 CLR can make use of it without a problem.

Interestingly, there’s another aspect that influences whether this code path is hit. As mentioned above, type equivalence is intended for use with interop libraries. As such, it only kicks in if your referenced assembly is marked with the PrimaryInteropAssembly attribute, e.g.:


.custom instance void [mscorlib]System.Runtime.InteropServices.PrimaryInteropAssemblyAttribute::.ctor(int32,int32) = ( 01 00 01 00 00 00 00 00 00 00 00 00 )


The Fix?

The issue is currently with Microsoft product support. Let’s see what they come up with; is it too esoteric for a hotfix…?

The Repro

Here’s some code and instructions on how to repro the problem.

1. Build the IL into a DLL using ilasm.
   "c:\WINNT\Microsoft.NET\Framework\v2.0.50727\ilasm.exe" /dll Test.il /output=Test.dll
2. Build the application into a .NET 4.0 EXE that references the DLL
   "c:\winnt\Microsoft.NET\Framework\v4.0.30319\csc.exe" TestConsumer.cs /reference:Test.dll
3. Run the resulting TestConsumer.exe application and you’ll get the exception

Test.il

.assembly extern mscorlib
{
.publickeytoken = (B7 7A 5C 56 19 34 E0 89 )
.ver 2:0:0:0
}
.assembly Test
{
.custom instance void [mscorlib]System.Runtime.InteropServices.PrimaryInteropAssemblyAttribute::.ctor(int32,int32) = ( 01 00 01 00 00 00 00 00 00 00 00 00 )
.hash algorithm 0x00008004
.ver 1:0:0:0
}
.module Test.dll
.imagebase 0x00400000
.file alignment 0x00000200
.stackreserve 0x00100000
.subsystem 0x0003
.corflags 0x00000001

.class public auto ansi sealed Test.MyEnum
extends [mscorlib]System.Enum
{
.field public specialname rtspecialname int32 value__
.field public static literal valuetype Test.MyEnum Zero = int32(0x00000000)
.field public static literal valuetype Test.MyEnum One = int32(0x00000001)
}

TestConsumer.cs

class X
{
    static void Main()
    {
        var v = Test.MyEnum.Zero;
    }
}

Collecting stack traces

One of the most helpful things the heap manager can do for you when investigating memory issues is to capture stack traces for each heap allocation. You can enable the “collect stack traces” heap flag using the gflags GUI or from within WinDbg:

0:006> !gflag +ust
Current NtGlobalFlag contents: 0x00001040
    hpc - Enable heap parameter checking
    ust - Create user mode stack trace database

This means that for each heap block (one located at 0x0bbf7308 in this case), you can see where it was allocated by using the -a (show all information) option:

0:006> !heap -p -a 0bbf7308
    address 0bbf7308 found in
    _HEAP @ a630000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        0bbf7308 0073 0000  [07]   0bbf7310    00380 - (busy)
        Trace: 401c
        7c96d6dc ntdll!RtlDebugAllocateHeap+0x000000e1
        7c949d18 ntdll!RtlAllocateHeapSlowly+0x00000044
        7c91b298 ntdll!RtlAllocateHeap+0x00000e64
        78134333 MSVCR80!malloc+0x00000077

But the obvious problem with this is that the stack trace always stops at malloc. Something’s allocating some memory? You don’t say…

It turns out that this is a well understood and documented issue with the Microsoft VC++ runtime, variously known as msvcrt, msvcr70, msvcr71, msvcr80, msvcr90, etc. Unfortunately they’re all built using the stack frame pointer omission optimisation. Well they’re built with the -O1 (favour small code) option, which implies -Oy. This means that the fast stack-walking algorithm the heap manager uses stops at functions without a return address. The only way to get a decent trace in this situation would be to use the DbgHelp API along with the .pdb files, which would be far too slow to do at each allocation site.

“Fixing” it

So, given that the source for the runtime library ships as part of Visual Studio, maybe it would be possible to build it without the -Oy option?

My first attempt at building it failed miserably with errors like:

NMAKE : fatal error U1073: don't know how to make 'build\intel\mt_obj\startup.lib'

Luckily this excellent page helped me get past this to a point where I could actually get a DLL built.

The next stage was to modify the build scripts to use different compiler switches. This was simply a case of changing line 69 of makefile.sub from:
CFLAGS=$(CFLAGS) -O1
to:
CFLAGS=$(CFLAGS) -O1 -Oy-

I thought I may have to also modify the build scripts to output a version of the DLL with the same name as the file I was replacing, msvcr80.dll, directly, in case there were internal references to the name in embedded manifests. There’s a section at the top of the build script for choosing a name for your private version of the library, but it strongly discourages use of the “reserved” MSVC* names. Luckily it turns out not to be necessary; the DLL is constructed in such a way as to be rename-able without any ill effects. I could build _sample_.dll (the default output name) and simply copy it to the destination directory in the SxS tree (C:\WINNT\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.3053_x-ww_b80fa8ca) and rename it.

Result

Now I get the expected full stack trace (names have been changed to protect the innocent):

0:006> !heap -p -a 0bbf7308
    address 0bbf7308 found in
    _HEAP @ a630000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        0bbf7308 0073 0000  [07]   0bbf7310    00380 - (busy)
        Trace: 401c
        7c96d6dc ntdll!RtlDebugAllocateHeap+0x000000e1
        7c949d18 ntdll!RtlAllocateHeapSlowly+0x00000044
        7c91b298 ntdll!RtlAllocateHeap+0x00000e64
        78134333 MSVCR80!malloc+0x00000077
        7816207f MSVCR80!operator new+0x0000001d
        fa92336 leakydll!std::allocator,std::allocator > > >::allocate+0x00000016
        fa9879b leakydll!std::vector >::resize+0x0000005b
        ...
        ...etc...

That’ll make it much easier to work out what’s happening and who’s responsible. Of course, you should be careful with this modified version. Only use it on development machines, and make sure it doesn’t escape into the wild: with great power comes great responsibility.

But what happens if during the lifetime of that stack based COM object, it gets used from .NET? A runtime callable wrapper (RCW) will be created around the object. And this RCW expects to be able to keep the underlying object alive by incrementing its reference count. Of course, the stack-based object will soon go out of scope, and regardless of its reference count the object will be destroyed and the pointer that the RCW contains will no longer be valid. It points into the stack, so when the RCW gets cleaned-up, the CLR will call via this pointer into memory that contains garbage and you’ll get something nasty like an access violation or illegal instruction exception.

It’s fairly easy to reproduce this to see where things go wrong. It can be useful to see where the CLR blows up, and how we can identify this as the cause.

Lets start by creating a simple pseudo-COM object that implements just the bare minimum to be usable:

class MyClass : public IUnknown
{
public:
	MyClass():l(0) {}
	STDMETHOD_(ULONG, AddRef)() { return InterlockedIncrement(&l); }
	STDMETHOD_(ULONG, Release)() { return InterlockedDecrement(&l); }
	STDMETHOD(QueryInterface)(REFIID iid, void ** ppvObject)
	{
		if (iid == IID_IUnknown)
		{
			*ppvObject = this;
			AddRef();
                        return S_OK;
		}
		return E_NOINTERFACE;
	}
private:
	long l;
};

We’ll also need a COM visible .NET object that will use the object. It doesn’t actually need to be COM visible, but that’s the easiest way to access it from C++, in my opinion.

I’ve created the COM object in F#. It’s a trivial class that has a single interface, with a single method that takes the object we pass to it and prints its type. This is enough for the RCW to be created.

open System
open System.Runtime.InteropServices
 
module Module1 =
 
    [<ComVisible(true); InterfaceType(ComInterfaceType.InterfaceIsIUnknown)>]
    type public IConsumer = 
        abstract member UseObject : o:obj -> unit
 
    [<ComVisible(true); ClassInterface(ClassInterfaceType.None)>]
    type public Consumer() =
        interface IConsumer with
            member this.UseObject (o:obj) =
                Console.WriteLine (sprintf "%A" (o.GetType()))

We can compile this into a DLL, then run regasm with the /tlb switch to generate a type library (TLB):

fsc -o:obj\Debug\testStackObjectsFs.dll Module1.fs
regasm /tlb:testStackObjectsFs.tlb testStackObjectsFs.dll

That can be #imported back into our test harness:

#import "testStackObjectsFs.tlb"

Now we’re ready to put together some code that creates an instance of our object on the stack and passes it to our .NET component:

void Foo()
{
	// Create an instance of our "COM object" on the stack
	MyClass obj;
 
	// Create a managed object
	testStackObjectsFs::IConsumerPtr mgd(__uuidof(testStackObjectsFs::Consumer));
 
	// and pass our COM object to it
	mgd->UseObject(_variant_t(&obj));
}
 
int _tmain(int argc, _TCHAR* argv[])
{
	// Initialise the COM runtime, for our purposes it doesn't
	// matter which threading model we use
	CoInitializeEx(NULL, COINIT_MULTITHREADED);
 
	// Call a separate function, to ensure stack-based objects
	// are out-of-scope on return.
	Foo();
 
	// Wait for some input
	_getch();
	return 0;
}

Now, if you run this from within Visual Studio, if you’re vigilant (and you haven’t got your debugger set to stop on access violations), then you’ll notice this in the output window after the return statement executes:

...
The thread 'Win32 Thread' (0x15b0) has exited with code 11001 (0x2af9).
The thread 'Win32 Thread' (0x1110) has exited with code 0 (0x0).
First-chance exception at 0x00850a2b in testStackObjects.exe: 0xC0000005: Access violation reading location 0x00850a2b.
The thread 'DebuggerRCThread::ThreadProcStatic' (0x1534) has exited with code 0 (0x0).
The thread 'RPC Callback Thread' (0x12b8) has exited with code 0 (0x0).
...

Lets ramp up WinDbg, attach to the process (that _getch comes in useful here) and find out what’s going on in a bit more detail.

If we let the app run to the point of failure in WinDbg, we can see that the CLR was in the act of shutting down when it caused the exception:

0:002> kp
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
00dae3fc 79f4c1b5 0xe06ff8
00dae450 79f4c26c mscorwks!ReleaseTransitionHelper+0x5f
00dae494 79f4c2d0 mscorwks!SafeReleaseHelper+0x8c
00dae4c8 79faaa01 mscorwks!SafeRelease+0x2f
00dae4fc 79faa7c8 mscorwks!IUnkEntry::Free+0x68
00dae510 79faa91d mscorwks!RCW::ReleaseAllInterfaces+0x18
00dae540 79faa949 mscorwks!RCW::ReleaseAllInterfacesCallBack+0xbd
00dae570 7a0792ac mscorwks!RCW::Cleanup+0x22
00dae57c 7a079714 mscorwks!RCWCleanupList::ReleaseRCWListRaw+0x16
00dae5ac 7a0797df mscorwks!RCWCleanupList::ReleaseRCWListInCorrectCtx+0xdf
00dae5fc 79fdc140 mscorwks!RCWCleanupList::CleanupAllWrappers+0x77
00dafe90 79fdc7aa mscorwks!RCWCache::ReleaseWrappersWorker+0x103
00dafed8 79fd9f95 mscorwks!ReleaseRCWsInCaches+0x27
00dafee0 79f3c76a mscorwks!InnerCoEEShutDownCOM+0x1e
00daff14 79f92015 mscorwks!WKS::GCHeap::FinalizerThreadStart+0x1fc
00daffb4 7c80b683 mscorwks!Thread::intermediateThreadProc+0x49
00daffec 00000000 kernel32!BaseThreadStart+0x37

Essentially it’s cleaning up the currently unused RCWs – including our malformed one – and as part of doing this, it’s calling Release on the underlying COM object, via the mscorwks!SafeRelease function. SafeRelease wraps the call to potentially (and definitely, in this case) dangerous unmanaged code with various exception handlers, enabling it to silently handle access violations.

If we run the app again, and this time break while it’s waiting for the keypress, before it attempts to clean up the RCWs, then we can examine the wrapper ourselves, using the approach I set out in this post.

List all of the untyped COM object wrappers:

0:002> !dumpheap -type System.__ComObject
 Address       MT     Size
01418628 79306e60       16
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
79306e60        1           16 System.__ComObject
Total 1 objects

Use the address of the object to obtain its object header:

0:002> dd 1418628-4 L1
01418624 08000002

Use the syncblk identifier in the header to get the syncblk:

0:002> !syncblk 2
Index SyncBlock MonitorHeld Recursion Owning Thread Info  SyncBlock Owner
    2 001e4d9c            0         0 00000000     none    01418628 System.__ComObject
-----------------------------
Total           2
CCW             0
RCW             0
ComClassFactory 0
Free            0

Get the address of the RCW from the sync block:

0:008> dd 001e4d9c+1c L1
001e4db8 001e7dc8
0:008> dd 001e7dc8+c L1
001e7dd4 001de828

And dump out the relevant bits of the RCW, the vtable of the object, at offset 0×88, and the IUnknown pointer, at offset 0×64:

0:008> dds 001de828+88 L1
001de8b0 0041ac78 testStackObjects!MyClass::`vftable'
0:008> dds 001de828+64 L1
001de88c 0012fe7c

We can use !address to do a quick sanity check on the pointer and verify what we know to be the case; it’s stack memory:

0:008> !address 0012fe7c
    00030000 : 00124000 - 0000c000
                    Type     00020000 MEM_PRIVATE
                    Protect  00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageStack
                    Pid.Tid  490.13dc

If we run the app on again to the point that it fails, we can clearly see the address of the object being passed as an argument to mscorwks!IUnkEntry::Free.

So the moral of the story is; don’t pretend some arbitrary piece of stack memory is a real, reference counted COM object. You may be saving the cost of a heap allocation, but even if your app works OK today, it may not tomorrow when someone introduces a piece of .NET code somewhere in your object graph.

Bonus Extra Content

As a bonus tip, here are a couple of WinDbg breakpoints that can be used to dump each RCW as it’s created and destroyed.

bu 79faa974 "dds @ecx L23; g"
bu 79faa538 "dd @esp+20 L1; dds poi(@esp+20)+88 L1; g"


So quite often in our mixed code-base, we find that the .NET garbage collection process doesn’t kick in when we need it to. For instance, when we’ve allocated a lot of memory in the COM code that .NET isn’t aware of. Memory exhaustion has to get pretty bad for the GC to occur at any other time than during a .NET allocation, either the system-wide low-memory event has to be signalled or an OutOfMemoryException needs to be thrown. In both of these cases it’s probably too late to do anything about it.

In this case it’s extremely useful to be able to see what .NET objects are still alive, and what COM objects they’re hanging on to. Unfortunately this isn’t as easy as it might seem.

The COM object itself hides within a weakly-typed System.__ComObject or a strongly-typed managed wrapper, depending on whether rich type information is available. Furthermore, a runtime controller RCW (runtime callable wrapper) is what actually holds a pointer to the object itself, and this structure is internal to mscorwks.dll.

So how can we untangle this and, on finding a __ComObject that happens to still be alive (i.e. is not reachable in the object graph and is therefore eligible for garbage collection) identify which COM object it’s hanging on to.

First of all, let’s see how many __ComObjects are still alive. In this case, it’s only one (phew!):

0:005> !DumpHeap -type __ComObject
 Address       MT     Size
01453b74 79306e60       16
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
79306e60        1           16 System.__ComObject
Total 1 objects

And you remember the layout of .NET objects in memory, don’t you? Of course you do! The 4 bytes prior to the address displayed (01453b74) are the “object header”. The exact content of the header is apparently undocumented. Let’s see what it contains (at least on a 32-bit platform, your mileage may vary):

0:005> dd 01453b74-4 L1
01453b70  08000002

According to various sources the object header contains 2 fields; a handle and a sync block index. If the object is an RCW, the handle is always 0×08000. You can use the index with SOS’s !syncblk command to de-reference it:

0:005> !syncblk 2
Index SyncBlock MonitorHeld Recursion Owning Thread Info  SyncBlock Owner
    2 001e0fec            0         0 00000000     none    01453b74 System.__ComObject
-----------------------------
Total           3
CCW             0
RCW             1
ComClassFactory 0
Free            0

The sync block itself is an undocumented structure, but after a bit of investigation, it turns out that at offset 0×1c there is a pointer to a further structure that contains the “interop information”:

0:005> dd 001e0fec+1c L1
001e1008  001e9510

And from this, we can obtain a pointer to the RCW itself. We’re almost there!

0:005> dd 001e9510+c L1
001e951c  001e5380

The RCW is a pretty large structure, but for our purposes there are only a couple of interesting fields: the IUnknown pointer at 0×64, and the object’s virtual function table pointer at 0×88. If you use dds you can easily see any symbols associated with these pointers:

0:005> dds 01e5380+64 L1
001e53e4  00ef6c24

0:005> dds 01e5380+88 L1
001e5408  00eb9710 rcwrepro!ATL::CComObject::`vftable'

This is the salient information; we now know exactly what type of COM object we’re dealing with. This is obviously a bit fragile, given that it relies on structures from mscorwks that may well change in newer versions of the runtime (I’ll check on .NET 4 when I get a chance). It’s also a bit of a pain to go through all these steps manually in WinDbg, so I put together a simple extension DLL to do it automatically given the address of the __ComObject. I’ll upload that and blog about it soon.

It’s obvious looking at the output from VMmap or windbg’s !address command what the largest contiguous block is, e.g.

0:008> !address -summary
....
Largest free region: Base 07300000 - Size 63ed0000 (1637184 KB)

But what if you need that number in order to make a decision at run-time? For instance, to decide whether your process is in a fit state to continue, or if it should instead commit hara-kiri. In that case, you need to access the information programmatically. That’s where the immensely useful VirtualQueryEx function comes in…

VirtualQueryEx gives you information on a single page of your virtual address space at a time. Pages size are dependent on the architecture and OS, but if you just want to iterate over all of them, you don’t need to add any special handling; the function returns the size of the page in an element of the MEMORY_BASIC_INFORMATION structure, so you can simply move to the ‘next’ page regardless of size.

If you’re interested in free space, you’ll need to find all the pages that have a state of MEM_FREE (0×10000), and that’s pretty much all there is to it. By keeping track of how much space falls into a continuous range of MEM_FREE pages you can get to the number reported by VMmap and !address.

Here’s some C++ code that returns the address of the largest free contiguous block in largestFreestartlargestFree. Enjoy!

	MEMORY_BASIC_INFORMATION mbi;
	DWORD start = 0;
	bool recording = false;
	DWORD freestart = 0, largestFreestart = 0;
	__int64 free = 0, largestFree = 0;
	while (true)
	{
		SIZE_T s = VirtualQueryEx(hproc, reinterpret_cast<lpvoid>(start), &mbi, sizeof(mbi));
		if (s != sizeof(mbi))
		{
			if (GetLastError() != ERROR_INVALID_PARAMETER)
				return ReportError(GetLastError(), _T("Failed to VirtualQueryEx at %08x"), start);
			else
				break;
		}
		if (mbi.State == MEM_FREE)
		{
			if (!recording)
				freestart = start;
			free += mbi.RegionSize;
			recording = true;
		}
		else
		{
			if (recording)
			{
				if (free > largestFree)
				{
					largestFree = free;
					largestFreestart = freestart;
				}
			}
			free = 0;
			recording = false;
		}
		start += mbi.RegionSize;
	}

(I had problems with WordPress choking on this long post, so I’ve split it into 2 parts. This the first part, the second part is here).

First things first: you can download it from here.

Using VMMap

VMMap graphically displays the contents of your processes’ virtual memory, which each type of memory colour coded. This means you can quickly get an idea of how all of your valuable 2GB address space is being used.

At this point it’s worth pointing out that this article refers to the 32 bit Windows XP platform only, as that’s what I use on a day-to-day basis. Moving to a 64 bit address space makes it so large that exhaustion of a process’ address space is pretty damn unlikely, discounting your application doing something pathological.

VMMap top pane

The number in the bottom left in the largest free block of address space. It’s the same number as reported by !address -summary in WinDbg. See the post here if you want to calculate it programmatically. If this number is small (less than 64MB) you’ll probably have out of memory issues, see below for more details.

Note: Generally the working set size is NOT the important number here; it’s a common mistake that people make when looking at memory usage figures in something like Task Manager. The working set size is merely an indication of the amount of virtual memory that has recently been accessed and it’s entirely possible to make the WS size number plummet by explicitly “flushing” the working set. You can do this by, for example, minimising a GUI application. As such, this is really not the number you want to be tracking. Instead, look at the Private Bytes or Virtual Memory Size counter in perfmon.

But back to VMMap. There are various different types of allocation reported. I’ll describe some of the most interesting ones:

Image

A DLL or other executable file, as typically loaded by LoadLibraryEx. You’ll see the protection is Execute/Copy on Write; this is the standard for executables, where the OS will share the memory between processes unless they modify it – which is very rare.

You may also see the same file loaded as a mapped image. For instance, when using a COM server that contains an embedded type library resource, it will be loaded in both ways; once as an executable using LoadLibrary and once as a mapped file using LoadTypeLib. You may also notice that non-ngen’d .NET images are mapped twice. This appears to be a bug that Microsoft are aware of, there’s a bug logged in Connect for it.

You can expand the image entry to see the size of the image sections, the file header, code (.text), resources (.rsrc) and relocations (.reloc) for example.

Mapped file

These are typically data files loaded using CreateFileMapping. Depending on how the mapping was created (whether an explicit maximum size was specified), either the whole of the file or some small section of it will be mapped. Of course, the whole point of memory mapped files is to be able to get access to sections of a file without having to load it all, so the chances are that only small files will be mapped in their entirety.

Memory mapped files are often used to share data between processes, so you may find that things put on the clipboard may appear as mapped address space, albeit transiently. This is especially obvious in Microsoft Office apps like Excel.

Heaps

This is the meta-data associated with each heap in the process. The actual allocations from the heap are stored separately in the address space (more on that later, see Private Data).

Some people are surprised that there can be multiple heaps in a single process, but it’s actually quite common practice in unmanaged code, especially where there are specific memory management requirements. MSXML is a widely used example of this; it does it’s own garbage collection using it’s own dedicated heaps.

There is always a default heap created by the loader during process creation, and others are created explicitly using HeapCreate. The address of the heap displayed by VMMap can be used with the WinDbg !heap command to delve more deeply into it’s contents and structure. For instance, !heap -m will display it’s segment details.

Managed heap

Obviously this is only relevant for applications using .NET – managed apps themselves or unmanaged apps that implicitly load .NET assemblies in some other way, say, via COM. The number displayed is the sum of the generation 0, 1, 2 and large object heap sizes.

Thread stack

Each thread in the OS has a stack that can grow to 1MB by default (although this is configurable at link time, and controllable programmatically). It starts as a block of uncommitted reserved memory, with the used space committed at the bottom, and a guard block that is used to determine when to expand the block downwards. VMMap helpfully displays the thread ID in the stack space row.

Remember that each thread will have it’s own stack space, so spinning up threads is not free in terms of virtual memory use. See here for more details on thread stack space.

Read on here for more…

You’ll get errors like:

0:007> !locks
NTSDEXTS: Unable to resolve ntdll!RTL_CRITICAL_SECTION_DEBUG type
NTSDEXTS: Please check your symbols

The suggested solution seems to be to roll-back to version 6.10.3.233, available from here, or you can just replace the version of ntsdexts.dll in the c:\program files\debugging tools for windows (x86)\winxp directory with the one from the earlier release.

Judging by the error message, I’m guessing that the new version may work if you happen to be using a debug (checked) build of the Windows kernel, but I don’t have one around to try it with.

To spoil the ending somewhat, we already knew the precise situation that was causing the problem. This is different to the average issue, where most of the time’s spent in isolating the conditions that cause it. In this case we had an excellent positive and negative repro case.

The users were attempting to use the legacy .local approach to changing DLL probing. This was originally implemented around the time of Windows 2000, before the current side-by-side system existed, and basically short-circuited the normal DLL search order to inject the current directory as the first potential location. See the DLL redirection MSDN page for more information.

There was already a version of the assembly in question installed in the GAC, they just wanted to use the .local file to override it. The trouble was, with certain builds of the DLL (we’ll call it XYZ.dll) in the directory along with the redirection file (appname.exe.local), an fatal InvalidProgramException would be generated by the .NET execution engine. With other builds of the same DLL it worked. Mysterious.

So the first thing to do was to try and find out exactly where the exception was occurring. I ramped up WinDbg, and took a look at what exceptions were being thrown:

(42c.1e90): CLR notification exception - code e0444143 (first chance)
(42c.1e90): Unknown exception - code 02345678 (first chance)
(42c.1e90): C++ EH exception - code e06d7363 (first chance)
(42c.1e90): C++ EH exception - code e06d7363 (first chance)
(42c.1e90): C++ EH exception - code e06d7363 (first chance)
(42c.1e90): CLR exception - code e0434f4d (first chance)

The last one generated a second chance exception and the exit. I changed the exception filter set to stop on .NET notifications – which are generated when assemblies are loaded – and for C++ exceptions:

0:000> sxe clrn
0:000> sxe eh

After restarting and running on from the CLR notification (generated whenever an assembly is loaded) I got a stack trace on the C++ exception that was being generated within the CLR:

0:000> kP
ChildEBP RetAddr
0012eea4 78158e89 KERNEL32!RaiseException+0x53
0012eedc 79fce58e MSVCR80!_CxxThrowException+0x46
0012ef0c 7a015309 mscorwks!RealCOMPlusThrow+0xd8
0012f2ac 79f0e795 mscorwks!UnsafeJitFunction+0x365
0012f350 79e87f52 mscorwks!MethodDesc::MakeJitWorker+0x1c1
0012f3a8 79e8809e mscorwks!MethodDesc::DoPrestub+0x486
0012f3f8 00341f3e mscorwks!PreStubWorker+0xeb

Hmmmm, interesting, so it looks like the JIT compiler is failing. I also noticed that there was a strange exception that I didn’t recognise:

(42c.1e90): Unknown exception - code 02345678 (first chance)

You’ve got to love those “made up” numbers. I added an exception filter for it and restarted again (good job this failure was quick to manifest).

Now I got an even more precise fault location:

0:000> kP
ChildEBP RetAddr
0012ec44 790af56c KERNEL32!RaiseException+0x53
0012ec60 7909deea mscorjit!getJit+0x38
0012ec7c 79066362 mscorjit!Compiler::fgFindJumpTargets+0x33d
0012ece0 790614c6 mscorjit!Compiler::fgFindBasicBlocks+0x4d
0012ed20 79061236 mscorjit!Compiler::compCompile+0x2bf
0012edb4 7906118c mscorjit!jitNativeCode+0xb8
0012edec 79f0f9cf mscorjit!CILJit::compileMethod+0x3d
0012ee58 79f0f945 mscorwks!invokeCompileMethodHelper+0x72
0012ee9c 79f0f8da mscorwks!invokeCompileMethod+0x31
0012eef4 79f0ea33 mscorwks!CallCompileMethodWithSEHWrapper+0x84
0012f2ac 79f0e795 mscorwks!UnsafeJitFunction+0x230
0012f350 79e87f52 mscorwks!MethodDesc::MakeJitWorker+0x1c1
0012f3a8 79e8809e mscorwks!MethodDesc::DoPrestub+0x486
0012f3f8 00341f3e mscorwks!PreStubWorker+0xeb

So now its looking like something’s going wrong deep down in the JITter. After stepping through the disassembly for a while I decided to have a look at precisely what method was being compiled at this point. You can use the DumpMD command from the sos extension to examine the method, which is the first parameter to UnsafeJitFunction (you can see parameters in the stack trace by using kb):

0:000> !DumpMD 0161ad08
Method Name: XXX()
Class: 01670a40
MethodTable: 0161ad18
mdToken: 0601626b
Module: 00d3323c
IsJitted: no
m_CodeOrIL: ffffffff

Now this is suspicious; the method as listed (XXX – name changed to protect the innocent) isn’t what we were expecting. In fact, it’s completely different. Let’s see what IL is being JITted:

0:000> !DumpIL 161ad08
error decoding IL

OK, so this is completely broken metadata.

Now, given that we were using the .local, we would have expected to see the DLL being loaded from the same directory as the application, so lets take a look and see what we’ve got loaded:

0:000> !DumpDomain
--------------------------------------
System Domain: 7a3bc8b8
...snip...
Assembly: 001abe28 [C:\WINNT\assembly\GAC_MSIL\XYZ.DLL]
ClassLoader: 001aa0b0
SecurityDescriptor: 001aa018
Module Name
00d3323c C:\WINNT\assembly\GAC_MSIL\XYZ.DLL

So this looks like the culprit; we’re still loading the version from the GAC, even though there’s a .local file present. Just to verify this I set a breakpoint on CreateFileW to show me which DLLs were being loaded and by what:

0:000> bu KERNEL32!CreateFileW "dpu (@esp+4) L1"

This tells the debugger to display the unicode string that is the first parameter to CreateFileW whenever it’s hit.
There are obviously lots of DLLs loaded, even in this simple case, but eventually it led me to the point where the GAC version was being loaded:

0012cf44 001ac5a0 "C:\WINNT\assembly\GAC_MSIL\XYZ.dll"
eax=00000000 ebx=00000001 ecx=79e7f683 edx=7c90eb94 esi=001ac4e0 edi=7c80ac0f
eip=7c810760 esp=0012cf40 ebp=0012cf64 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
KERNEL32!CreateFileW:
7c810760 8bff mov edi,edi

0:000> kp L30
ChildEBP RetAddr
0012cf3c 79e90f12 KERNEL32!CreateFileW
0012cf64 79f29306 mscorwks!WszCreateFile+0x72
0012cfac 79ed27ac mscorwks!PEImage::GetFileHandle+0x60
0012cfdc 79e981ae mscorwks!PEImage::GetLayoutInternal+0xcf
0012d020 79e98e69 mscorwks!PEImage::GetLayout+0x8a
0012d0bc 79e98934 mscorwks!RuntimeOpenImageInternal+0x103
0012d104 79e988a9 mscorwks!GetAssemblyMDInternalImportEx+0x9d
0012d11c 79ec314d mscorwks!CreateMetaDataImport+0x16
0012d13c 79ec30ee mscorwks!CAssemblyManifestImport::Init+0x35
0012d164 79ed39ca mscorwks!CreateAssemblyManifestImport+0x53
0012d180 79ed3943 mscorwks!CreateAssemblyFromManifestFile+0x48
0012d3e8 79ed380f mscorwks!CheckExistsInGAC+0x179
0012d418 79ed320a mscorwks!CreateAssemblyFromCacheLookup+0x9b
...snip...

So then I waited until the exception was fired, and used !dlls to show me what other DLLs were loaded:

0:000> !dlls
...
0x00243a68: C:\Temp\interopx\XYZ.dll
Base 0x036a0000 EntryPoint 0x00000000 Size 0x0095c000
Flags 0x90404000 LoadCount 0x00000001 TlsIndex 0x00000000
LDRP_ENTRY_PROCESSED
LDRP_COR_IMAGE
...

Ouch. We’ve loaded some metadata from the c:\winnt\assembly\GAC_MSIL\XYZ.dll version, and then we’ve actually loaded the c:\temp\interopx\XYZ.dll module. It turned out that in some cases the metadata matched, and we got away with it, and at other times (for different builds) it didn’t and we hit the InvalidProgramException.

So, the moral of the story is, don’t mix .local and the .NET assembly loader. And if you do, prepare to break out your WinDbg skills.

The problem I was trying to solve was how to get more information from first chance exceptions. We have a great deal of library code that uses catch(…) blocks to prevent any exceptions – even structured ones like access violations – from escaping into the application. This is pretty important when your code is running inside an Excel session that someone may’ve been working on for hours (and which could take minutes just to re-calculate in the event of having to re-open it). Unfortunately, because the catch handlers are built into all the code, if you discover something causing access violations, say, there’s no single place that you can change to get more diagnostic information.

So, with source level changes ruled out, how else could you catch exceptions?

Well, the obvious thing that does this already is the debugger, so I decided to look at the debugging API, and see if it was possible to use that. It turns out that the implementation is pretty simple; essentially the debugger simply waits on an event in a loop, receiving further specific information about the event when it’s fired.

There are several ‘informational’ events, DLLs being loaded and unloaded etc, and, more interestingly, there’s one called EXCEPTION_ACCESS_VIOLATION. Voila!

When this event is received, we can use the additional information we’re passed, along with some thread context data, to create a minidump. This is an extremely useful mechanism for snapshotting the process in a rich enough way to provide full post-mortem debugging abilities.

Here’s some sample code to create a minidump using the with enough information to be able to (amongst other things):

• Get stack traces
• Get memory usage information (using !address -summary)

Caveat: Example code only, please excuse the hard-coded path!

The debugger loop is then wrapped up as a stand-alone application that can be passed the PID of a process to debug. Assuming we’ve got sufficient permissions, our debugger can attach to that process using DebugActiveProcess(pid).

So, here we have a relatively lightweight way of ‘watching’ another process and grabbing lots of potentially useful diagnostic information in the event that something untoward happens. Of course when using this in the wild you’d also need some means of tidying up the generated files, and potentially some means of logging that the event had occurred, but these are relatively straightforward to implement.