Collecting stack traces

One of the most helpful things the heap manager can do for you when investigating memory issues is to capture stack traces for each heap allocation. You can enable the “collect stack traces” heap flag using the gflags GUI or from within WinDbg:

0:006> !gflag +ust
Current NtGlobalFlag contents: 0x00001040
    hpc - Enable heap parameter checking
    ust - Create user mode stack trace database

This means that for each heap block (one located at 0x0bbf7308 in this case), you can see where it was allocated by using the -a (show all information) option:

0:006> !heap -p -a 0bbf7308
    address 0bbf7308 found in
    _HEAP @ a630000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        0bbf7308 0073 0000  [07]   0bbf7310    00380 - (busy)
        Trace: 401c
        7c96d6dc ntdll!RtlDebugAllocateHeap+0x000000e1
        7c949d18 ntdll!RtlAllocateHeapSlowly+0x00000044
        7c91b298 ntdll!RtlAllocateHeap+0x00000e64
        78134333 MSVCR80!malloc+0x00000077

But the obvious problem with this is that the stack trace always stops at malloc. Something’s allocating some memory? You don’t say…

It turns out that this is a well understood and documented issue with the Microsoft VC++ runtime, variously known as msvcrt, msvcr70, msvcr71, msvcr80, msvcr90, etc. Unfortunately they’re all built using the stack frame pointer omission optimisation. Well they’re built with the -O1 (favour small code) option, which implies -Oy. This means that the fast stack-walking algorithm the heap manager uses stops at functions without a return address. The only way to get a decent trace in this situation would be to use the DbgHelp API along with the .pdb files, which would be far too slow to do at each allocation site.

“Fixing” it

So, given that the source for the runtime library ships as part of Visual Studio, maybe it would be possible to build it without the -Oy option?

My first attempt at building it failed miserably with errors like:

NMAKE : fatal error U1073: don't know how to make 'build\intel\mt_obj\startup.lib'

Luckily this excellent page helped me get past this to a point where I could actually get a DLL built.

The next stage was to modify the build scripts to use different compiler switches. This was simply a case of changing line 69 of makefile.sub from:
CFLAGS=$(CFLAGS) -O1
to:
CFLAGS=$(CFLAGS) -O1 -Oy-

I thought I may have to also modify the build scripts to output a version of the DLL with the same name as the file I was replacing, msvcr80.dll, directly, in case there were internal references to the name in embedded manifests. There’s a section at the top of the build script for choosing a name for your private version of the library, but it strongly discourages use of the “reserved” MSVC* names. Luckily it turns out not to be necessary; the DLL is constructed in such a way as to be rename-able without any ill effects. I could build _sample_.dll (the default output name) and simply copy it to the destination directory in the SxS tree (C:\WINNT\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.3053_x-ww_b80fa8ca) and rename it.

Result

Now I get the expected full stack trace (names have been changed to protect the innocent):

0:006> !heap -p -a 0bbf7308
    address 0bbf7308 found in
    _HEAP @ a630000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        0bbf7308 0073 0000  [07]   0bbf7310    00380 - (busy)
        Trace: 401c
        7c96d6dc ntdll!RtlDebugAllocateHeap+0x000000e1
        7c949d18 ntdll!RtlAllocateHeapSlowly+0x00000044
        7c91b298 ntdll!RtlAllocateHeap+0x00000e64
        78134333 MSVCR80!malloc+0x00000077
        7816207f MSVCR80!operator new+0x0000001d
        fa92336 leakydll!std::allocator,std::allocator > > >::allocate+0x00000016
        fa9879b leakydll!std::vector >::resize+0x0000005b
        ...
        ...etc...

That’ll make it much easier to work out what’s happening and who’s responsible. Of course, you should be careful with this modified version. Only use it on development machines, and make sure it doesn’t escape into the wild: with great power comes great responsibility.

You can do it without the mouse, too. Just make sure the window is selected in the task bar (perhaps by using the Windows key and then tabbing to the appropraite icon), then hit Alt-Space, Alt-M, and then using the cursor keys.

The only problem is, some application developers choose to change the system menu – the menu visible from clicking the app’s icon in the top left, or by right-clicking the task bar icon – perhaps deciding for some insane reason not to include the standard Move option (as an aside, this breaks one of the fundamental tenets of usability: don’t change – or in this case, remove – existing, established behaviour). If this is the case, you can instead use a handful of lines of Windows API code to access the window and move it programatically. Here’s the entire code of a program that will do exactly that:

#include <windows.h>
#include <tchar.h>
 
int _tmain(int argc, TCHAR *argv[])
{
	if (argc < 2)
	{
		_tprintf(_T("usage: mover windowtitle\n"));
		return -1;
	}
	HWND hwnd = FindWindow(NULL, argv[1]);
	if (hwnd)
	{
		SetWindowPos(hwnd, HWND_TOP, 0, 0, 400,400, 0);
	}
	else
		_tprintf(_T("Unable to find window %s\n"), argv[1]);
	return 0;
}

Compile it up and you can use it as a handy little utility to move arbitrary top level windows, e.g. if you’ve got regedit running:

mover "Registry Editor"

The only tricky thing is finding the exact title of the window to use. Well, you wouldn’t want it to be too easy, would you?

But what happens if during the lifetime of that stack based COM object, it gets used from .NET? A runtime callable wrapper (RCW) will be created around the object. And this RCW expects to be able to keep the underlying object alive by incrementing its reference count. Of course, the stack-based object will soon go out of scope, and regardless of its reference count the object will be destroyed and the pointer that the RCW contains will no longer be valid. It points into the stack, so when the RCW gets cleaned-up, the CLR will call via this pointer into memory that contains garbage and you’ll get something nasty like an access violation or illegal instruction exception.

It’s fairly easy to reproduce this to see where things go wrong. It can be useful to see where the CLR blows up, and how we can identify this as the cause.

Lets start by creating a simple pseudo-COM object that implements just the bare minimum to be usable:

class MyClass : public IUnknown
{
public:
	MyClass():l(0) {}
	STDMETHOD_(ULONG, AddRef)() { return InterlockedIncrement(&l); }
	STDMETHOD_(ULONG, Release)() { return InterlockedDecrement(&l); }
	STDMETHOD(QueryInterface)(REFIID iid, void ** ppvObject)
	{
		if (iid == IID_IUnknown)
		{
			*ppvObject = this;
			AddRef();
                        return S_OK;
		}
		return E_NOINTERFACE;
	}
private:
	long l;
};

We’ll also need a COM visible .NET object that will use the object. It doesn’t actually need to be COM visible, but that’s the easiest way to access it from C++, in my opinion.

I’ve created the COM object in F#. It’s a trivial class that has a single interface, with a single method that takes the object we pass to it and prints its type. This is enough for the RCW to be created.

open System
open System.Runtime.InteropServices
 
module Module1 =
 
    [<ComVisible(true); InterfaceType(ComInterfaceType.InterfaceIsIUnknown)>]
    type public IConsumer = 
        abstract member UseObject : o:obj -> unit
 
    [<ComVisible(true); ClassInterface(ClassInterfaceType.None)>]
    type public Consumer() =
        interface IConsumer with
            member this.UseObject (o:obj) =
                Console.WriteLine (sprintf "%A" (o.GetType()))

We can compile this into a DLL, then run regasm with the /tlb switch to generate a type library (TLB):

fsc -o:obj\Debug\testStackObjectsFs.dll Module1.fs
regasm /tlb:testStackObjectsFs.tlb testStackObjectsFs.dll

That can be #imported back into our test harness:

#import "testStackObjectsFs.tlb"

Now we’re ready to put together some code that creates an instance of our object on the stack and passes it to our .NET component:

void Foo()
{
	// Create an instance of our "COM object" on the stack
	MyClass obj;
 
	// Create a managed object
	testStackObjectsFs::IConsumerPtr mgd(__uuidof(testStackObjectsFs::Consumer));
 
	// and pass our COM object to it
	mgd->UseObject(_variant_t(&obj));
}
 
int _tmain(int argc, _TCHAR* argv[])
{
	// Initialise the COM runtime, for our purposes it doesn't
	// matter which threading model we use
	CoInitializeEx(NULL, COINIT_MULTITHREADED);
 
	// Call a separate function, to ensure stack-based objects
	// are out-of-scope on return.
	Foo();
 
	// Wait for some input
	_getch();
	return 0;
}

Now, if you run this from within Visual Studio, if you’re vigilant (and you haven’t got your debugger set to stop on access violations), then you’ll notice this in the output window after the return statement executes:

...
The thread 'Win32 Thread' (0x15b0) has exited with code 11001 (0x2af9).
The thread 'Win32 Thread' (0x1110) has exited with code 0 (0x0).
First-chance exception at 0x00850a2b in testStackObjects.exe: 0xC0000005: Access violation reading location 0x00850a2b.
The thread 'DebuggerRCThread::ThreadProcStatic' (0x1534) has exited with code 0 (0x0).
The thread 'RPC Callback Thread' (0x12b8) has exited with code 0 (0x0).
...

Lets ramp up WinDbg, attach to the process (that _getch comes in useful here) and find out what’s going on in a bit more detail.

If we let the app run to the point of failure in WinDbg, we can see that the CLR was in the act of shutting down when it caused the exception:

0:002> kp
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
00dae3fc 79f4c1b5 0xe06ff8
00dae450 79f4c26c mscorwks!ReleaseTransitionHelper+0x5f
00dae494 79f4c2d0 mscorwks!SafeReleaseHelper+0x8c
00dae4c8 79faaa01 mscorwks!SafeRelease+0x2f
00dae4fc 79faa7c8 mscorwks!IUnkEntry::Free+0x68
00dae510 79faa91d mscorwks!RCW::ReleaseAllInterfaces+0x18
00dae540 79faa949 mscorwks!RCW::ReleaseAllInterfacesCallBack+0xbd
00dae570 7a0792ac mscorwks!RCW::Cleanup+0x22
00dae57c 7a079714 mscorwks!RCWCleanupList::ReleaseRCWListRaw+0x16
00dae5ac 7a0797df mscorwks!RCWCleanupList::ReleaseRCWListInCorrectCtx+0xdf
00dae5fc 79fdc140 mscorwks!RCWCleanupList::CleanupAllWrappers+0x77
00dafe90 79fdc7aa mscorwks!RCWCache::ReleaseWrappersWorker+0x103
00dafed8 79fd9f95 mscorwks!ReleaseRCWsInCaches+0x27
00dafee0 79f3c76a mscorwks!InnerCoEEShutDownCOM+0x1e
00daff14 79f92015 mscorwks!WKS::GCHeap::FinalizerThreadStart+0x1fc
00daffb4 7c80b683 mscorwks!Thread::intermediateThreadProc+0x49
00daffec 00000000 kernel32!BaseThreadStart+0x37

Essentially it’s cleaning up the currently unused RCWs – including our malformed one – and as part of doing this, it’s calling Release on the underlying COM object, via the mscorwks!SafeRelease function. SafeRelease wraps the call to potentially (and definitely, in this case) dangerous unmanaged code with various exception handlers, enabling it to silently handle access violations.

If we run the app again, and this time break while it’s waiting for the keypress, before it attempts to clean up the RCWs, then we can examine the wrapper ourselves, using the approach I set out in this post.

List all of the untyped COM object wrappers:

0:002> !dumpheap -type System.__ComObject
 Address       MT     Size
01418628 79306e60       16
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
79306e60        1           16 System.__ComObject
Total 1 objects

Use the address of the object to obtain its object header:

0:002> dd 1418628-4 L1
01418624 08000002

Use the syncblk identifier in the header to get the syncblk:

0:002> !syncblk 2
Index SyncBlock MonitorHeld Recursion Owning Thread Info  SyncBlock Owner
    2 001e4d9c            0         0 00000000     none    01418628 System.__ComObject
-----------------------------
Total           2
CCW             0
RCW             0
ComClassFactory 0
Free            0

Get the address of the RCW from the sync block:

0:008> dd 001e4d9c+1c L1
001e4db8 001e7dc8
0:008> dd 001e7dc8+c L1
001e7dd4 001de828

And dump out the relevant bits of the RCW, the vtable of the object, at offset 0×88, and the IUnknown pointer, at offset 0×64:

0:008> dds 001de828+88 L1
001de8b0 0041ac78 testStackObjects!MyClass::`vftable'
0:008> dds 001de828+64 L1
001de88c 0012fe7c

We can use !address to do a quick sanity check on the pointer and verify what we know to be the case; it’s stack memory:

0:008> !address 0012fe7c
    00030000 : 00124000 - 0000c000
                    Type     00020000 MEM_PRIVATE
                    Protect  00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageStack
                    Pid.Tid  490.13dc

If we run the app on again to the point that it fails, we can clearly see the address of the object being passed as an argument to mscorwks!IUnkEntry::Free.

So the moral of the story is; don’t pretend some arbitrary piece of stack memory is a real, reference counted COM object. You may be saving the cost of a heap allocation, but even if your app works OK today, it may not tomorrow when someone introduces a piece of .NET code somewhere in your object graph.

Bonus Extra Content

As a bonus tip, here are a couple of WinDbg breakpoints that can be used to dump each RCW as it’s created and destroyed.

bu 79faa974 "dds @ecx L23; g"
bu 79faa538 "dd @esp+20 L1; dds poi(@esp+20)+88 L1; g"


This is such a common source of bugs, and such an important requirement, that from Vista onwards Microsoft introduced a new set of functions in the Windows API explicitly to support it: One-Time Initialization.

And even if you get away with doing naughty things in DllMain now, don’t think that it’ll stay that way forever. We got away with it for years, then when .NET came along it introduced all sorts of additional correctness checks. For instance, the Managed Debugging Assistant (MDA) in Visual Studio will shout loudly should you attempt to run managed code during DllMain.

Managed Debugging Assistant 'LoaderLock' has detected a problem in 'C:\YourApp.vshost.exe'.
Additional Information: Attempting managed execution inside OS Loader lock. Do not attempt to run managed code inside a DllMain or image initialization function since doing so can cause the application to hang.

And it’s easier than you think to do so. For example, calling something as innocuous as GetWindowText can result in managed code being run.

How can you get around this?

One of the approaches I’ve used has been to make use of Win32 asynchronous procedure calls. Specifically you can call QueueUserAPC to add a function to the queue, and this can contain the initialisation you would’ve otherwise done in DllMain.

However there is significant gotcha regarding use of APCs: your function will not be called until the thread is in an “alertable wait state”. This means you (or some other code on the thread) need to call an alertable wait function such as SleepEx, WaitForSingleObjectEx and specify TRUE for the alertable parameter.

Once your APC is getting called successfully you’ll be in much better place; your code will be executed outside of the scope of the dreaded OS loader lock and you’ll be doing things by the book, hopefully avoiding all the potential pitfalls that lie in wait within DllMain.

But one of the things I couldn’t get over from that article is the verbosity. I’ve just got too used to the succintness of F#. I know this is only example code, so it’s intended to demonstrate usage, but the whole point of lambdas is to reduce boilerplate and make the intent of your code more obvious. So, let’s compare the C++0x implementation with the F# version.

Here’s the C++ version, with the comments removed (including the “increment the counter” comment from the original article… Oh, please):

#include <algorithm>
#include <iostream>
#include <vector>
using namespace std;
 
int main()
{
   vector<int> v;
   for (int i = 0; i < 10; ++i)
   {
      v.push_back(i);
   }
 
   int evenCount = 0;
   for_each(v.begin(), v.end(), [&evenCount] (int n) {
      cout << n;
 
      if (n % 2 == 0)
      {
         cout << " is even " << endl;
 
         evenCount++;
      }
      else
      {
         cout << " is odd " << endl;
      }
   });
 
   cout << "There are " << evenCount << " even numbers in the vector." << endl;
}

As well as the verbosity, notice the weird syntax used to capture the evenCount variable.

Without further ado, here’s an F# version – not a definitive one necessarily – that returns the value rather than doing nasty side-effecting IO by printing it out to the screen:

seq { 0..9 } |> Seq.filter (fun e -> e % 2 = 0) |> Seq.length

If you want to make it a bit more like the original code, you can add back the printfs, and lay it out a bit more logically:

let count =
    seq { 0..9 }
    |> Seq.fold (fun count e ->
        match e % 2 with
        | 0 -> printf "%d is even\n" e; count + 1
        | _ -> printf "%d is odd\n" e; count) 0
printf "There are %d even numbers" count

I guess the comparison isn’t really a fair one. The fundamental difference between C++ and F# lambdas is that they’re an afterthought in C++. In F# they’re a core, fundamental part of the language, and have been from day one. Functions are created, passed and bound as a matter of course.

The C++ heritage becomes obvious when you look at the finer points of its lambda syntax. For instance, having to specify whether to capture variables by reference or by value, in the “capture clause” or lambda-introducer. This is the type of distinction that both .NET programmers and functional programmers are not used to having to make.

Other nice F# features

Although not directly related to lambdas, there are some other F# language features that help make code more concise and understandable:

1. Sequence expressions

   F# has an extremely terse syntax for generating and manipulating sequences – collections that implement IEnumerable. This means you can easily generate collections and iterate over them, without the potential for off-by-one errors, the scourge of for

2. Fewer brackets

   In F#, whitespace is significant. It takes some time to get used to, but it ultimately results in cleaner code. It's been interesting to see the use of #light gradually become more and more standard as F# has neared an official release. Now it's implicit in all code. Try it, you'll like it.

3. Type-safe printf

   For some reason I always preferred the C style printf string formatting to C++s cout stream based version. There's something more readable about printf, but it suffers from a terrible lack of compile time type safety. This can be disastrous if, for instance, you happen to wrongly type the insert in an error message or other code path that's rarely hit. The one time you need it, the error handling itself generates an error! Ouch.

   
   Luckily, F# combines the best of both worlds with a compile-time type-safe printf! At first glance this can seem like some kind of magic to C++ developers (well, it did for me).

Now that F# is officially "out there" and available as a first class citizen in Visual Studio 2010, you should really take a look. I'm increasingly convinced that the cost of using C++, not necessarily the raw, low-level cost, but the business costs involved with writing performant, maintainable, manageable code in C++, is getting too much to bear. But that's probably the basis for another post.

It’s obvious looking at the output from VMmap or windbg’s !address command what the largest contiguous block is, e.g.

0:008> !address -summary
....
Largest free region: Base 07300000 - Size 63ed0000 (1637184 KB)

But what if you need that number in order to make a decision at run-time? For instance, to decide whether your process is in a fit state to continue, or if it should instead commit hara-kiri. In that case, you need to access the information programmatically. That’s where the immensely useful VirtualQueryEx function comes in…

VirtualQueryEx gives you information on a single page of your virtual address space at a time. Pages size are dependent on the architecture and OS, but if you just want to iterate over all of them, you don’t need to add any special handling; the function returns the size of the page in an element of the MEMORY_BASIC_INFORMATION structure, so you can simply move to the ‘next’ page regardless of size.

If you’re interested in free space, you’ll need to find all the pages that have a state of MEM_FREE (0×10000), and that’s pretty much all there is to it. By keeping track of how much space falls into a continuous range of MEM_FREE pages you can get to the number reported by VMmap and !address.

Here’s some C++ code that returns the address of the largest free contiguous block in largestFreestartlargestFree. Enjoy!

	MEMORY_BASIC_INFORMATION mbi;
	DWORD start = 0;
	bool recording = false;
	DWORD freestart = 0, largestFreestart = 0;
	__int64 free = 0, largestFree = 0;
	while (true)
	{
		SIZE_T s = VirtualQueryEx(hproc, reinterpret_cast<lpvoid>(start), &mbi, sizeof(mbi));
		if (s != sizeof(mbi))
		{
			if (GetLastError() != ERROR_INVALID_PARAMETER)
				return ReportError(GetLastError(), _T("Failed to VirtualQueryEx at %08x"), start);
			else
				break;
		}
		if (mbi.State == MEM_FREE)
		{
			if (!recording)
				freestart = start;
			free += mbi.RegionSize;
			recording = true;
		}
		else
		{
			if (recording)
			{
				if (free > largestFree)
				{
					largestFree = free;
					largestFreestart = freestart;
				}
			}
			free = 0;
			recording = false;
		}
		start += mbi.RegionSize;
	}

Why? Well, because it’s too easy to use them as an excuse for not defining your shared library interfaces properly.

The reason this is on my mind recently is that several hundred, yes, you heard that right, several hundred DLLs have been released by my group over the last, ooh, 10 years or so. They are all still in use. Each of them has burned into it a copy of the library that deals with interfacing with Excel. That means each of these has it’s own little internal copy of the current state-of-the-art. The problem with that is; the state-of-the-art moves on. And how do you go about updating the DLLs that are already in production? You have to re-release them. In an environment where thes DLLs are used for marking the profit and loss on a large derivatives trading book, that’s not a small undertaking. And it’s made worse if, say the DLL in question was last released with a different version of the compiler.

My approach would be to refactor this shared static library (.lib) into a stand-alone DLL.

At this point, people start saying “oh, but then you’ve got a single point of failure, if you release a broken version of that DLL, everything will stop working!”. Not exactly a compelling argument. If the functionality of the DLL is well defined, and there are well known entry points it should be easy to put together a comprehensive black-box test suite. In fact we already do that with all our other DLLs (COM servers). The fact that this shared library *isn’t* a DLL has meant that it’s fallen through the testing cracks; another good reason to refactor it.

The internal interface to the shared library is already relatively well defined. It has a set of header files that define all of the functions and classes that are consumed by others. It’s a relatively small step to compile it as a DLL, rather than a static library. The problem then becomes one of maintenance, dealing with the inevitable changes to the external interface in a backwardly compatible way.

And that’s the problem. It requires some effort. Elsewhere in our codebase we use COM as a magic cure-all for avoiding having to deal with versioning: interface immutability rules. All interfaces are public, no published interface ever changes, object identity is based purely on interfaces supported. If you haven’t got these crutches to rely on, then you have to enforce the rules yourself, which can be both logistically and technically difficult when you’re dealing with C++.

But it’s not impossible. And I really think it would be better than having hundreds of DLLs all containing subtley different versions of the same code, and being unable to change behaviour across the board without having to build, test and release them all.

Maybe you’ve got a different opinion?