Dark Days

We should bear in mind that in some camps, C++ never went away. The financial, scientific and games programming communities have bravely kept the fires burning.

Why? Primarily performance. In these industries performance is a feature, it’s a key differentiator, and it will help you make money. Can you say the same for an enterprise app (there I go, using that term disparagingly again) that just displays a form interface over a relational database?

However, even in the C++ bastions, the language was getting a bad press. I can only speak for the finance industry, where I’ve been working for the past 10 years, but even here the focus briefly switched away from raw performance to complex (overly-complex, dangerously-complex, as hindsight tells us) trades where getting to market first with something new was key: developer productivity was key. Flexibility and generality was key.

In that case, you don’t want to be dealing with the vagaries of subtle errors, crashes and memory leaks that can easily creep in when you’re using a language as low-level as C++. You just want to be able to release quickly. You probably want to be using a managed language.

In fact, C++ was getting a bad press all over. The functional power that was provided in languages like Haskell and F# – and being hastily added to C# – was missing or awkwardly implemented in C++. The provision of things like the .NET thread pool provided programmers on the CLR an easy way to schedule asynchronous tasks, and it was being touted as the only way to survive in the highly-parallel near-future. Of course, it was possible to do this stuff in C++, it was just more difficult.

Despite this, the programming community widely understood the power that C++ brings, but also that it has to be wielded carefully. Witness the oft-repeated jokes about the fact that when you shoot yourself in the foot with C++, you tend to blow your leg off. There were even tensions within Microsoft, with the pro-managed DevDiv vs the hardcore, native-favouring Windows division.

The New Order

So what’s changed? Herb Sutter’s recent talk covered many of the reasons why C++ is relevant again, and the most obvious reason is the emergence and importance of mobile devices and data centres.

They have a completely different set of requirements than the desktop: they focus on squeezing the best experience out of every available cycle (and corresponding minute of battery life), and the most performance out of every watt of data centre power per degree of cooling. We’re seeing the impact of the smartphone in your pocket and the data-centric web and social media reflecting back onto the languages we use to write software.

Managed languages fall down in some important areas here. They sacrifice memory efficiency to the god of garbage collection and they offer fewer opportunities for aggressive compiler optimisations, including, in the case of .NET, making use of chip-provided performance features like SIMD instructions.

Nowadays in more and more of the computing industry, it’s the hardware that’s driving the development choices, not the other way around.

With resource-constricted platforms, you want to be doing more of the heavy lifting offline, at compile time, not at run-time with a JIT compiler.

How the iPhone showed devs don’t care

While (parts of) Microsoft were busy pushing the managed languages hard, Apple was quietly producing game-changing pieces of hardware that the device-carrying public were buying in their hordes. And they weren’t just buying devices, they were buying apps to play on these devices – in their millions. Lots of developers wanted a piece of this action but – quelle horreur! – You had to use an obscure variant of C to create them.

But – surprise, surprise – developers did it anyway. Luckily, they weren’t on their own, but this time rather than a managed runtime, they had an API to use. And they had to use it; it was, and is, literally a condition of sale that your app uses only the APIs that Apple provide and in the way that they intended. Whatever your view on the fairness of this approach, having a restricted surface area and well defined set of libraries helps developers win back some of the productivity they might otherwise have sacrificed by using Objective-C.

Microsoft must’ve been looking askance at this, wondering how Apple had so effectively switched the model around. MS had been based on the idea of developer productivity being king and the hardware being utterly insignificant. Now developers were writing code in a language that was broadly derided by people in the programming community, yet they were in a virtuous circle where writing apps helped to ship hardware that increased the market for their apps.

We need libraries

Is it possible to find a good balance between the raw power of C++ and the productivity of managed languages? I think it is; and partly it comes down to having a good set of libraries. When I say good, I mean consistent, relevant, well-documented, easily available set of libraries. Can you see where C++ may have struggled here before?

For the low-level, programming task oriented basics, Boost is the answer. Seriously, Boost always seems to be the answer to questions of “how do I do X in C++?” where X is a modern programming technique. Smart pointers, lambdas, higher-order functions… they’re all there.

A few years ago if you were working in C++, you didn’t have many options apart from vanilla STL, Boost, and writing-it-yourself (again). Now, the STL and the language itself is richer and anecdotally, organisations are more receptive to the use of Boost. So hopefully, this combination gives us back access to most of the functionality that’s available to managed developers in their pre-packaged base class libraries.

C++ and Apps

But how is that going to help people use it to produce compelling apps and “experiences” (cough) and specifically, how is Microsoft going to use its developer productivity nous to keep Windows relevant in the new hardware markets?

They need something else; something like the Apple Core frameworks. It looks like this is what Microsoft is talking about with Windows 8 and the Windows Runtime. It’s suddenly not embarrassed to admit that, hey, Windows itself is written in C++, so maybe you shouldn’t have to go through a managed ‘interpretation’ of the existing APIs, one that’s exclusively available to managed languages.

Of course, Microsoft can’t do anything as radical as forcing all developers targeting a mobile or tablet platform to use C++. For them, programmers, rather than any particular hardware, are the bread-and-butter, so they tend to have a slightly less dictator-style relationship with devs than Apple does.

Judging from what I’ve seen from Build, Microsofts chosen approach is to provide language-specific “projections” (their term) from the Windows Runtime layer into a variety of languages, including C++. The native C++ types are mapped into corresponding concepts in the vernacular of the higher level languages; Javascript, C# etc. They’ve gone to some lengths, it seems, to ensure that transitions between levels are efficient. Using C++ at both levels should be the fast path, and of course you can mix it freely with code that, for example, uses Boost or other C++ libraries.

Additionally and importantly, as well as basic types, collections etc, the Windows Runtime also contains all of the rich APIs for accessing app-level functionality; things like image capture, sound and geolocation. So suddenly, all of this stuff is available in all its glory directly from C++ – and other languages too via cheap bridges. What they’ve essentially got is an identical copy of the architecture that exists in the iOS world, while still attempting to keep non C++ developers on-side.

The Return…?

So, have Microsoft managed to bring C++ in from the cold? They seem to have admitted, at least to themselves, that developers will need to use it in order to effectively target some platforms, and they’ve used their tooling experience to make it look and feel like a managed language in their IDEs, as well as leveraging their compiler technology to do some additional useful performance tricks. It feels like C++ was a fundamental concern in the redesign of the new alternative to the creaking Win32 API.

But will the transition be something to make current developers switch to C++, especially those who’ve been used to living in the cosseted managed world? Perhaps, if the Microsoft-based tablet market becomes as compelling – read, large – as Apple’s. Otherwise I don’t see any reason why developers who don’t have specific platform or other runtime requirements would move; for the majority of environments, developer time is still the dominant cost, so developer productivity will still be paramount.

The gap between managed and unmanaged languages has certainly narrowed in that area, but it still exists. C++ is still proving surprisingly relevant, despite its 30+ years, and hopefully its use in new and different areas of computing will feed back into improvements in the language and libraries for everyone’s benefit.

Lock; literal images 'r' us

One of the functions that immediately caught my eye was LdrLockLoaderLock. I’d previously spent quite a few frustrating hours trying to figure out how to determine whether some code was being executed from DllMain, i.e. while in the loader lock, so I could avoid doing anything dodgy – or indeed, anything at all.

The case I was looking at was some logging library code that was used, amongst other things, to record the fact that DLLs were being unloaded. Unfortunately when this was called from DllMain, it sometimes caused a deadlock, for all the reasons we already know about. The library code was called from lots of DLLs, so it wasn’t feasible to fix all of the call sites, instead I had to make the logging a no-op when it’s not safe.

I’m embarrassed to say that my previous attempt to detect the lock involved some pretty heinous hackery. I worked out the memory address (the offset within ntdll.dll) where the loader lock critical section is located, cast that bit of memory to a CRITICAL_SECTION and tested it. I even had to provide the ability to change the offset based on the version of ntdll being used, in a vain attempt to reduce its fragility. Ouch. It was very nasty, and to be honest although it worked in the cases where I tested it, I was reluctant to release it.

Luckily, along comes LdrLockLoaderLock to save my blushes. It appears to give me exactly the functionality I need; you can pass a flag to tell it to return immediately if the lock’s already been taken, and there’s a status parameter that can be used to tell if you got the lock – whereupon you can call the corresponding LdrUnlockLoaderLock. Nice!

I wonder if this is what’s used by the Visual Studio loader lock managed debugging assistant to determine if your managed code is being run under the loader lock?

What is .NET type equivalence?

Described at a high level here, .NET 4.0 type equivalence essentially gives you a way of indicating that different .NET types represent the same underlying COM type and is most commonly used in COM interop scenarios. One of the reasons for its introduction is to save developers from having to ship large interop DLLs with their software, e.g. the multi-megabyte Microsoft.Office.Interop. Instead the compiler can inline the definition of any types used, and mark them appropriately as representing the original COM types.

The error

We noticed that whenever we built and ran an application that referenced a DLL using .NET 2.0, it worked. Doing the same thing with .NET 4.0 caused a BadImageFormatException.

Unhandled Exception: System.BadImageFormatException: Could not load file or assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies. An attempt was made to load a program with an incorrect format.
at X.Main()


Let’s dig!

So, the BadImageFormatException doesn’t actually tell us much. Let’s break out WinDbg and see what we can find. Running the faulting app we can see several C++ exceptions before the CLR exception is thrown:

(178c.790): C++ EH exception - code e06d7363 (first chance)
...
(178c.790): C++ EH exception - code e06d7363 (first chance)
(178c.790): CLR exception - code e0434352 (first chance)

I changed the exception handling settings to stop on C++ exceptions (sxe eh) then ran again to see where things were going wrong. It stopped here:

0:000> kp
ChildEBP RetAddr
0012d15c 79084c0f KERNEL32!RaiseException+0x53
0012d194 793371be MSVCR100_CLR0400!_CxxThrowException+0x48
0012d5e4 79455cae clr!EEFileLoadException::Throw+0x1a8
0012d634 794558d2 clr!CompareTypeTokens+0x200
0012d6b0 791b5c00 clr!IsTypeDefEquivalent+0x102
0012d6d4 791b2ca8 clr!MethodTableBuilder::CheckForTypeEquivalence+0x94
0012d7ac 791b27c9 clr!MethodTableBuilder::BuildMethodTableThrowing+0x60d
0012d9a4 791a4578 clr!ClassLoader::CreateTypeHandleForTypeDefThrowing+0x88e

Interesting. Notice how the call stack contains some .NET 4.0 specific methods relating to the new type equivalence feature. We’re hitting a new code path, which is consistent with the fact that running against a down-level CLR works.

After a bit more toing-and-froing, I discovered that the C++ exception is thrown when clr!MDInternalRO::IsValidToken returns an error. By disassembling the function we can see it’s just looking at various bits in the token value, and it decides that the value passed (0×02000000) isn’t valid. Looking at the output from ildasm that token doesn’t appear anywhere. And if we add a dump of the value, we can see that it indeed doesn’t look like the other tokens:

0:000> bu clr!MDInternalRO::IsValidToken "dd esp+8 L1; g"
...
0012f5a8  02000001
0012f31c  06000001
0012f2c0  02000002
0012f0f4  02000002
0012ebe4  01000001
0012e944  23000001
...
0012d5f4  02000000
(18ec.1ec8): C++ EH exception - code e06d7363 (first chance)

What’s the culprit?

So it looks pretty conclusive; the DLL contains something that the CLR isn’t expecting. But what? It’s time to break out the oldest tool in the troubleshooting box: the binary chop!

Eventually I got the referenced DLL down to only a single simple construct. Can you guess what it is? A global literal value. A real global value, one that isn’t even part of a type. Crazy huh? In IL it looks like this:

.field public static literal valuetype Test.MyEnum LiteralValue = int32(0x00000001)

It’s a literal value of an enumerated type. That’s important: using a value of a simple type (say int32) does not provoke the error.

Now, I wasn’t even sure that this is a valid IL construct, but according to the ECMA IL spec, specifically partition II, section 15, it is:

The CLI also supports global fields, which are fields declared outside of any type definition. Global fields shall be static.

So it looks like we’re not doing anything illegal, backed up by the fact that the .NET 2.0 CLR can make use of it without a problem.

Interestingly, there’s another aspect that influences whether this code path is hit. As mentioned above, type equivalence is intended for use with interop libraries. As such, it only kicks in if your referenced assembly is marked with the PrimaryInteropAssembly attribute, e.g.:


.custom instance void [mscorlib]System.Runtime.InteropServices.PrimaryInteropAssemblyAttribute::.ctor(int32,int32) = ( 01 00 01 00 00 00 00 00 00 00 00 00 )


The Fix?

The issue is currently with Microsoft product support. Let’s see what they come up with; is it too esoteric for a hotfix…?

The Repro

Here’s some code and instructions on how to repro the problem.

1. Build the IL into a DLL using ilasm.
   "c:\WINNT\Microsoft.NET\Framework\v2.0.50727\ilasm.exe" /dll Test.il /output=Test.dll
2. Build the application into a .NET 4.0 EXE that references the DLL
   "c:\winnt\Microsoft.NET\Framework\v4.0.30319\csc.exe" TestConsumer.cs /reference:Test.dll
3. Run the resulting TestConsumer.exe application and you’ll get the exception

Test.il

.assembly extern mscorlib
{
.publickeytoken = (B7 7A 5C 56 19 34 E0 89 )
.ver 2:0:0:0
}
.assembly Test
{
.custom instance void [mscorlib]System.Runtime.InteropServices.PrimaryInteropAssemblyAttribute::.ctor(int32,int32) = ( 01 00 01 00 00 00 00 00 00 00 00 00 )
.hash algorithm 0x00008004
.ver 1:0:0:0
}
.module Test.dll
.imagebase 0x00400000
.file alignment 0x00000200
.stackreserve 0x00100000
.subsystem 0x0003
.corflags 0x00000001

.class public auto ansi sealed Test.MyEnum
extends [mscorlib]System.Enum
{
.field public specialname rtspecialname int32 value__
.field public static literal valuetype Test.MyEnum Zero = int32(0x00000000)
.field public static literal valuetype Test.MyEnum One = int32(0x00000001)
}

TestConsumer.cs

class X
{
    static void Main()
    {
        var v = Test.MyEnum.Zero;
    }
}

Collecting stack traces

One of the most helpful things the heap manager can do for you when investigating memory issues is to capture stack traces for each heap allocation. You can enable the “collect stack traces” heap flag using the gflags GUI or from within WinDbg:

0:006> !gflag +ust
Current NtGlobalFlag contents: 0x00001040
    hpc - Enable heap parameter checking
    ust - Create user mode stack trace database

This means that for each heap block (one located at 0x0bbf7308 in this case), you can see where it was allocated by using the -a (show all information) option:

0:006> !heap -p -a 0bbf7308
    address 0bbf7308 found in
    _HEAP @ a630000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        0bbf7308 0073 0000  [07]   0bbf7310    00380 - (busy)
        Trace: 401c
        7c96d6dc ntdll!RtlDebugAllocateHeap+0x000000e1
        7c949d18 ntdll!RtlAllocateHeapSlowly+0x00000044
        7c91b298 ntdll!RtlAllocateHeap+0x00000e64
        78134333 MSVCR80!malloc+0x00000077

But the obvious problem with this is that the stack trace always stops at malloc. Something’s allocating some memory? You don’t say…

It turns out that this is a well understood and documented issue with the Microsoft VC++ runtime, variously known as msvcrt, msvcr70, msvcr71, msvcr80, msvcr90, etc. Unfortunately they’re all built using the stack frame pointer omission optimisation. Well they’re built with the -O1 (favour small code) option, which implies -Oy. This means that the fast stack-walking algorithm the heap manager uses stops at functions without a return address. The only way to get a decent trace in this situation would be to use the DbgHelp API along with the .pdb files, which would be far too slow to do at each allocation site.

“Fixing” it

So, given that the source for the runtime library ships as part of Visual Studio, maybe it would be possible to build it without the -Oy option?

My first attempt at building it failed miserably with errors like:

NMAKE : fatal error U1073: don't know how to make 'build\intel\mt_obj\startup.lib'

Luckily this excellent page helped me get past this to a point where I could actually get a DLL built.

The next stage was to modify the build scripts to use different compiler switches. This was simply a case of changing line 69 of makefile.sub from:
CFLAGS=$(CFLAGS) -O1
to:
CFLAGS=$(CFLAGS) -O1 -Oy-

I thought I may have to also modify the build scripts to output a version of the DLL with the same name as the file I was replacing, msvcr80.dll, directly, in case there were internal references to the name in embedded manifests. There’s a section at the top of the build script for choosing a name for your private version of the library, but it strongly discourages use of the “reserved” MSVC* names. Luckily it turns out not to be necessary; the DLL is constructed in such a way as to be rename-able without any ill effects. I could build _sample_.dll (the default output name) and simply copy it to the destination directory in the SxS tree (C:\WINNT\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.3053_x-ww_b80fa8ca) and rename it.

Result

Now I get the expected full stack trace (names have been changed to protect the innocent):

0:006> !heap -p -a 0bbf7308
    address 0bbf7308 found in
    _HEAP @ a630000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        0bbf7308 0073 0000  [07]   0bbf7310    00380 - (busy)
        Trace: 401c
        7c96d6dc ntdll!RtlDebugAllocateHeap+0x000000e1
        7c949d18 ntdll!RtlAllocateHeapSlowly+0x00000044
        7c91b298 ntdll!RtlAllocateHeap+0x00000e64
        78134333 MSVCR80!malloc+0x00000077
        7816207f MSVCR80!operator new+0x0000001d
        fa92336 leakydll!std::allocator,std::allocator > > >::allocate+0x00000016
        fa9879b leakydll!std::vector >::resize+0x0000005b
        ...
        ...etc...

That’ll make it much easier to work out what’s happening and who’s responsible. Of course, you should be careful with this modified version. Only use it on development machines, and make sure it doesn’t escape into the wild: with great power comes great responsibility.

You can do it without the mouse, too. Just make sure the window is selected in the task bar (perhaps by using the Windows key and then tabbing to the appropraite icon), then hit Alt-Space, Alt-M, and then using the cursor keys.

The only problem is, some application developers choose to change the system menu – the menu visible from clicking the app’s icon in the top left, or by right-clicking the task bar icon – perhaps deciding for some insane reason not to include the standard Move option (as an aside, this breaks one of the fundamental tenets of usability: don’t change – or in this case, remove – existing, established behaviour). If this is the case, you can instead use a handful of lines of Windows API code to access the window and move it programatically. Here’s the entire code of a program that will do exactly that:

#include <windows.h>
#include <tchar.h>
 
int _tmain(int argc, TCHAR *argv[])
{
	if (argc < 2)
	{
		_tprintf(_T("usage: mover windowtitle\n"));
		return -1;
	}
	HWND hwnd = FindWindow(NULL, argv[1]);
	if (hwnd)
	{
		SetWindowPos(hwnd, HWND_TOP, 0, 0, 400,400, 0);
	}
	else
		_tprintf(_T("Unable to find window %s\n"), argv[1]);
	return 0;
}

Compile it up and you can use it as a handy little utility to move arbitrary top level windows, e.g. if you’ve got regedit running:

mover "Registry Editor"

The only tricky thing is finding the exact title of the window to use. Well, you wouldn’t want it to be too easy, would you?

type public MyControl() =
    inherit ItemsControl()
 
    [<defaultValue>]
    static val mutable FooProperty : DependencyProperty
 
    static member OnFooChanged (dob:DependencyObject) args =
        (dob :?> MyControl).Update ()
 
    [<system.ComponentModel.Bindable(true)>]
    member public this.Foo
        with get() : string  = string (base.GetValue(MyControl.FooProperty))
        and  set(r : string) = base.SetValue(MyControl.FooProperty, r)
 
    override this.OnPropertyChanged (args) =
        match args.Property.Name with
        | "Foo" -> this.Update ()
        | _          -> ()
 
    member internal this.Update () =
        System.Diagnostics.Debug.WriteLine (sprintf "Updating %A" (base.GetValue(MyControl.FooProperty)))
 
    static do
        let metadata = PropertyMetadata(null, PropertyChangedCallback (MyControl.OnFooChanged) )
        MyControl.FooProperty <- DependencyProperty.Register("Foo", typeof<string>, typeof<MyControl>, metadata)

Constructor

We don’t have one! Well, that’s not strictly true. There’s no per-instance set-up that we need to do here, so instead we have a default, parameter-less constructor implied by the “empty brackets” syntax in the type declaration. If we wanted to execute some code in the constructor, we could add the following:

    do
        Debug.WriteLine "Constructing."

It’s also possible to add further constructors (perhaps parameterised differently) but when using WPF bear in mind that instances of your class will often be created from a XAML declaration, which generally uses the default constructor and then sets properties as required. Mutable objects: ug.

Inheritance

Our type derives from the WPF ItemsControl class using the inherit keyword. Of course, we’re still subject to the single-inheritance limit of .NET (not a bad thing, if you ask me – no more tortured object hierarchies):

type public MyControl() =
    inherit ItemsControl()

Note that we have to include the empty brackets on the inherited type name, as this will be constructed implicitly when our derived class is constructed. We can access the inherited class from elsewhere in the code using the base keyword.

Static members

Dependency properties are a WPF construct that provide external storage of property values. They allow deep trees of objects to efficiently use lots of properties where they often have the default value, which is commonly the case in WPF. In order to use them with your class you have to do a few things, including creating a static value to hold the property and its metadata. We create a mutable static member for this:

    [<defaultValue>]
    static val mutable FooProperty : DependencyProperty

Why a mutable static value? If you’ve used F# already you might be aware that it’s also possible to declare an immutable static variable and its initial value in one shot with a let:

    static let FooProperty = DependencyProperty.RegisterProperty ("Foo", typeof<string>, typeof<MyControl>)

Unfortunately, this results in your DP being private. Although the CLR property is still accessible, anything that attempts to access the DP directly – for instance, code within the WPF libraries – won’t see it. This means you have to use the mutable style, which is unfortunate.

Static methods

We’ve declared a static member function that is used to receive notifications when our DP is changed (although it’s complete overkill in this example, because we’ve already declared our DP with metadata that tells it to notify us when it changes):

    static member OnFooChanged (dob:DependencyObject) args =
        (dob :?> MyControl).Update ()

As you’d expect, there’s no this parameter on the signature, a static member doesn’t have any implicit object instance to work on. Luckily the arguments to most event functions include the DependencyObject that raised the notification. That means we can downcast dynamically to our expected type (with : ?>) and use it. Bear in mind that this is more like casting than as in C#, as it will throw an InvalidCastException at runtime rather than returning null.

Properties

In order for our dependency property to be easily accessible we can expose it as a plain old CLR property. The implementation of the getter and setter simply defers all of the actual work of storing and retrieving the value to the underlying dependency property.

    member public this.Foo
        with get() : string  = string (base.GetValue(MyControl.FooProperty))
        and  set(r : string) = base.SetValue(MyControl.FooProperty, r)

Notice how we have to cast the obj returned from GetValue into the correct type. This is another example of having to bridge the gap between the dynamically typed WPF property system and F#’s static typing.

Overridden members

As well as providing new member functions and properties, we may need to override existing ones. Member functions marked abstract in the base class can be overridden using the override keyword:

    override this.OnPropertyChanged (args) =
        match args.Property.Name with
        | "Foo" -> this.Update ()
        | _          -> ()

The F# compiler provides the same kind of checking that the C# compiler does, warning you if you inadvertantly hide a base class function by creating an override with the same name but not marking it with override.

Static constructor

As mentioned before, we use the static constructor to initialise our mutable static variables. The syntax is similar to the do syntax of a normal constructor, with the addition of the static keyword:

    static do
        let metadata = PropertyMetadata(null, PropertyChangedCallback (MyControl.OnFooChanged) )
        ...

Static constructors are run once per class, regardless of how many instances of the class you have. WPF relies quite heavily on static, class-based functionality; mostly because a lot of what’s set-up is per-class configuration – it’s not going to change during the lifetime of the application – so you may find yourself doing a fair amount of work in a static constructor.

So, there’s a quick run around some of the object-oriented features of the F# language: classes, inheritance, instance constructors, overridden member functions, static member functions and constructors. You can see how using WPF means you lose some of the benefits of the F# language; notably immutability and static typing. If you’re an experienced functional programmer getting deep into creating WPF or Silverlight custom controls you may find yourself using these OO constructs more than you’d like. Although F# makes it relatively painless in practice, mixing this heavily object oriented style of programming with a functional approach can still be a little hard to stomach at times.

This is such a common source of bugs, and such an important requirement, that from Vista onwards Microsoft introduced a new set of functions in the Windows API explicitly to support it: One-Time Initialization.

And even if you get away with doing naughty things in DllMain now, don’t think that it’ll stay that way forever. We got away with it for years, then when .NET came along it introduced all sorts of additional correctness checks. For instance, the Managed Debugging Assistant (MDA) in Visual Studio will shout loudly should you attempt to run managed code during DllMain.

Managed Debugging Assistant 'LoaderLock' has detected a problem in 'C:\YourApp.vshost.exe'.
Additional Information: Attempting managed execution inside OS Loader lock. Do not attempt to run managed code inside a DllMain or image initialization function since doing so can cause the application to hang.

And it’s easier than you think to do so. For example, calling something as innocuous as GetWindowText can result in managed code being run.

How can you get around this?

One of the approaches I’ve used has been to make use of Win32 asynchronous procedure calls. Specifically you can call QueueUserAPC to add a function to the queue, and this can contain the initialisation you would’ve otherwise done in DllMain.

However there is significant gotcha regarding use of APCs: your function will not be called until the thread is in an “alertable wait state”. This means you (or some other code on the thread) need to call an alertable wait function such as SleepEx, WaitForSingleObjectEx and specify TRUE for the alertable parameter.

Once your APC is getting called successfully you’ll be in much better place; your code will be executed outside of the scope of the dreaded OS loader lock and you’ll be doing things by the book, hopefully avoiding all the potential pitfalls that lie in wait within DllMain.

I had a look at the text of the first exception, and from that it was obvious:


Unable to cast COM object of type 'System.__ComObject' to interface type 'IWhatever'. This operation failed because the QueryInterface call on the COM component for the interface with IID '{CAD939D0-5E5D-11D7-AA0B-0002B33FE9DX}' failed due to the following error: Bad variable type. (Exception from HRESULT: 0x80020008 (DISP_E_BADVARTYPE)).


The message contains curly brackets around the IID, and they’re being intrepreted as insertion points by a call to String.Format!

Workaround

So that explained why it only happens in debugger runs; as this post helpfully points out the WPF databinding is enabled by default in that case. To workaround it, you can disable logging programmatically or using the App.config file to only display Critical output. So if your app is named foo.exe, create a file foo.exe.config that contains:

<configuration>

  <system.diagnostics>

    <sources>

      <source name="System.Windows.Data" switchName="SourceSwitch"/>

    </sources>

    <switches>

      <add name="SourceSwitch" value="Critical" />

    </switches>

  </system.diagnostics>

</configuration>

Fix…?

Hopefully there’ll be a fix for this at some point soon, as disabling the logging is a pretty large hammer to crack this small nut.

So quite often in our mixed code-base, we find that the .NET garbage collection process doesn’t kick in when we need it to. For instance, when we’ve allocated a lot of memory in the COM code that .NET isn’t aware of. Memory exhaustion has to get pretty bad for the GC to occur at any other time than during a .NET allocation, either the system-wide low-memory event has to be signalled or an OutOfMemoryException needs to be thrown. In both of these cases it’s probably too late to do anything about it.

In this case it’s extremely useful to be able to see what .NET objects are still alive, and what COM objects they’re hanging on to. Unfortunately this isn’t as easy as it might seem.

The COM object itself hides within a weakly-typed System.__ComObject or a strongly-typed managed wrapper, depending on whether rich type information is available. Furthermore, a runtime controller RCW (runtime callable wrapper) is what actually holds a pointer to the object itself, and this structure is internal to mscorwks.dll.

So how can we untangle this and, on finding a __ComObject that happens to still be alive (i.e. is not reachable in the object graph and is therefore eligible for garbage collection) identify which COM object it’s hanging on to.

First of all, let’s see how many __ComObjects are still alive. In this case, it’s only one (phew!):

0:005> !DumpHeap -type __ComObject
 Address       MT     Size
01453b74 79306e60       16
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
79306e60        1           16 System.__ComObject
Total 1 objects

And you remember the layout of .NET objects in memory, don’t you? Of course you do! The 4 bytes prior to the address displayed (01453b74) are the “object header”. The exact content of the header is apparently undocumented. Let’s see what it contains (at least on a 32-bit platform, your mileage may vary):

0:005> dd 01453b74-4 L1
01453b70  08000002

According to various sources the object header contains 2 fields; a handle and a sync block index. If the object is an RCW, the handle is always 0×08000. You can use the index with SOS’s !syncblk command to de-reference it:

0:005> !syncblk 2
Index SyncBlock MonitorHeld Recursion Owning Thread Info  SyncBlock Owner
    2 001e0fec            0         0 00000000     none    01453b74 System.__ComObject
-----------------------------
Total           3
CCW             0
RCW             1
ComClassFactory 0
Free            0

The sync block itself is an undocumented structure, but after a bit of investigation, it turns out that at offset 0×1c there is a pointer to a further structure that contains the “interop information”:

0:005> dd 001e0fec+1c L1
001e1008  001e9510

And from this, we can obtain a pointer to the RCW itself. We’re almost there!

0:005> dd 001e9510+c L1
001e951c  001e5380

The RCW is a pretty large structure, but for our purposes there are only a couple of interesting fields: the IUnknown pointer at 0×64, and the object’s virtual function table pointer at 0×88. If you use dds you can easily see any symbols associated with these pointers:

0:005> dds 01e5380+64 L1
001e53e4  00ef6c24

0:005> dds 01e5380+88 L1
001e5408  00eb9710 rcwrepro!ATL::CComObject::`vftable'

This is the salient information; we now know exactly what type of COM object we’re dealing with. This is obviously a bit fragile, given that it relies on structures from mscorwks that may well change in newer versions of the runtime (I’ll check on .NET 4 when I get a chance). It’s also a bit of a pain to go through all these steps manually in WinDbg, so I put together a simple extension DLL to do it automatically given the address of the __ComObject. I’ll upload that and blog about it soon.

It’s obvious looking at the output from VMmap or windbg’s !address command what the largest contiguous block is, e.g.

0:008> !address -summary
....
Largest free region: Base 07300000 - Size 63ed0000 (1637184 KB)

But what if you need that number in order to make a decision at run-time? For instance, to decide whether your process is in a fit state to continue, or if it should instead commit hara-kiri. In that case, you need to access the information programmatically. That’s where the immensely useful VirtualQueryEx function comes in…

VirtualQueryEx gives you information on a single page of your virtual address space at a time. Pages size are dependent on the architecture and OS, but if you just want to iterate over all of them, you don’t need to add any special handling; the function returns the size of the page in an element of the MEMORY_BASIC_INFORMATION structure, so you can simply move to the ‘next’ page regardless of size.

If you’re interested in free space, you’ll need to find all the pages that have a state of MEM_FREE (0×10000), and that’s pretty much all there is to it. By keeping track of how much space falls into a continuous range of MEM_FREE pages you can get to the number reported by VMmap and !address.

Here’s some C++ code that returns the address of the largest free contiguous block in largestFreestartlargestFree. Enjoy!

	MEMORY_BASIC_INFORMATION mbi;
	DWORD start = 0;
	bool recording = false;
	DWORD freestart = 0, largestFreestart = 0;
	__int64 free = 0, largestFree = 0;
	while (true)
	{
		SIZE_T s = VirtualQueryEx(hproc, reinterpret_cast<lpvoid>(start), &mbi, sizeof(mbi));
		if (s != sizeof(mbi))
		{
			if (GetLastError() != ERROR_INVALID_PARAMETER)
				return ReportError(GetLastError(), _T("Failed to VirtualQueryEx at %08x"), start);
			else
				break;
		}
		if (mbi.State == MEM_FREE)
		{
			if (!recording)
				freestart = start;
			free += mbi.RegionSize;
			recording = true;
		}
		else
		{
			if (recording)
			{
				if (free > largestFree)
				{
					largestFree = free;
					largestFreestart = freestart;
				}
			}
			free = 0;
			recording = false;
		}
		start += mbi.RegionSize;
	}