What is .NET type equivalence?

Described at a high level here, .NET 4.0 type equivalence essentially gives you a way of indicating that different .NET types represent the same underlying COM type and is most commonly used in COM interop scenarios. One of the reasons for its introduction is to save developers from having to ship large interop DLLs with their software, e.g. the multi-megabyte Microsoft.Office.Interop. Instead the compiler can inline the definition of any types used, and mark them appropriately as representing the original COM types.

The error

We noticed that whenever we built and ran an application that referenced a DLL using .NET 2.0, it worked. Doing the same thing with .NET 4.0 caused a BadImageFormatException.

Unhandled Exception: System.BadImageFormatException: Could not load file or assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies. An attempt was made to load a program with an incorrect format.
at X.Main()


Let’s dig!

So, the BadImageFormatException doesn’t actually tell us much. Let’s break out WinDbg and see what we can find. Running the faulting app we can see several C++ exceptions before the CLR exception is thrown:

(178c.790): C++ EH exception - code e06d7363 (first chance)
...
(178c.790): C++ EH exception - code e06d7363 (first chance)
(178c.790): CLR exception - code e0434352 (first chance)

I changed the exception handling settings to stop on C++ exceptions (sxe eh) then ran again to see where things were going wrong. It stopped here:

0:000> kp
ChildEBP RetAddr
0012d15c 79084c0f KERNEL32!RaiseException+0x53
0012d194 793371be MSVCR100_CLR0400!_CxxThrowException+0x48
0012d5e4 79455cae clr!EEFileLoadException::Throw+0x1a8
0012d634 794558d2 clr!CompareTypeTokens+0x200
0012d6b0 791b5c00 clr!IsTypeDefEquivalent+0x102
0012d6d4 791b2ca8 clr!MethodTableBuilder::CheckForTypeEquivalence+0x94
0012d7ac 791b27c9 clr!MethodTableBuilder::BuildMethodTableThrowing+0x60d
0012d9a4 791a4578 clr!ClassLoader::CreateTypeHandleForTypeDefThrowing+0x88e

Interesting. Notice how the call stack contains some .NET 4.0 specific methods relating to the new type equivalence feature. We’re hitting a new code path, which is consistent with the fact that running against a down-level CLR works.

After a bit more toing-and-froing, I discovered that the C++ exception is thrown when clr!MDInternalRO::IsValidToken returns an error. By disassembling the function we can see it’s just looking at various bits in the token value, and it decides that the value passed (0×02000000) isn’t valid. Looking at the output from ildasm that token doesn’t appear anywhere. And if we add a dump of the value, we can see that it indeed doesn’t look like the other tokens:

0:000> bu clr!MDInternalRO::IsValidToken "dd esp+8 L1; g"
...
0012f5a8  02000001
0012f31c  06000001
0012f2c0  02000002
0012f0f4  02000002
0012ebe4  01000001
0012e944  23000001
...
0012d5f4  02000000
(18ec.1ec8): C++ EH exception - code e06d7363 (first chance)

What’s the culprit?

So it looks pretty conclusive; the DLL contains something that the CLR isn’t expecting. But what? It’s time to break out the oldest tool in the troubleshooting box: the binary chop!

Eventually I got the referenced DLL down to only a single simple construct. Can you guess what it is? A global literal value. A real global value, one that isn’t even part of a type. Crazy huh? In IL it looks like this:

.field public static literal valuetype Test.MyEnum LiteralValue = int32(0x00000001)

It’s a literal value of an enumerated type. That’s important: using a value of a simple type (say int32) does not provoke the error.

Now, I wasn’t even sure that this is a valid IL construct, but according to the ECMA IL spec, specifically partition II, section 15, it is:

The CLI also supports global fields, which are fields declared outside of any type definition. Global fields shall be static.

So it looks like we’re not doing anything illegal, backed up by the fact that the .NET 2.0 CLR can make use of it without a problem.

Interestingly, there’s another aspect that influences whether this code path is hit. As mentioned above, type equivalence is intended for use with interop libraries. As such, it only kicks in if your referenced assembly is marked with the PrimaryInteropAssembly attribute, e.g.:


.custom instance void [mscorlib]System.Runtime.InteropServices.PrimaryInteropAssemblyAttribute::.ctor(int32,int32) = ( 01 00 01 00 00 00 00 00 00 00 00 00 )


The Fix?

The issue is currently with Microsoft product support. Let’s see what they come up with; is it too esoteric for a hotfix…?

The Repro

Here’s some code and instructions on how to repro the problem.

1. Build the IL into a DLL using ilasm.
   "c:\WINNT\Microsoft.NET\Framework\v2.0.50727\ilasm.exe" /dll Test.il /output=Test.dll
2. Build the application into a .NET 4.0 EXE that references the DLL
   "c:\winnt\Microsoft.NET\Framework\v4.0.30319\csc.exe" TestConsumer.cs /reference:Test.dll
3. Run the resulting TestConsumer.exe application and you’ll get the exception

Test.il

.assembly extern mscorlib
{
.publickeytoken = (B7 7A 5C 56 19 34 E0 89 )
.ver 2:0:0:0
}
.assembly Test
{
.custom instance void [mscorlib]System.Runtime.InteropServices.PrimaryInteropAssemblyAttribute::.ctor(int32,int32) = ( 01 00 01 00 00 00 00 00 00 00 00 00 )
.hash algorithm 0x00008004
.ver 1:0:0:0
}
.module Test.dll
.imagebase 0x00400000
.file alignment 0x00000200
.stackreserve 0x00100000
.subsystem 0x0003
.corflags 0x00000001

.class public auto ansi sealed Test.MyEnum
extends [mscorlib]System.Enum
{
.field public specialname rtspecialname int32 value__
.field public static literal valuetype Test.MyEnum Zero = int32(0x00000000)
.field public static literal valuetype Test.MyEnum One = int32(0x00000001)
}

TestConsumer.cs

class X
{
    static void Main()
    {
        var v = Test.MyEnum.Zero;
    }
}

But what happens if during the lifetime of that stack based COM object, it gets used from .NET? A runtime callable wrapper (RCW) will be created around the object. And this RCW expects to be able to keep the underlying object alive by incrementing its reference count. Of course, the stack-based object will soon go out of scope, and regardless of its reference count the object will be destroyed and the pointer that the RCW contains will no longer be valid. It points into the stack, so when the RCW gets cleaned-up, the CLR will call via this pointer into memory that contains garbage and you’ll get something nasty like an access violation or illegal instruction exception.

It’s fairly easy to reproduce this to see where things go wrong. It can be useful to see where the CLR blows up, and how we can identify this as the cause.

Lets start by creating a simple pseudo-COM object that implements just the bare minimum to be usable:

class MyClass : public IUnknown
{
public:
	MyClass():l(0) {}
	STDMETHOD_(ULONG, AddRef)() { return InterlockedIncrement(&l); }
	STDMETHOD_(ULONG, Release)() { return InterlockedDecrement(&l); }
	STDMETHOD(QueryInterface)(REFIID iid, void ** ppvObject)
	{
		if (iid == IID_IUnknown)
		{
			*ppvObject = this;
			AddRef();
                        return S_OK;
		}
		return E_NOINTERFACE;
	}
private:
	long l;
};

We’ll also need a COM visible .NET object that will use the object. It doesn’t actually need to be COM visible, but that’s the easiest way to access it from C++, in my opinion.

I’ve created the COM object in F#. It’s a trivial class that has a single interface, with a single method that takes the object we pass to it and prints its type. This is enough for the RCW to be created.

open System
open System.Runtime.InteropServices
 
module Module1 =
 
    [<ComVisible(true); InterfaceType(ComInterfaceType.InterfaceIsIUnknown)>]
    type public IConsumer = 
        abstract member UseObject : o:obj -> unit
 
    [<ComVisible(true); ClassInterface(ClassInterfaceType.None)>]
    type public Consumer() =
        interface IConsumer with
            member this.UseObject (o:obj) =
                Console.WriteLine (sprintf "%A" (o.GetType()))

We can compile this into a DLL, then run regasm with the /tlb switch to generate a type library (TLB):

fsc -o:obj\Debug\testStackObjectsFs.dll Module1.fs
regasm /tlb:testStackObjectsFs.tlb testStackObjectsFs.dll

That can be #imported back into our test harness:

#import "testStackObjectsFs.tlb"

Now we’re ready to put together some code that creates an instance of our object on the stack and passes it to our .NET component:

void Foo()
{
	// Create an instance of our "COM object" on the stack
	MyClass obj;
 
	// Create a managed object
	testStackObjectsFs::IConsumerPtr mgd(__uuidof(testStackObjectsFs::Consumer));
 
	// and pass our COM object to it
	mgd->UseObject(_variant_t(&obj));
}
 
int _tmain(int argc, _TCHAR* argv[])
{
	// Initialise the COM runtime, for our purposes it doesn't
	// matter which threading model we use
	CoInitializeEx(NULL, COINIT_MULTITHREADED);
 
	// Call a separate function, to ensure stack-based objects
	// are out-of-scope on return.
	Foo();
 
	// Wait for some input
	_getch();
	return 0;
}

Now, if you run this from within Visual Studio, if you’re vigilant (and you haven’t got your debugger set to stop on access violations), then you’ll notice this in the output window after the return statement executes:

...
The thread 'Win32 Thread' (0x15b0) has exited with code 11001 (0x2af9).
The thread 'Win32 Thread' (0x1110) has exited with code 0 (0x0).
First-chance exception at 0x00850a2b in testStackObjects.exe: 0xC0000005: Access violation reading location 0x00850a2b.
The thread 'DebuggerRCThread::ThreadProcStatic' (0x1534) has exited with code 0 (0x0).
The thread 'RPC Callback Thread' (0x12b8) has exited with code 0 (0x0).
...

Lets ramp up WinDbg, attach to the process (that _getch comes in useful here) and find out what’s going on in a bit more detail.

If we let the app run to the point of failure in WinDbg, we can see that the CLR was in the act of shutting down when it caused the exception:

0:002> kp
ChildEBP RetAddr
WARNING: Frame IP not in any known module. Following frames may be wrong.
00dae3fc 79f4c1b5 0xe06ff8
00dae450 79f4c26c mscorwks!ReleaseTransitionHelper+0x5f
00dae494 79f4c2d0 mscorwks!SafeReleaseHelper+0x8c
00dae4c8 79faaa01 mscorwks!SafeRelease+0x2f
00dae4fc 79faa7c8 mscorwks!IUnkEntry::Free+0x68
00dae510 79faa91d mscorwks!RCW::ReleaseAllInterfaces+0x18
00dae540 79faa949 mscorwks!RCW::ReleaseAllInterfacesCallBack+0xbd
00dae570 7a0792ac mscorwks!RCW::Cleanup+0x22
00dae57c 7a079714 mscorwks!RCWCleanupList::ReleaseRCWListRaw+0x16
00dae5ac 7a0797df mscorwks!RCWCleanupList::ReleaseRCWListInCorrectCtx+0xdf
00dae5fc 79fdc140 mscorwks!RCWCleanupList::CleanupAllWrappers+0x77
00dafe90 79fdc7aa mscorwks!RCWCache::ReleaseWrappersWorker+0x103
00dafed8 79fd9f95 mscorwks!ReleaseRCWsInCaches+0x27
00dafee0 79f3c76a mscorwks!InnerCoEEShutDownCOM+0x1e
00daff14 79f92015 mscorwks!WKS::GCHeap::FinalizerThreadStart+0x1fc
00daffb4 7c80b683 mscorwks!Thread::intermediateThreadProc+0x49
00daffec 00000000 kernel32!BaseThreadStart+0x37

Essentially it’s cleaning up the currently unused RCWs – including our malformed one – and as part of doing this, it’s calling Release on the underlying COM object, via the mscorwks!SafeRelease function. SafeRelease wraps the call to potentially (and definitely, in this case) dangerous unmanaged code with various exception handlers, enabling it to silently handle access violations.

If we run the app again, and this time break while it’s waiting for the keypress, before it attempts to clean up the RCWs, then we can examine the wrapper ourselves, using the approach I set out in this post.

List all of the untyped COM object wrappers:

0:002> !dumpheap -type System.__ComObject
 Address       MT     Size
01418628 79306e60       16
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
79306e60        1           16 System.__ComObject
Total 1 objects

Use the address of the object to obtain its object header:

0:002> dd 1418628-4 L1
01418624 08000002

Use the syncblk identifier in the header to get the syncblk:

0:002> !syncblk 2
Index SyncBlock MonitorHeld Recursion Owning Thread Info  SyncBlock Owner
    2 001e4d9c            0         0 00000000     none    01418628 System.__ComObject
-----------------------------
Total           2
CCW             0
RCW             0
ComClassFactory 0
Free            0

Get the address of the RCW from the sync block:

0:008> dd 001e4d9c+1c L1
001e4db8 001e7dc8
0:008> dd 001e7dc8+c L1
001e7dd4 001de828

And dump out the relevant bits of the RCW, the vtable of the object, at offset 0×88, and the IUnknown pointer, at offset 0×64:

0:008> dds 001de828+88 L1
001de8b0 0041ac78 testStackObjects!MyClass::`vftable'
0:008> dds 001de828+64 L1
001de88c 0012fe7c

We can use !address to do a quick sanity check on the pointer and verify what we know to be the case; it’s stack memory:

0:008> !address 0012fe7c
    00030000 : 00124000 - 0000c000
                    Type     00020000 MEM_PRIVATE
                    Protect  00000004 PAGE_READWRITE
                    State    00001000 MEM_COMMIT
                    Usage    RegionUsageStack
                    Pid.Tid  490.13dc

If we run the app on again to the point that it fails, we can clearly see the address of the object being passed as an argument to mscorwks!IUnkEntry::Free.

So the moral of the story is; don’t pretend some arbitrary piece of stack memory is a real, reference counted COM object. You may be saving the cost of a heap allocation, but even if your app works OK today, it may not tomorrow when someone introduces a piece of .NET code somewhere in your object graph.

Bonus Extra Content

As a bonus tip, here are a couple of WinDbg breakpoints that can be used to dump each RCW as it’s created and destroyed.

bu 79faa974 "dds @ecx L23; g"
bu 79faa538 "dd @esp+20 L1; dds poi(@esp+20)+88 L1; g"


So quite often in our mixed code-base, we find that the .NET garbage collection process doesn’t kick in when we need it to. For instance, when we’ve allocated a lot of memory in the COM code that .NET isn’t aware of. Memory exhaustion has to get pretty bad for the GC to occur at any other time than during a .NET allocation, either the system-wide low-memory event has to be signalled or an OutOfMemoryException needs to be thrown. In both of these cases it’s probably too late to do anything about it.

In this case it’s extremely useful to be able to see what .NET objects are still alive, and what COM objects they’re hanging on to. Unfortunately this isn’t as easy as it might seem.

The COM object itself hides within a weakly-typed System.__ComObject or a strongly-typed managed wrapper, depending on whether rich type information is available. Furthermore, a runtime controller RCW (runtime callable wrapper) is what actually holds a pointer to the object itself, and this structure is internal to mscorwks.dll.

So how can we untangle this and, on finding a __ComObject that happens to still be alive (i.e. is not reachable in the object graph and is therefore eligible for garbage collection) identify which COM object it’s hanging on to.

First of all, let’s see how many __ComObjects are still alive. In this case, it’s only one (phew!):

0:005> !DumpHeap -type __ComObject
 Address       MT     Size
01453b74 79306e60       16
total 1 objects
Statistics:
      MT    Count    TotalSize Class Name
79306e60        1           16 System.__ComObject
Total 1 objects

And you remember the layout of .NET objects in memory, don’t you? Of course you do! The 4 bytes prior to the address displayed (01453b74) are the “object header”. The exact content of the header is apparently undocumented. Let’s see what it contains (at least on a 32-bit platform, your mileage may vary):

0:005> dd 01453b74-4 L1
01453b70  08000002

According to various sources the object header contains 2 fields; a handle and a sync block index. If the object is an RCW, the handle is always 0×08000. You can use the index with SOS’s !syncblk command to de-reference it:

0:005> !syncblk 2
Index SyncBlock MonitorHeld Recursion Owning Thread Info  SyncBlock Owner
    2 001e0fec            0         0 00000000     none    01453b74 System.__ComObject
-----------------------------
Total           3
CCW             0
RCW             1
ComClassFactory 0
Free            0

The sync block itself is an undocumented structure, but after a bit of investigation, it turns out that at offset 0×1c there is a pointer to a further structure that contains the “interop information”:

0:005> dd 001e0fec+1c L1
001e1008  001e9510

And from this, we can obtain a pointer to the RCW itself. We’re almost there!

0:005> dd 001e9510+c L1
001e951c  001e5380

The RCW is a pretty large structure, but for our purposes there are only a couple of interesting fields: the IUnknown pointer at 0×64, and the object’s virtual function table pointer at 0×88. If you use dds you can easily see any symbols associated with these pointers:

0:005> dds 01e5380+64 L1
001e53e4  00ef6c24

0:005> dds 01e5380+88 L1
001e5408  00eb9710 rcwrepro!ATL::CComObject::`vftable'

This is the salient information; we now know exactly what type of COM object we’re dealing with. This is obviously a bit fragile, given that it relies on structures from mscorwks that may well change in newer versions of the runtime (I’ll check on .NET 4 when I get a chance). It’s also a bit of a pain to go through all these steps manually in WinDbg, so I put together a simple extension DLL to do it automatically given the address of the __ComObject. I’ll upload that and blog about it soon.

In order to obtain the type information about the COM objects, the tool uses type libraries, and the associated ITypeLib and ITypeInfo interfaces, which, with a little effort, can be used to iterate over all the coclasses, interfaces and functions in the library. But the difficulty lies in obtaining a type library when all you’re given is an already-instantiated object. In theory, COM allows you to know no more about an object than what interfaces it supports. But in practice, there are a variety of ways you can circumvent this and get to the type information.

For unmanaged COM objects you can use the information in the registry (or SxS configuration) and obtain the server (DLL) that contain a TLB embedded as a resource, or the type library filename itself. I won’t go into that now, there’s plenty of information about the location of these common registry keys elsewhere on the internet.

But for managed COM objects – well, COM callable wrappers (CCWs) – you have a different problem: registry scraping will never work and there may not even be an associated type library. The InprocServer32 registry entry always points to mscoree.dll, which obviously doesn’t have an embedded type library, and unless you’ve registered the assembly with /tlb (which is a pain) then you won’t have entries under HKEY_CLASSES_ROOT\Typelib and a TLB file to load.

So, if you’re in the unmanaged world, and all you’ve got is a pointer to a live CCW, what can you do?

Well, the easiest thing is to use IProvideClassInfo. This is supported by all CCWs, and provides a way to get an auto-generated (by the CLR) ITypeInfo implementation for the managed class. In fact, this is what I actually used to implement the solution eventually, but along the way I discovered some other interesting aspects of the CCW.

There is another interface that it supports: _Object, the unmanaged version of System.Object, which supports basic .NET functionality such as ToString and GetType. I couldn’t find it declared anywhere in the Platform or .NET SDK headers, so I put together a version that I could use from C++:

Despite the presence of the virtual functions in this “interface”, we’re not actually going to call them. Instead we’ll call through the IDispatch that it derives from. It may be possible to use them directly, but see the comment describing what happens when I tried it. Calling via IDispatch may seem slightly odd, because the object itself claims not to support it (QueryInteface returns E_NOINTERFACE).

The methods on the _Object interface have well-known DISPIDs:

So we can use that to invoke the GetType method:

And we get back a _Type interface that allows us to navigate around the type information in the same way as we can with System.Type! Just #import mscorlib.tlb and you get all the interfaces you need to e.g. iterate over all the interfaces implemented by a type, and invoke a function on them:

So this turns out to be quite nice: you can get rich managed type information even if you’re running in the unmanaged world.

A bit of background: as you may know, if you add functions to the worksheet or workbook code in Excel then they appear as callable methods on the objects themselves. This is achieved with the use of dynamic dispatch and IDispatch. For example, creating a workbook with this function in it’s VBA code:

Public Function Foo() As Double
    Foo = 100
End Function

Means you can call it like this:

MsgBox CStr(ThisWorkbook.Foo)

As well as being able to call it like this from within the Excel session (i.e. in other VBA code in the process), you can also access it externally using the COM object model that Office applications expose. For instance, you can use VBScript:

Dim excel, wkb
Set excel = CreateObject("Excel.Application")
Set wkb = excel.Workbooks.Open("a.xls")
WScript.Echo wkb.Foo


Or, more interesting, using F#:

The key part here is that we’re using wks.GetType() to get a managed representation of the unmanaged Excel COM interface. Under the covers this is creating a runtime callable wrapper (RCW) to wrap the worksheet COM object.

However, the problem we were seeing was that opening multiple sheets resulted in failures to call the method in certain situations. Although the VBA signature was exactly the same in all of the sheets, it seemed that opening b.xls after a.xls, would fail; returning null when we expected it to return a value. If we opened c.xls after b.xls, it would fail in a different way; never actually making it to the body of the function. Very odd.

My first suspicion was that it was somehow related to COM object vs .NET object lifetime. This is quite a common problem whens invoking Excel using managed code. It’s bad mixing COM and .NET anyway; generally deterministic, reference-counted lifetime semantics don’t play well with the garbage collector. Throwing an app with a full-blown UI being managed as COM object into the mix just complicates matters further. Anyway, it’s been widely discussed, so I won’t say any more about it here; suffice to say that calling Marshal.ReleaseCOMObject and GC.Collect got us to the point where we could see the Excel process terminating, so we knew that the failure wasn’t due to some state being cached inside there.

So we concentrated on different aspects of the problem:

• Given the pattern of failures, it seemed that the order of opening the sheets and calling affected the outcome. This hinted that something was persisting betweeen calls, but not in the client (Excel) site.
• The code had previously worked when written in VBScript, so there was nothing intrinsically wrong with the operations we were performing.

This seemed to strongly indicate that something was being cached at the .NET level. And the major difference between the .NET code and the VBScript was that the former used Type.GetType() on the worksheet object to get it’s managed representation, while the latter used the IDispatch directly.

So it looked like GetType() was caching some information about the particular IDispatch implementation that it encountered first, then reusing that for subsequent worksheet implementations which actually had a slightly different layout, i.e. although they also had the Foo function which we were trying to call, they had a different set of other dynamic functions too.

After a bit of digging about I uncovered this gem: the mapping between interface pointers and runtime callable wrappers. Which seemed to describe exactly what we were seeing. The first time through the loop, the runtime is asked to get the type and is given a COM pointer, for which it creates a RCW that we can use to invoke Foo. The second time through the loop, the runtime thinks it’s seen the object before, so rather than perform the expensive operation of creating the RCW again, it just returns the original one. The problem is, the underlying COM object is different!

So, in order to prevent the runtime for trying to cache the RCW, we need to use Marshal.GetUniqueObjectForIUnknown and that does the trick nicely. We first need to get an IUnknown for our object, than convert that back into an object, which is actually a RCW:

Although it’s less efficent, at least the code works, and it finally allows us to call the dynamic methods on the workbook object from F# .NET.

It will be interesting to see how dynamic in .NET 4.0 addresses this kind of issue.

Why? Well, because it’s too easy to use them as an excuse for not defining your shared library interfaces properly.

The reason this is on my mind recently is that several hundred, yes, you heard that right, several hundred DLLs have been released by my group over the last, ooh, 10 years or so. They are all still in use. Each of them has burned into it a copy of the library that deals with interfacing with Excel. That means each of these has it’s own little internal copy of the current state-of-the-art. The problem with that is; the state-of-the-art moves on. And how do you go about updating the DLLs that are already in production? You have to re-release them. In an environment where thes DLLs are used for marking the profit and loss on a large derivatives trading book, that’s not a small undertaking. And it’s made worse if, say the DLL in question was last released with a different version of the compiler.

My approach would be to refactor this shared static library (.lib) into a stand-alone DLL.

At this point, people start saying “oh, but then you’ve got a single point of failure, if you release a broken version of that DLL, everything will stop working!”. Not exactly a compelling argument. If the functionality of the DLL is well defined, and there are well known entry points it should be easy to put together a comprehensive black-box test suite. In fact we already do that with all our other DLLs (COM servers). The fact that this shared library *isn’t* a DLL has meant that it’s fallen through the testing cracks; another good reason to refactor it.

The internal interface to the shared library is already relatively well defined. It has a set of header files that define all of the functions and classes that are consumed by others. It’s a relatively small step to compile it as a DLL, rather than a static library. The problem then becomes one of maintenance, dealing with the inevitable changes to the external interface in a backwardly compatible way.

And that’s the problem. It requires some effort. Elsewhere in our codebase we use COM as a magic cure-all for avoiding having to deal with versioning: interface immutability rules. All interfaces are public, no published interface ever changes, object identity is based purely on interfaces supported. If you haven’t got these crutches to rely on, then you have to enforce the rules yourself, which can be both logistically and technically difficult when you’re dealing with C++.

But it’s not impossible. And I really think it would be better than having hundreds of DLLs all containing subtley different versions of the same code, and being unable to change behaviour across the board without having to build, test and release them all.

Maybe you’ve got a different opinion?

No doubt this will be addressed in the CTP release, due in a couple of weeks.

• Get CLSID from the ProgID
• Look up filename in HKCR\CLSID\{clsguid}\InprocServer32

Now obviously, if you attempt this on a machine where the components are only being used via SxS (i.e. have never been registered) the CLSIDFromProgID step will work – OLE32.DLL, which implements this function, is aware of SxS – but the subsequent steps won’t because the information isn’t in the registry.

I guess this is really breaking because we’re taking advantage of our knowledge of COM internals, rather than going via the official APIs. Although as far as I know the “correct” way of doing the progid->(type library) DLL mapping is via IProvideClassInfo, but that relies on the COM objects supporting this interface, and unfortunately we have, err, several – hundred – that don’t.

So, I set about looking to see if there was a way to get this information using the activation context APIs. All the information is there in the manifest, so how do we get it – without something nasty like querying the manifest XML directly?

There are only a handful of functions in the ActCtx API, and one of them – FindActCtxSectionGuid – looks relevant. By calling this with the ACTIVATION_CONTEXT_SECTION_COM_SERVER_REDIRECTION flag, it looks like we can get some data from the manifest based on our COM object CLSID.

Here’s the problem. The returned data from this function is a ACTCTX_SECTION_KEYED_DATA structure, and as far as I can tell this is essentially an opaque blob, with a couple of length indicators. I couldn’t find any documentation about what the lpData member was supposed to point to (if you know any better, please let me know)!

I decided to break out WinDbg, and see what OLE32!CLSIDFromProgID did, as I assumed that this must be doing something similar. It was! In fact, it was calling FindActCtxSectionString to map the prog ID to a CLSID, then using this in a call to FindActCtxSectionGuid. After a bit of disassembly, and some staring at the memory window in Visual Studio, I got a good enough idea of the contents to be able to figure out how the data referenced the filename:

typedef struct tagSECTION_DATA

{

DWORD dwSize; // 0×78 (120) structure size?

DWORD _2; // 0×00

DWORD dwSectionType; // 0×04 (ACTIVATION_CONTEXT_SECTION_COM_SERVER_REDIRECTION)

GUID clsid; // CLSID of class?

GUID _5; // Some other GUID

GUID _6; // CLSID of class again…?

GUID _7; // NULL

DWORD dwFileNameLength; // file name size in bytes

DWORD dwFileNameSectionOffset; // file name offset into data.lpSectionBase

DWORD dwProgIDLength; // progid size in bytes

DWORD dwProgIDOffset; // offset from start of this structure to progid (0×78)

BYTE _8[28]; //Unknown

// Prog ID string follows

} SECTION_DATA;

So now you can cast the data member to this structure and easily extract the filename, voila!

ACTCTX_SECTION_KEYED_DATA data;

data.cbSize = sizeof(data);

if (!FindActCtxSectionGuid(FIND_ACTCTX_SECTION_KEY_RETURN_HACTCTX,

NULL,

ACTIVATION_CONTEXT_SECTION_COM_SERVER_REDIRECTION,

&guid,

&data))

{

return GetLastError();

}

if(data.ulDataFormatVersion == 1) // Fail-safe in case internal format changes…?

{

// Cast returned data to our structure type

SECTION_DATA *pdata = static_cast<SECTION_DATA *>(data.lpData);

// DLL filename can be found in the section base data at specified offset

std::wstring filename(

reinterpret_cast<wchar_t *>( ((BYTE *)data.lpSectionBase + pdata->dwFileNameSectionOffset) ));

}

So the SxS compliant version of the CLSID to filename/typelibrary is:

• Get CLSID from the ProgID
• Look up GUID in current activation context using FindActCtxSectionGuid
• Decipher returned data to get filename

I’m sure this will break horribly when they change the internal format of the activation context data (in fact, it’s probably different now between XP and Vista – many things in the SxS world are), but hopefully we can use the ulDataFormatVersion to do some basic sanity checking.

Now if only some of those useful looking functions in sxs.dll were documented…

One of my aims at work is to simplify the development and testing regime as much as possible, and this mostly consists of making sure we’re using the most appropriate tools and technologies wherever possible. In the context of correctness checking this involves determining whether the time we spend installing, configuring and chasing down false positives in third-party tools such as Purify outweighs their benefits.

As a rule I’d always favour using built-in, vendor provided hooks rather than a bolt-on product, and as such I was interested in what we could do with the IMallocSpy functionality in the COM runtime, as our group’s software is almost all COM based.

It turned up some interesting things…

The first was the behaviour of CoRevokeMallocSpy. Stupidly I originally neglected to check the return value, and then when I did, found it was returning E_ACCESSDENIED. It turns out that this means there are still allocations that occurred when the spy was active that haven’t been freed. Given that my use of the spy was to check for leaked allocations in the first place, this was a bit annoying; at least, it meant I couldn’t overload the lifetime of my spy object, to, say, dump a list of leaks when it was destroyed. If there were any leaks it never got destroyed!

The next problem was that it seemed whenever I had an app that called the apparently innocous CLSIDFromProgID, it would leak memory. For example:

addr: 0x0015eb20:
size: 0x2c (44)
contents:
c0 31 fd 76 30 c8 15 00 01 00 00 00 01 00 12 00 .1.v0...........
28 00 10 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
01 f0 ad ba bc ec 15 00 94 ec 15 00             ............
[0x77583315] CSpyMalloc_Alloc+0x49
[0x774fd073] CoTaskMemAlloc+0x13
[0x76fd18fb] operator new+0xe
[0x76fd5c6b] StgDatabase::InitClbFile+0x2e
[0x76fdc190] StgDatabase::InitDatabase+0x623c
[0x770076fa] OpenComponentLibraryEx+0x3e
[0x77005306] OpenComponentLibraryTS+0x1a
[0x76fd954d] _RegGetICR+0x761f
[0x76fd1f24] CoRegGetICR+0xffff877d
[0x76fd6a20] IsSelfRegProgID+0x65
[0x76fd80f9] CComCLBCatalog::GetClassInfoFromProgId+0x1783
[0x77518a6d] CComCatalog::GetClassInfoFromProgId+0x100
[0x77518964] CComCatalog::GetClassInfoFromProgId+0x1e
[0x775188a0] CLSIDFromProgID+0x76
[0x004120f5] wmain+0xa5
[0x004173a6] __tmainCRTStartup+0x1a6
[0x004171ed] wmainCRTStartup+0xd
[0x7c816fd7] BaseProcessStart+0x23

Looking at the stack trace, it seemed there was some kind of internal caching going on, but what was confusing me was that I was under the impression that all memory allocated by the COM runtime would be freed by the time CoUninitialize was done. After all, you can’t make any further COM calls after this point. If you don’t believe me, just try using a static CComPtr, and see what happens in DllMainCRTStartup when your app exits.

After a bit of poking about with WinDbg (thank goodness we get decent symbols for the OS now), I could see that some kind of “database” was being created within CLBCATQ.DLL:

0:000> k
ChildEBP RetAddr¼br> 0012faf4 770076fa CLBCATQ!StgDatabase::InitDatabase
0012fb18 77005306 CLBCATQ!OpenComponentLibraryEx+0x3e
0012fb34 76fd954d CLBCATQ!OpenComponentLibraryTS+0x1a
0012fdd0 76fd1f24 CLBCATQ!_RegGetICR+0x205
0012fdf0 76fd6a20 CLBCATQ!CoRegGetICR+0x29
0012fe48 76fd80f9 CLBCATQ!IsSelfRegProgID+0x6b
0012fe88 77518a6d CLBCATQ!CComCLBCatalog::GetClassInfoFromProgId+0x51
0012fec0 77518964 ole32!CComCatalog::GetClassInfoFromProgId+0x149
0012fee0 775188a0 ole32!CComCatalog::GetClassInfoFromProgId+0x1e
0012ff0c 00401340 ole32!CLSIDFromProgID+0x95
0012ff7c 00401cae testleakcheck!wmain+0x60
0012ffc0 7c816fd7 testleakcheck!__tmainCRTStartup+0x10f [f:\rtm\vctools\crt_bld\self_x86\crt\src\crtexe.c @ 583]
0012fff0 00000000 kernel32!BaseProcessStart+0x23

I could see by looking at the exports that there was a function called CoRegCleanup in CLBCATQ.DLL that looked like it could be used to free up this storage before I did my leak checking. Calling it by dynamically getting the function pointer using GetProcAddress did make a difference, but there was still some memory not freed, and I didn’t feel comfortable using an undocumented function in this way.

Then I remembered the magical OANOCACHE environment variable.

This is used to tell the COM runtime not to cache memory used for BSTRs, VARIANTs, SAFEARRAYs, or anything else allocated using CoTaskMalloc. So, I set the variable, re-ran the test and voila! the apparent leaks disappeared. There must be something in CLBCATQ that detects the environment varibale and disables it’s internal cache.

So the moral of the story is; if you’re attempting to reliably track memory usage with IMallocSpy, remember to make sure you have OANOCACHE set, otherwise you will always end up with memory not being freed until late into process teardown.

It’s a fairly unusual edge case (at least, if you’re not being passed VARIANTs from a script language), where the _variant_t helper class attempts to extract a standard COM interface (IUnknown or IDispatch) from a VARIANT that’s being passed “byref”, e.g. has a type or’ed with VT_BYREF.

In this case the code in comip.h uses VariantCopy to “derefence” the pointer – convert it from, say, VT_DISPATCH|VT_BYREF to VT_DISPATCH – but if this succeeds, it then proceeds to use the wrong local variable, varSrc, rather than varDest that it’s just converted. The effect is that it causes an access violation.

Here’s the code snippet in question:

// Try to extract either IDispatch* or an IUnknown* from

// the VARIANT

//

HRESULT QueryStdInterfaces(const _variant_t& varSrc) throw()

{

if (V_VT(&varSrc) == VT_DISPATCH) {

return _QueryInterface(V_DISPATCH(&varSrc));

}

if (V_VT(&varSrc) == VT_UNKNOWN) {

return _QueryInterface(V_UNKNOWN(&varSrc));

}

// We have something other than an IUnknown or an IDispatch.

// Can we convert it to either one of these?

// Try IDispatch first

//

VARIANT varDest;

VariantInit(&varDest);

HRESULT hr = VariantChangeType(&varDest, const_cast(static_cast<const VARIANT*>(&varSrc)), 0, VT_DISPATCH);

if (SUCCEEDED(hr)) {

hr = _QueryInterface(V_DISPATCH(&varSrc)); // Should be &varDest

}

if (hr == E_NOINTERFACE) {

// That failed … so try IUnknown

//

VariantInit(&varDest);

hr = VariantChangeType(&varDest, const_cast(static_cast<const VARIANT*>(&varSrc)), 0, VT_UNKNOWN);

if (SUCCEEDED(hr)) {

hr = _QueryInterface(V_UNKNOWN(&varSrc)); // Should be &varDest

}

}

VariantClear(&varDest);

return hr;

}

Looks like it’s the same in Visual Studio 2005 SP1, but Microsoft are aware of the problem, and hopefully there’ll be a patch soon.

Hope this saves you some time!