1. Exclamation marks in dialog box messages
Look, it’s probably the 50th time I’ve seen this message today. The fact that this application “Cannot connect to database!” is no longer a surprise. You may have noticed that most professional software uses a neutral tone for its communication with the user. Try that. Also:
1a. Double negatives in confirmation dialogs
“Are you sure you don’t want to lose your changes?” Err… what? No. I mean YES. Oh sh*t. Any dialog that requires you to stop and try to parse the question in order to work out if you’re about to destroy several hours of work is not doing its job. It’s getting in the way of you doing your job. In fact, convoluted messages are such a serious issue that Microsoft even tried to help developers to help their users by introducing a whole new kind of dialog box in Vista: the question/task dialog. Good luck with that.

2. No tab ordering defined\mouse only navigation
Because no-one’s ever going to use the keyboard to navigate the zillion controls in your data entry app, are they? This one actually surprises me, because I’d have thought that developers would’ve needed to navigate quickly through the application while they’re writing it. Well, that doesn’t seem to be the case. Pretty much all commercial apps are good counter examples. I don’t mean to hold up Microsoft Office as a paragon of UI virtue, but they definitely do the “alternate way of navigating everything” thing well. Everything you need can be reached by both keyboard and mouse. Unplug your mouse and try that with your favourite piece of in-house software and see how you get on.

3. Group boxes around everything
This is a bit of a WinForms specific one. The clue is in the name: group boxes are for grouping logically related controls, not for providing a kewl recessed border around every single one of the controls in your dialog. Don’t kid yourself that this is doing anything other than using up some valuable screen real estate. (See if you can spot another pet peeve in the example dialog, too).

4. Icons created in the IDE
Look, Visual Studio’s a really good integrated development environment, but it ain’t no Photoshop. Don’t try and use it to create icons. And while you’re at it, please don’t make icons consisting solely of the name of your application (inevitably an acronym) in pixel font and primary colours. Oh, and don’t just steal various icons from another application, unless you’re going to steal the whole lot; one of the key visual aspects of a good UI is consistency. Mixing your hand-drawn 4-bit icons with the glorious 32-bit shiny ones you borrowed is going to be jarring. In fact, why not go the whole way and get someone who can actually draw to create your icons for you? After all, you wouldn’t have someone who wasn’t a programmer writing the code, would you…?

5. Data grids
Otherwise known as the “Excel is the pinnacle of user interface design” disease. Break the habit. Generally, the more types of controls that are embedded in your grid, the less likely that it’s the right kind of interface paradigm. Yeah, I’m looking at you, person embedding a calendar control, drop down box, graph, slider and checkbox in each row of a data grid. And whatever your 3rd-party grid provider of choice says, it’s not going to do screen redraw performance any good, either.

6. Not implemented message boxes
Ahh, the GUI equivalent of source code TODO comments. Of course, it’s an in-house software give-away; no commercial (desktop) software would be brazen enough to ship with bits of functionality dangling from the stumps of buttons and menu items. Would it? Feel free to provide counter-examples if you have them.

7. Excessive use of dialog boxes
Warning: dialog boxes are considered harmful (to usability). They’re the equivalent of restraining your user by force and preventing her from moving until she answers your question. That might be OK for matters of life or death (i.e. data loss), but not otherwise. Every time you find yourself about to add a new message box, stop, take a deep breath and consider whether it’s really necessary.

So, if you’re a victim or one, many, or all of these user interface faux pas, all I can say is: sorry. I’ve been responsible for doing at least one of these things myself over the years. Consider this post repentance for my user interface sins.

I heart spreadsheets

Unfortunately, given the ever-increasing focus on accountability and scalability in these domains, Excel is also an increasingly inappropriate platform for delivery of this sort of functionality. How can organisations ease the transition away from this environment to something more easily manageable, while retaining it’s positive benefits?

Disclaimer: These are my own views and not those of my employer. The scenarios I describe are fictitious and not based on actual practices at my place of employment. Any similarity to actual events and behaviours is coincidental.

I see two main aspects of Excel’s operation that prove valuable in general, and in finance in particular: visibility and immediacy.

Visibility

From an end user’s point of view, one of the most positive aspects of Excel is it’s transparency. Even in an environment where complex processing goes on inside a plugin – say, an Excel user defined function (UDF) packaged as an XLL – it’s still only in the scope of a single function call in a single cell. You continue to get good visibility of the inputs and outputs from the function.

You can also get a good higher-level view of how the different parts of the calculation fit together. Either by inspection, editing the formula to quickly see which cells it references, or by using the formula auditing toolbar to get an overlay of dependent and precedent functions, you can see roughly how various parts of the calculation fit together.

This, combined with the logical separation of functionality into separate worksheets, gives you the impression of understandability.

The problem is that it’s just that – an impression – there is no structure enforced by the worksheet separation, you’re free to link between them as you see fit, and this can result in an extremely complex and virtually indecipherable “flow” of calculation within a workbook.

Immediacy

Luckily, to offset the complexity we have another important aspect of Excel’s usability: it’s immediacy.

Trading users can very quickly modify calculation trees to perform ad-hoc additional calculations such as applying spreads, and to gauge the effect of market moves on their portfolio, i.e. “what-if” scenarios.

Structuring users can fairly easily – given an existing set of functional pieces – assemble a model to price a complex new trade type. Although this is a much less common use case now that the demand for getting new structured products to market quickly has essentially vanished.

As well as this type of gross modification, it’s also possible to just repeatedly re-evaluate some small part of the calculation to quickly determine how it is affected by other changes.

The immediacy of the Excel environment, combined with the use of the Visual Basic for Applications (VBA) scripting language, enables both business and IT users to rapidly build out new sheets to perform ad-hoc tasks. Rapid application development, or RAD as it’s known, is all well and good for performing some transient, ad-hoc calculation for limited use, but unfortunately these tools have the habit of becoming indispensable parts of an organisation’s workflow. That’s when you start to see serious problems with scalability and maintainability.

The problem?

So the fundamental tension here is that exactly the same features which make Excel useful to traders, structurers, etc., also make it highly unsuitable for use in these environments. You really don’t want traders to be able to subtley alter the flow of the calculation, as much as they might like to. Excel should really be a read-only view on some underlying trade data source and calculation engine, but this isn’t always how it’s implemented.

You also want to provide some flexibility – the ability to experiment, explore and examine – but within certain constraints and without compromising future requirements. Being able to price a trade is a good start, but you will subsequently need to book it easily, re-value it reliably and potentially risk manage a large portfolio of them.

Moving away from Excel

Many houses provide underlying quant analytics implemented in C++, with a native C++ interface and a variety of other (sometimes auto-generated) wrappers that make it accessible in other environments. Ideally, the amount of actual functionality in these wrapper layers should be kept to an absolute minimum. It should be commodity code that can be regenerated as and when required.

However, even if this is the case, unless you have the ability in the underlying code to fan-out the processing you will still be constrained by the operating system process limits, e.g. 4gb address space.

The common alternative to hosting pricing and risk management within Excel is to write new dedicated applications to perform the calculations and (hopefully) provide a means to inspect and troubleshoot it.

Unfortunately these systems are built by teams who aren’t particularly familiar with the way the front office uses Excel. It’s often the case that spreadsheet development teams and systems development teams don’t interact; the former tend to be based on the trading desk while the latter are in another part of the building.

So the end result can end up being systems that work in a way developers feel comfortable with, rather than the way business users would like them to work. This can make the system seem like a black box to it’s users. And to make it worse these are the users who are accustomed to Excel’s high levels of transparency.

Although the system may be enforcing the fact that trade value is strictly a function of the trade, market and static data, having some transparency can still be valuable in giving skilled users confidence in the underlying calculation. They can apply their not-inconsiderable intuitive abilities to quickly gauge the correctness or otherwise of the valuation. It can help the quants move away from their native Excel/Matlab-style iterative development environment. And when groups need to work together to reconcile differences, everyone can benefit.

The problem? This amount of transparency has to be built into the system or the quant library pretty much from the ground up.

Solutions

One possible approach is to make your system a general purpose calculation engine. This has some appeal, but you have to be careful about sacrificing performance for generality. You might be tempted to try exactly matching the semantics of Excel’s calculation and type system. This can be a nightmare, with all sorts of limitations and nasty edge cases. For example, the fact that errors are a simple numeric value with no facility to carry extra information is crippling, and also the lack of support for objects as first class values in the type system.

However you choose to implement your calculation engine, ensure that it’s loosely coupled enough to be run out-of-process or across machine boundaries without code changes.

Here are some bullet-point takeaways for easing the transition:

1. Provide a means for users to examine intermediate calculations, and surface it in the UI.
   This will give your intermediate and end users confidence in the calculations
2. Provide a means to capture and re-play valuations, including all their inputs and outputs.
   This will assist in reconciling valuations, and in passing snapshots around between groups.
3. Provide the ability to drive the system programatically.
   Along with persistent valuations, this will help in regression testing

I hope this post has served to give you some insights into the potential challenges of moving users away from Excel into more robust systems, why some of those users may be reluctant to move, and how you can help to ease the transition. And of course, let me know via the comments if you’ve got any observations on the subject.

A bit of background: as you may know, if you add functions to the worksheet or workbook code in Excel then they appear as callable methods on the objects themselves. This is achieved with the use of dynamic dispatch and IDispatch. For example, creating a workbook with this function in it’s VBA code:

Public Function Foo() As Double
    Foo = 100
End Function

Means you can call it like this:

MsgBox CStr(ThisWorkbook.Foo)

As well as being able to call it like this from within the Excel session (i.e. in other VBA code in the process), you can also access it externally using the COM object model that Office applications expose. For instance, you can use VBScript:

Dim excel, wkb
Set excel = CreateObject("Excel.Application")
Set wkb = excel.Workbooks.Open("a.xls")
WScript.Echo wkb.Foo


Or, more interesting, using F#:

The key part here is that we’re using wks.GetType() to get a managed representation of the unmanaged Excel COM interface. Under the covers this is creating a runtime callable wrapper (RCW) to wrap the worksheet COM object.

However, the problem we were seeing was that opening multiple sheets resulted in failures to call the method in certain situations. Although the VBA signature was exactly the same in all of the sheets, it seemed that opening b.xls after a.xls, would fail; returning null when we expected it to return a value. If we opened c.xls after b.xls, it would fail in a different way; never actually making it to the body of the function. Very odd.

My first suspicion was that it was somehow related to COM object vs .NET object lifetime. This is quite a common problem whens invoking Excel using managed code. It’s bad mixing COM and .NET anyway; generally deterministic, reference-counted lifetime semantics don’t play well with the garbage collector. Throwing an app with a full-blown UI being managed as COM object into the mix just complicates matters further. Anyway, it’s been widely discussed, so I won’t say any more about it here; suffice to say that calling Marshal.ReleaseCOMObject and GC.Collect got us to the point where we could see the Excel process terminating, so we knew that the failure wasn’t due to some state being cached inside there.

So we concentrated on different aspects of the problem:

• Given the pattern of failures, it seemed that the order of opening the sheets and calling affected the outcome. This hinted that something was persisting betweeen calls, but not in the client (Excel) site.
• The code had previously worked when written in VBScript, so there was nothing intrinsically wrong with the operations we were performing.

This seemed to strongly indicate that something was being cached at the .NET level. And the major difference between the .NET code and the VBScript was that the former used Type.GetType() on the worksheet object to get it’s managed representation, while the latter used the IDispatch directly.

So it looked like GetType() was caching some information about the particular IDispatch implementation that it encountered first, then reusing that for subsequent worksheet implementations which actually had a slightly different layout, i.e. although they also had the Foo function which we were trying to call, they had a different set of other dynamic functions too.

After a bit of digging about I uncovered this gem: the mapping between interface pointers and runtime callable wrappers. Which seemed to describe exactly what we were seeing. The first time through the loop, the runtime is asked to get the type and is given a COM pointer, for which it creates a RCW that we can use to invoke Foo. The second time through the loop, the runtime thinks it’s seen the object before, so rather than perform the expensive operation of creating the RCW again, it just returns the original one. The problem is, the underlying COM object is different!

So, in order to prevent the runtime for trying to cache the RCW, we need to use Marshal.GetUniqueObjectForIUnknown and that does the trick nicely. We first need to get an IUnknown for our object, than convert that back into an object, which is actually a RCW:

Although it’s less efficent, at least the code works, and it finally allows us to call the dynamic methods on the workbook object from F# .NET.

It will be interesting to see how dynamic in .NET 4.0 addresses this kind of issue.

Why? Well, because it’s too easy to use them as an excuse for not defining your shared library interfaces properly.

The reason this is on my mind recently is that several hundred, yes, you heard that right, several hundred DLLs have been released by my group over the last, ooh, 10 years or so. They are all still in use. Each of them has burned into it a copy of the library that deals with interfacing with Excel. That means each of these has it’s own little internal copy of the current state-of-the-art. The problem with that is; the state-of-the-art moves on. And how do you go about updating the DLLs that are already in production? You have to re-release them. In an environment where thes DLLs are used for marking the profit and loss on a large derivatives trading book, that’s not a small undertaking. And it’s made worse if, say the DLL in question was last released with a different version of the compiler.

My approach would be to refactor this shared static library (.lib) into a stand-alone DLL.

At this point, people start saying “oh, but then you’ve got a single point of failure, if you release a broken version of that DLL, everything will stop working!”. Not exactly a compelling argument. If the functionality of the DLL is well defined, and there are well known entry points it should be easy to put together a comprehensive black-box test suite. In fact we already do that with all our other DLLs (COM servers). The fact that this shared library *isn’t* a DLL has meant that it’s fallen through the testing cracks; another good reason to refactor it.

The internal interface to the shared library is already relatively well defined. It has a set of header files that define all of the functions and classes that are consumed by others. It’s a relatively small step to compile it as a DLL, rather than a static library. The problem then becomes one of maintenance, dealing with the inevitable changes to the external interface in a backwardly compatible way.

And that’s the problem. It requires some effort. Elsewhere in our codebase we use COM as a magic cure-all for avoiding having to deal with versioning: interface immutability rules. All interfaces are public, no published interface ever changes, object identity is based purely on interfaces supported. If you haven’t got these crutches to rely on, then you have to enforce the rules yourself, which can be both logistically and technically difficult when you’re dealing with C++.

But it’s not impossible. And I really think it would be better than having hundreds of DLLs all containing subtley different versions of the same code, and being unable to change behaviour across the board without having to build, test and release them all.

Maybe you’ve got a different opinion?

bu EXCEL!MdCallBack+0xa880 "dds @eax+0x4 L1; g"

You’ll need to adjust the offset for other versions of Excel – and I haven’t tried it yet with 2007. Assuming you’ve got symbols available for the XLLs being called, you’ll get something like this:

0013bc50 15109730 addin1!addin1_function1
0013bc50 12c0e918 addin2!addin2_anotherfunctions

This technique can be useful when troubleshooting – to identify the last addin call being made before a failure perhaps – and is also quite interesting to just watch and see the pattern in which your XLL UDFs get called.

The framework itself is written as a COM server that lives inside the Excel process, and controls it via automation, although it also works out-of-process. It supports IDispatch, so is usable from scripting languages, too. For instance I’ve put together an HTML/JavaScript based UI that can invoke the framework directly from a web browser.

The main problem is the fact that Excel is the host application. There are lots of strangenesses around the Excel automation API, but it still does a pretty good job, in my opinion, of exposing so much functionality. The real issue is that it’s a black-box. As is the VBA engine (VBE.DLL). This can make it difficult to track down problems, especially where I’m crossing backward and forward between native C++/COM code and VBA callbacks.

Still, this is a satisfying project, because it helps to formalise people’s use of spreadsheets, without them even realising. It’s easier for end-users to drive; they select what type of product they want to value from a simple GUI and the framework does the rest. And from a development point of view it means the analytics group have fewer monolithic and difficult to maintain spreadsheets. And of course, forcing us to think about the model in component terms also means that if we should ever move away from Excel (!) we would, at least in theory, have an easier migration path.