Work: Using F# in anger

During working hours (and sometimes beyond) I’ve been developing a large project using lots of F#, a little bit of C# and a bit too much XAML. It’s quite an interesting mix, with technologies that have a vastly different outlook on life: F# with it’s strong, compile-time type checking and XAML with its, err, not-so-strong type checking. In fact, XAML and WPF play so fast and loose with types, that errors in the binding mechanism only happen at run-time and are almost always eaten silently. As you can imagine, this makes for some fun debugging. All I can say is thank goodness for Snoop.

We have a quite a substantial amount of ViewModel code in F# (tens of thousands of lines), much of it using Async to allow us to create responsive UIs without requiring tons of boilerplate code. Generally it works nicely, but I do have some concerns about the amount of thread hopping going on, and how that will affect performance. Especially as we break things down into pretty small pieces of functionality, sometimes just a few simple operations, and each is run on the thread pool. I don’t have any concrete figures to go on here, and I’m hoping we don’t end up in a situation where we sacrifice user productivity for developer productivity. Time (and perfmon) will tell.

Debugging

I’ve done less hardcore WinDbg stuff this year. In fact, a couple of times I’ve been guilty of jumping in with the debugger where it probably would’ve been more effective to take a high-level look first. A case of having a debugger-shaped hammer, and everything looking like a nail.

Moving to 64bit

A very positive thing that happend this year was the move to 64-bit Windows 7 for development at work, meaning that Visual Studio actually stays up without crashing for more than an hour. Previously I’d been having a terrible time: the combination of a couple of large-ish F# projects and 32-bit XP caused frequent VS out-of-memory errors, which was very frustrating. Luckily the move to Windows 7 has resolved it, and I can even run more than a handful of apps at the same time; what a luxury.

Home: iOS

After a long day at work, I finally get to go home, put the kids to bed and then my feet up, and break-out Xcode for a bit of iOS hacking. This year I’ve been feeling like I’m finally getting my head around some of the things that are fundamental in iOS development; from Objective-C concepts like release/retain semantics to iOS specifics, view controllers, table views etc. Previously I’ve been guilty of some copy-and-paste coding, mostly because it’s pretty slow going getting up to speed with new tech (and the myriad iOS APIs) when you’re only getting to do it for a few hours while knackered after work. Well, that’s my excuse anyway.

I’ve been trying to finish off some of the apps that I’ve started, but not shipped. I tend to procrastinate a bit (massive understatement), and of course things aren’t helped by the development ‘fat tail’, where the last 20% – polishing and getting things ‘just right’ – takes at least 80% of the time. My 2010 post on Core Animation flip clocks continues to get a lot of traffic, and I finally bit the bullet and posted a (free!) app that uses it and it’s getting a lot of downloads, which is nice. I also had a look at iAd, which I’ll reserve judgement on for the time being, at the minute it’s not looking particularly lucrative…

As well as iOS stuff, I also took a look at OpenGL. I tend to do this every now and again, each time dipping in a little further and understanding a bit more. This time round it was looking at shaders (in OpenGL ES 2.0) and integrating with Bullet physics. My learning was certainly helped by this excellent Learning Modern 3D Graphics Programming tutorial.

Kinect and F#

Also while at home I finally got around to setting-up by my Kinect to do a bit of F# hacking. In the end I didn’t produce very much, but it was pretty good fun anyway.

Bring on 2012

So, to sum it up: more of the same really, I guess. Although this year there’s definitely been a bit more of a focus on UI development, in both WPF and iOS, which I’ve been enjoying. It does tend to feel a bit schizophrenic, switching between technologies and platforms at the end of the working day. It can be interesting to identify the common themes and significant differences in the two approaches to GUI development.

This year I’m hoping to make my blogging a bit more regular, but who knows if I’ll be able to stick to it… If there’s anything you’d particularly like me to focus on, feel free to leave a comment.

Dark Days

We should bear in mind that in some camps, C++ never went away. The financial, scientific and games programming communities have bravely kept the fires burning.

Why? Primarily performance. In these industries performance is a feature, it’s a key differentiator, and it will help you make money. Can you say the same for an enterprise app (there I go, using that term disparagingly again) that just displays a form interface over a relational database?

However, even in the C++ bastions, the language was getting a bad press. I can only speak for the finance industry, where I’ve been working for the past 10 years, but even here the focus briefly switched away from raw performance to complex (overly-complex, dangerously-complex, as hindsight tells us) trades where getting to market first with something new was key: developer productivity was key. Flexibility and generality was key.

In that case, you don’t want to be dealing with the vagaries of subtle errors, crashes and memory leaks that can easily creep in when you’re using a language as low-level as C++. You just want to be able to release quickly. You probably want to be using a managed language.

In fact, C++ was getting a bad press all over. The functional power that was provided in languages like Haskell and F# – and being hastily added to C# – was missing or awkwardly implemented in C++. The provision of things like the .NET thread pool provided programmers on the CLR an easy way to schedule asynchronous tasks, and it was being touted as the only way to survive in the highly-parallel near-future. Of course, it was possible to do this stuff in C++, it was just more difficult.

Despite this, the programming community widely understood the power that C++ brings, but also that it has to be wielded carefully. Witness the oft-repeated jokes about the fact that when you shoot yourself in the foot with C++, you tend to blow your leg off. There were even tensions within Microsoft, with the pro-managed DevDiv vs the hardcore, native-favouring Windows division.

The New Order

So what’s changed? Herb Sutter’s recent talk covered many of the reasons why C++ is relevant again, and the most obvious reason is the emergence and importance of mobile devices and data centres.

They have a completely different set of requirements than the desktop: they focus on squeezing the best experience out of every available cycle (and corresponding minute of battery life), and the most performance out of every watt of data centre power per degree of cooling. We’re seeing the impact of the smartphone in your pocket and the data-centric web and social media reflecting back onto the languages we use to write software.

Managed languages fall down in some important areas here. They sacrifice memory efficiency to the god of garbage collection and they offer fewer opportunities for aggressive compiler optimisations, including, in the case of .NET, making use of chip-provided performance features like SIMD instructions.

Nowadays in more and more of the computing industry, it’s the hardware that’s driving the development choices, not the other way around.

With resource-constricted platforms, you want to be doing more of the heavy lifting offline, at compile time, not at run-time with a JIT compiler.

How the iPhone showed devs don’t care

While (parts of) Microsoft were busy pushing the managed languages hard, Apple was quietly producing game-changing pieces of hardware that the device-carrying public were buying in their hordes. And they weren’t just buying devices, they were buying apps to play on these devices – in their millions. Lots of developers wanted a piece of this action but – quelle horreur! – You had to use an obscure variant of C to create them.

But – surprise, surprise – developers did it anyway. Luckily, they weren’t on their own, but this time rather than a managed runtime, they had an API to use. And they had to use it; it was, and is, literally a condition of sale that your app uses only the APIs that Apple provide and in the way that they intended. Whatever your view on the fairness of this approach, having a restricted surface area and well defined set of libraries helps developers win back some of the productivity they might otherwise have sacrificed by using Objective-C.

Microsoft must’ve been looking askance at this, wondering how Apple had so effectively switched the model around. MS had been based on the idea of developer productivity being king and the hardware being utterly insignificant. Now developers were writing code in a language that was broadly derided by people in the programming community, yet they were in a virtuous circle where writing apps helped to ship hardware that increased the market for their apps.

We need libraries

Is it possible to find a good balance between the raw power of C++ and the productivity of managed languages? I think it is; and partly it comes down to having a good set of libraries. When I say good, I mean consistent, relevant, well-documented, easily available set of libraries. Can you see where C++ may have struggled here before?

For the low-level, programming task oriented basics, Boost is the answer. Seriously, Boost always seems to be the answer to questions of “how do I do X in C++?” where X is a modern programming technique. Smart pointers, lambdas, higher-order functions… they’re all there.

A few years ago if you were working in C++, you didn’t have many options apart from vanilla STL, Boost, and writing-it-yourself (again). Now, the STL and the language itself is richer and anecdotally, organisations are more receptive to the use of Boost. So hopefully, this combination gives us back access to most of the functionality that’s available to managed developers in their pre-packaged base class libraries.

C++ and Apps

But how is that going to help people use it to produce compelling apps and “experiences” (cough) and specifically, how is Microsoft going to use its developer productivity nous to keep Windows relevant in the new hardware markets?

They need something else; something like the Apple Core frameworks. It looks like this is what Microsoft is talking about with Windows 8 and the Windows Runtime. It’s suddenly not embarrassed to admit that, hey, Windows itself is written in C++, so maybe you shouldn’t have to go through a managed ‘interpretation’ of the existing APIs, one that’s exclusively available to managed languages.

Of course, Microsoft can’t do anything as radical as forcing all developers targeting a mobile or tablet platform to use C++. For them, programmers, rather than any particular hardware, are the bread-and-butter, so they tend to have a slightly less dictator-style relationship with devs than Apple does.

Judging from what I’ve seen from Build, Microsofts chosen approach is to provide language-specific “projections” (their term) from the Windows Runtime layer into a variety of languages, including C++. The native C++ types are mapped into corresponding concepts in the vernacular of the higher level languages; Javascript, C# etc. They’ve gone to some lengths, it seems, to ensure that transitions between levels are efficient. Using C++ at both levels should be the fast path, and of course you can mix it freely with code that, for example, uses Boost or other C++ libraries.

Additionally and importantly, as well as basic types, collections etc, the Windows Runtime also contains all of the rich APIs for accessing app-level functionality; things like image capture, sound and geolocation. So suddenly, all of this stuff is available in all its glory directly from C++ – and other languages too via cheap bridges. What they’ve essentially got is an identical copy of the architecture that exists in the iOS world, while still attempting to keep non C++ developers on-side.

The Return…?

So, have Microsoft managed to bring C++ in from the cold? They seem to have admitted, at least to themselves, that developers will need to use it in order to effectively target some platforms, and they’ve used their tooling experience to make it look and feel like a managed language in their IDEs, as well as leveraging their compiler technology to do some additional useful performance tricks. It feels like C++ was a fundamental concern in the redesign of the new alternative to the creaking Win32 API.

But will the transition be something to make current developers switch to C++, especially those who’ve been used to living in the cosseted managed world? Perhaps, if the Microsoft-based tablet market becomes as compelling – read, large – as Apple’s. Otherwise I don’t see any reason why developers who don’t have specific platform or other runtime requirements would move; for the majority of environments, developer time is still the dominant cost, so developer productivity will still be paramount.

The gap between managed and unmanaged languages has certainly narrowed in that area, but it still exists. C++ is still proving surprisingly relevant, despite its 30+ years, and hopefully its use in new and different areas of computing will feed back into improvements in the language and libraries for everyone’s benefit.

#I @"C:\winnt\Microsoft.NET\Framework\v2.0.50727\"
#r @"Microsoft.Build.Tasks.dll"
#r @"Microsoft.Build.Utilities.dll"
#r @"Microsoft.Build.Framework.dll"
#r @"Microsoft.Build.Engine.dll"
 
open System
open Microsoft.Build.Framework
open Microsoft.Build.Utilities
open Microsoft.Build.Tasks
 
do
    let t = 
        ResolveAssemblyReference(Silent = true,
            Assemblies = [| upcast TaskItem( ItemSpec="class1" ) |],
            AllowedAssemblyExtensions = [|".dll"|],
            AppConfigFile = "c:\\dev\\myapp\\myapp.exe.config",
            SearchPaths = [|"{GAC}"|]
            )
    match t.Execute() with
    | true  -> printf "%A\n" t.ResolvedFiles
    | false -> printf "Task failed"

The output is:

[|C:\WINNT\assembly\GAC_MSIL\class1\1.1.8.0__120bce90f68b861a\class1.dll|]

The code makes good use of F#’s named constructor arguments, which map to property setters on the object we’ve just created. We can also create the arrays of TaskItems and strings fairly easily, although an explicit upcast is required in order to obtain the ITaskItem (F# doesn’t upcast implicitly like many object oriented languages).

I’ve just discovered that Jomo Fisher wrote about exactly the same thing back in May 2008! He mentions using an object expression in order to provide an implementation of IBuildEngine, which I’m getting away with by setting the Silent property on the ResolveAssemblyReference task, meaning that it never attempts to obtain the logging context from the build engine.

Another gotcha here is the fact that versions of various interfaces including IBuildEngine and ITaskItem exist in both the 2.0.0.0 and 3.5.0.0 versions of the Microsoft.Build.Framework assembly. Be careful that you reference the correct version for the task that you’re calling. Normally it will be the 2.0 versions, in which case you’ll need to use explicit file references in your script, adding the directory to the library include path:

#I @"C:\winnt\Microsoft.NET\Framework\v2.0.50727\"
#r @"Microsoft.Build.Framework.dll"

Rather than partially qualified assembly references:

#r @"Microsoft.Build.Framework"

As the latter will resolve to the latest installed version of the assembly, 3.5, which is not what you want. Look out for errors like:

System.InvalidCastException: Unable to cast object of type ‘Microsoft.Build.Utilities.TaskItem’ to type ‘Microsoft.Build.Framework.ITaskItem’.

Alternatives

The Tools|Options MSBuild verbosity setting

I’m making use of a few F#/.NET idioms to make the process easier. For instance, sequences (.NET’s IEnumerable) are used as the source of uniformly distributed random numbers. Also, to verify that the resulting numbers are actually normally distributed, we can easily use existing WPF/silverlight controls to visualise the values direct from within Visual Studio, in a manner similar to this previous post – but without having to write the plotting code ourselves.

Implementation

So firstly, we create the random number stream:

let rr = System.Random(System.DateTime.Now.Minute)
let rands = seq { while true do yield rr.NextDouble() }

As you can see rands is an infinite sequence. As such, you should be wary of the “eager” functions from the Seq module that attempt to consume the entire sequence at once, they’ll obviously cause a problem here. Also notice that I’ve picked a somewhat dubious seed for the random number generator, as this is only for demonstration purposes.

In order to obtain normally distributed random numbers I’m using the Box-Muller transform. I started off with an implementation transcribed from VBA (from Wilmott et al) – yes, yes, I know – but of course this used mutable references. Not nice. So I reworked it a little to use Seq.pick, which takes items from the sequence while a predicate returns None:

let boxMuller _ =
    let (x,dist) = 
        rands
        |> Seq.pick (fun x ->
            let x = 2.0 * x - 1.0
            let y = 2.0 * ((Seq.take 1 rands) |> Seq.hd) - 1.0
            match (x * x + y * y) with
                | dist when dist < 1.0 -> Some (x,dist)
                | dist -> None
            ) 
    x * Math.Sqrt(-2.0 * Math.Log(dist) / dist)

It’s not particularly efficient as we’re consuming two uniform numbers to generate one normal one – hence the additional Seq.take within the pick. Of course we could change the function to return the 2 generated numbers as an extra element in the tuple.

Next, let’s generate some numbers, and get them in a suitable form to display:

let makeData _ =
    Seq.init 100000 boxMuller
    |> Seq.countBy (fun d -> 
        let d = decimal d
        Math.Round(d, 1))
    |> Seq.sortBy fst

We initialise a sequence with an arbitrary amount of numbers (100000, enough to get a good distribution, hopefully), then use Seq.countBy to do a naive grouping that allows us to see how the numbers are distributed, i.e. counting how many numbers fit in each 0.10 bucket. We then sort the output by bucket.

Visualisation

Now we can display what we’ve got. We can create a simple WPF object-model using imperative code, including the very useful chart control from the WpfToolkit. The chart control enables us to add a series and set our seq<decimal * float> directly as the series ItemsSource, which is the standard way to specify the items for a collection control. If you specify the tuple Item1 and Item2 properties for the bindings, WPF will be able to access the data from each element, and that’s all you need.

    let series = 
        ColumnSeries(
            IndependentValueBinding = Data.Binding("Item1"),
            DependentValueBinding = Data.Binding("Item2"),
            ItemsSource = makeData ())
    let chart = Chart()
    chart.Series.Add series
    Window(
        Name="Plot",
        Title="Normally distributed random numbers",
        Width=900.0,
        Height=700.0,
        Content=chart,
        Visibility=Visibility.Visible)

Here’s the resulting output:
As I’ve mentioned before, this is a fantastically immediate way of doing numerical development in Visual Studio. You can enter all of this code directly into F# interactive within VS, iterate, and quickly see the effect that your changes have. It’s just as interactive as using the graph features in Excel, and the resulting code is much more easily reusable.

XD5RRV3XDNHN

I remembered seeing a post on that Windows SDK blog that I’d come across relating to a similar issue with the XAML editor (I’ve been doing some work with WPF recently, more on that in a later post) so thought I’d try the steps they recommend, in short, re-registering TextMgrP.dll:

regsvr32 "%CommonProgramFiles%\Microsoft Shared\MSEnv\TextMgrP.dll"

…and all my problems went away. Hope you find this useful.

which causes “error FS0072: Lookup on object of indeterminate type based on information prior to this program point”, to the correct:

(These are obviously simplistic examples!)

So I broke out the Visual Studio macro editor for the first time in a while, and put together something to toggle brackets around the currently selected text. It’s naive, but, combined with Shift+Alt+Left Arrow to select the previous word, it’s effective:

Copy this text into a module within your macro project, and assign a suitable keystroke using Tools|Customize|Keyboard.