Fanboy? Moi...?

See you back here in September!

Cornell Box in the evening

What is a “global illumination renderer”?

In short; a global illumination renderer attempts to simulate the way that light behaves in the real world. It bounces rays of light around the scene, noting which surfaces are hit, and how each of them affects the ray. Of course real light travels fast – at about the speed of light, in fact – and our simulated rays can’t match that, so although the resulting effect looks natural, it’s not a very efficient way of getting there. Graphics systems that need to be near real-time (i.e. games), use a variety of short-cuts to try and get the same effect without requiring the same amount of number crunching.

Even this implementation is forced to use a couple of techniques to improve performance, I won’t go into them here, as they’re explained pretty thoroughly on the Minilight site.

Interestingly the algorithm uses a Monte Carlo technique to simulate the random path of the ray through the scene and the same technique is used in finance to model how the price of an asset will change through time.

Translating to F#

Room with sun and light, 800 paths per pixel

Some of the ML constructs caused warnings from the F# compiler as I didn’t use the explicit ML compatibility mode. For instance, the use of the ^ operator to concatenate strings:

warning FS0062: This construct is for ML compatibility. Consider using the '+' operator instead. This may require a type annotation to indicate it acts on strings.

Some other changes included:

• Using dot rather than # notation to access class members
• Using square bracket for array element access (array.[n] rather than array.(n))
• Removing let...and for non-recursive bindings

There was also the opportunity to make use of some of the .NET framework classes, things like System.Drawing.Bitmap to write pixels directly into a bitmap and then save it into a .PNG file. Much easier than trying to find something to read the .PPM file that the original version emits.

Potential improvements

One of the most obvious improvements that could be made is to make use of the parallelism support in F# and .NET. The algorithm itself is easily (if not embarrassingly) parallelisable, given that it iterates over the pixels in the output image, and performs independent operations on each of them. Unfortunately I haven’t had a chance to look at this yet – and the Windows VM I’m using only has a single processor anyway, so on a purely selfish note, I wouldn’t see any immediate performance improvements. It should be as simple as changing the pixel loop to use Parallel.For, or creating an async computation expression.

Something else to note: I err’d on the side of caution when it comes to saving intermediate images, so every iteration it creates and writes the file. This will be doing lots of allocation and strictly unnecessary work, especially for simple scenes.

Given this approach is so completely compute bound it would be interesting to see how moving some of the calculations to a GPU would affect it. It might even be possible to use quotations for some of the inner loops and then generate Nvidia intermediate language (PTX) from them – that’s something I’ve been wanting to do for a while.

I did run the code through the Visual Studio 2010 profiler to get a rough idea of where the time was being spent; it looked like the majority was in the leaf of the calculations where numerical operations where being performed on the vectors, which is consistent with what we’d expect.

Use the source

So, here’s the source. No warranties implied, use at your own risk, your mileage may vary, all errors are mine etc, etc.
minilight_fsharp

How can you screw-up creating an electronic version of an existing sign?

Here’s the offending item:
It’s a replacement for a combination of magnetic labels and hand scrawled messages that convey the current state of each of the London underground lines. They’re generally one of “good service”, “minor delays”, “severe delays”, or “oh no, you’ll never get to work”.

For the majority of people, their only interaction with this system is walking past, early in the morning, when they’re not at their sharpest, and trying to figure out if their tube line is working. As such, if there was a magnetic label stuck on your line, you knew things weren’t good. Given the lines themselves are colour coded, it’s easy to determine which line you’re interested in. Bear in mind that people aren’t going to get closer than about 2 metres to the board.

And herein lies the problem: by replacing the analogue board with a digital version, it’s no longer possible to ascertain by the physical presence of a label or writing, what state the line is in.

Let me say that again. It’s virtually impossible to tell, even from relatively close, if there are delays. Take a look at that screenshot (dodgy quality as it is) and see if you can tell which line has “minor delays”. Yeah, not easy is it? Now try doing that as you’re briskly walking past at 7.46am.

Underground line status (detail)

I’m surprised that they’ve relied solely on text and haven’t used any other means of distinguishing the significant information. Some other things that could’ve been used:

• Colour (of the text itself, the background, or some indicator. Need some inspiration? How about taking a look at something way out there, like, say, traffic lights)
• Size
• Weight (using font weight to indicate relative importance)
• Icons (or some other kind of visual indicators, nothing flashy)

I’m sure there are a whole load of advantages to having this digital system; most notably that it’s connected, and can be automatically and centrally updated. If only they’d spent any time on thinking about how people were actually going to use it!

Harry Beck would be turning in his grave.

What is .NET type equivalence?

Described at a high level here, .NET 4.0 type equivalence essentially gives you a way of indicating that different .NET types represent the same underlying COM type and is most commonly used in COM interop scenarios. One of the reasons for its introduction is to save developers from having to ship large interop DLLs with their software, e.g. the multi-megabyte Microsoft.Office.Interop. Instead the compiler can inline the definition of any types used, and mark them appropriately as representing the original COM types.

The error

We noticed that whenever we built and ran an application that referenced a DLL using .NET 2.0, it worked. Doing the same thing with .NET 4.0 caused a BadImageFormatException.

Unhandled Exception: System.BadImageFormatException: Could not load file or assembly 'mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089' or one of its dependencies. An attempt was made to load a program with an incorrect format.
at X.Main()


Let’s dig!

So, the BadImageFormatException doesn’t actually tell us much. Let’s break out WinDbg and see what we can find. Running the faulting app we can see several C++ exceptions before the CLR exception is thrown:

(178c.790): C++ EH exception - code e06d7363 (first chance)
...
(178c.790): C++ EH exception - code e06d7363 (first chance)
(178c.790): CLR exception - code e0434352 (first chance)

I changed the exception handling settings to stop on C++ exceptions (sxe eh) then ran again to see where things were going wrong. It stopped here:

0:000> kp
ChildEBP RetAddr
0012d15c 79084c0f KERNEL32!RaiseException+0x53
0012d194 793371be MSVCR100_CLR0400!_CxxThrowException+0x48
0012d5e4 79455cae clr!EEFileLoadException::Throw+0x1a8
0012d634 794558d2 clr!CompareTypeTokens+0x200
0012d6b0 791b5c00 clr!IsTypeDefEquivalent+0x102
0012d6d4 791b2ca8 clr!MethodTableBuilder::CheckForTypeEquivalence+0x94
0012d7ac 791b27c9 clr!MethodTableBuilder::BuildMethodTableThrowing+0x60d
0012d9a4 791a4578 clr!ClassLoader::CreateTypeHandleForTypeDefThrowing+0x88e

Interesting. Notice how the call stack contains some .NET 4.0 specific methods relating to the new type equivalence feature. We’re hitting a new code path, which is consistent with the fact that running against a down-level CLR works.

After a bit more toing-and-froing, I discovered that the C++ exception is thrown when clr!MDInternalRO::IsValidToken returns an error. By disassembling the function we can see it’s just looking at various bits in the token value, and it decides that the value passed (0×02000000) isn’t valid. Looking at the output from ildasm that token doesn’t appear anywhere. And if we add a dump of the value, we can see that it indeed doesn’t look like the other tokens:

0:000> bu clr!MDInternalRO::IsValidToken "dd esp+8 L1; g"
...
0012f5a8  02000001
0012f31c  06000001
0012f2c0  02000002
0012f0f4  02000002
0012ebe4  01000001
0012e944  23000001
...
0012d5f4  02000000
(18ec.1ec8): C++ EH exception - code e06d7363 (first chance)

What’s the culprit?

So it looks pretty conclusive; the DLL contains something that the CLR isn’t expecting. But what? It’s time to break out the oldest tool in the troubleshooting box: the binary chop!

Eventually I got the referenced DLL down to only a single simple construct. Can you guess what it is? A global literal value. A real global value, one that isn’t even part of a type. Crazy huh? In IL it looks like this:

.field public static literal valuetype Test.MyEnum LiteralValue = int32(0x00000001)

It’s a literal value of an enumerated type. That’s important: using a value of a simple type (say int32) does not provoke the error.

Now, I wasn’t even sure that this is a valid IL construct, but according to the ECMA IL spec, specifically partition II, section 15, it is:

The CLI also supports global fields, which are fields declared outside of any type definition. Global fields shall be static.

So it looks like we’re not doing anything illegal, backed up by the fact that the .NET 2.0 CLR can make use of it without a problem.

Interestingly, there’s another aspect that influences whether this code path is hit. As mentioned above, type equivalence is intended for use with interop libraries. As such, it only kicks in if your referenced assembly is marked with the PrimaryInteropAssembly attribute, e.g.:


.custom instance void [mscorlib]System.Runtime.InteropServices.PrimaryInteropAssemblyAttribute::.ctor(int32,int32) = ( 01 00 01 00 00 00 00 00 00 00 00 00 )


The Fix?

The issue is currently with Microsoft product support. Let’s see what they come up with; is it too esoteric for a hotfix…?

The Repro

Here’s some code and instructions on how to repro the problem.

1. Build the IL into a DLL using ilasm.
   "c:\WINNT\Microsoft.NET\Framework\v2.0.50727\ilasm.exe" /dll Test.il /output=Test.dll
2. Build the application into a .NET 4.0 EXE that references the DLL
   "c:\winnt\Microsoft.NET\Framework\v4.0.30319\csc.exe" TestConsumer.cs /reference:Test.dll
3. Run the resulting TestConsumer.exe application and you’ll get the exception

Test.il

.assembly extern mscorlib
{
.publickeytoken = (B7 7A 5C 56 19 34 E0 89 )
.ver 2:0:0:0
}
.assembly Test
{
.custom instance void [mscorlib]System.Runtime.InteropServices.PrimaryInteropAssemblyAttribute::.ctor(int32,int32) = ( 01 00 01 00 00 00 00 00 00 00 00 00 )
.hash algorithm 0x00008004
.ver 1:0:0:0
}
.module Test.dll
.imagebase 0x00400000
.file alignment 0x00000200
.stackreserve 0x00100000
.subsystem 0x0003
.corflags 0x00000001

.class public auto ansi sealed Test.MyEnum
extends [mscorlib]System.Enum
{
.field public specialname rtspecialname int32 value__
.field public static literal valuetype Test.MyEnum Zero = int32(0x00000000)
.field public static literal valuetype Test.MyEnum One = int32(0x00000001)
}

TestConsumer.cs

class X
{
    static void Main()
    {
        var v = Test.MyEnum.Zero;
    }
}

_toplayer = [CALayer layer];

Splitting images

Now we’ve got the layers, we need something to but into them. I didn’t want to create lots of pre-processed images for the top and bottom sections of each number, so instead I wrote a routine to split the images in code, drawing each half into an image context and grabbing a UIImage from it. This is then stored in an array for later use. We can get any of the elements from the array and set it as the content of the layer as required.

// The size of each part is half the height of the whole image
CGSize size = CGSizeMake([img size].width, [img size].height/2);
 
// Create image-based graphics context for top half
UIGraphicsBeginImageContext(size);
 
// Draw into context, bottom half is cropped off
[img drawAtPoint:CGPointMake(0.0,0.0)];
// Grab the current contents of the context as a UIImage 
// and add it to our array
UIImage *top = [UIGraphicsGetImageFromCurrentImageContext() retain];			
[_imagesTop addObject:(id)top];
 
// Now draw the image starting half way down, to get the bottom half
[img drawAtPoint:CGPointMake(0.0,-[img size].height/2)];
// And store that image in the array too
UIImage *bottom = [UIGraphicsGetImageFromCurrentImageContext() retain];			
[_imagesBottom addObject:(id)bottom];
 
UIGraphicsEndImageContext();

Now our _imagesTop and _imagesBottom arrays contains all the images we need, let’s get ‘em moving.

Applying animation

The axes and anchor point of a CALayer

_fliplayer.anchorPoint = CGPointMake(0.5, 1.0);

I use the CATransform3DMakeRotation function to create a transform around the X axis using the vector [1,0,0]. For the first part of the motion, we go from 0 radians to pi/2 radians (0 to 90 degrees), using an explicit CABasicAnimation object:

CABasicAnimation *topAnim = [CABasicAnimation animationWithKeyPath:@"transform"];
topAnim.duration=0.25;
topAnim.repeatCount=1;
topAnim.fromValue= [NSValue valueWithCATransform3D:CATransform3DMakeRotation(0.0, 1, 0, 0)];
float f = -M_PI/2;
topAnim.toValue=[NSValue valueWithCATransform3D:CATransform3DMakeRotation(f, 1, 0, 0)];
topAnim.delegate = self;
topAnim.removedOnCompletion = FALSE;
[_fliplayer addAnimation:topAnim forKey:[NSString stringWithUTF8String:k_TopAnimKey]];

Drawing layer content

Unfortunately a CALayer doesn’t have a “back”, so creating the flip layer is not as easy as it could be. We need to add a step in the process to change the image displayed by the flip layer half way through its fall. We can do this in a variety of ways, I’ve chosen to set a delegate object to perform the drawing of the layer contents:

[_fliplayer setDelegate:_layerHelper];

This means our delegate object (_layerHelper) has its drawLayer method called whenever the layer needs to display its contents. Be wary though, this may not be as often as you think! Core graphics aggressively caches the contents of the layer, and will only call your delegate when absolutely necessary. You can force it to happen by calling setNeedsDisplay, which invalidates the contents of the layer:

[_fliplayer setNeedsDisplay];

I encapsulated this in the helper object itself, by providing a custom implementation of the isTop property setter that calls setNeedsDisplay. At least that way callers don’t have to be aware of this requirement.

The drawLayer method itself simply draws one of two images into the context, depending on whether it’s in ‘top’ mode. A slight further complication is that when drawing the bottom part, the image needs to be inverted to display upside down. This is achieved by applying a translation and scale to the context before drawing the image:

CGContextSaveGState(context);
CGContextTranslateCTM(context, 0.0f, r.size.height);
CGContextScaleCTM(context, 1.0f, -1.0f);
 
CGContextDrawImage(context, r, [_imgTop CGImage]);
 
CGContextRestoreGState(context);

Avoiding implicit animation

I’ve tried to avoid using implicit animation here, instead using explicit construction of CABasicAnimation objects, because we need some control of when things happen, and the order in which they happen. Normally, it would be easier to just set some property of the view and let the OS manage animating it. Many properties have default transition that are applied whenever they’re changed, so it takes some additional effort to make them change immediately.

In our example we change the contents of the top half and show the flip layer, but we need that to happen immediately so we disable actions within an explicit transaction that we then commit. This gives us the control we need, and avoids the default behaviour of the transition occurring when the message loop next runs.

[CATransaction begin];
[CATransaction setValue:(id)kCFBooleanTrue
				forKey:kCATransactionDisableActions];
_toplayer.contents = (id)[[_imagesTop objectAtIndex:[self getNextIndex]] CGImage];
_fliplayer.hidden = NO;
[CATransaction commit];

Getting some perspective

Although the images I’ve used to illustrate this post are isometric, what we really want is a front-on perspective view, so that the falling piece seems to stick out. For this we have to set a sublayerTransform on the view’s inherent layer, by setting an individual element on a transform:

CATransform3D aTransform = CATransform3DIdentity;
float zDistance = 1000;
aTransform.m34 = 1.0 / -zDistance;	
[self layer].sublayerTransform = aTransform;

This gives us a much more realistic look.

Putting it together

Wow, so now we’ve got some appropriately sliced images, some layers and some animations. Let’s put them together to get the final effect.

The animation states (last is same as first)

- (void)animationDidStop:(CAAnimation *)oldAnimation finished:(BOOL)flag
{
	[self changeState];
}

• TopToMiddle: show flip layer, and start rotation through 90 degrees
• MiddleToBottom: change image on flip layer, start rotation through further 90 degrees
• ResetForNext: change displayed top and bottom layer contents and hide flip layer

Let’s see how it looks in action:

#I @"C:\winnt\Microsoft.NET\Framework\v2.0.50727\"
#r @"Microsoft.Build.Tasks.dll"
#r @"Microsoft.Build.Utilities.dll"
#r @"Microsoft.Build.Framework.dll"
#r @"Microsoft.Build.Engine.dll"
 
open System
open Microsoft.Build.Framework
open Microsoft.Build.Utilities
open Microsoft.Build.Tasks
 
do
    let t = 
        ResolveAssemblyReference(Silent = true,
            Assemblies = [| upcast TaskItem( ItemSpec="class1" ) |],
            AllowedAssemblyExtensions = [|".dll"|],
            AppConfigFile = "c:\\dev\\myapp\\myapp.exe.config",
            SearchPaths = [|"{GAC}"|]
            )
    match t.Execute() with
    | true  -> printf "%A\n" t.ResolvedFiles
    | false -> printf "Task failed"

The output is:

[|C:\WINNT\assembly\GAC_MSIL\class1\1.1.8.0__120bce90f68b861a\class1.dll|]

The code makes good use of F#’s named constructor arguments, which map to property setters on the object we’ve just created. We can also create the arrays of TaskItems and strings fairly easily, although an explicit upcast is required in order to obtain the ITaskItem (F# doesn’t upcast implicitly like many object oriented languages).

I’ve just discovered that Jomo Fisher wrote about exactly the same thing back in May 2008! He mentions using an object expression in order to provide an implementation of IBuildEngine, which I’m getting away with by setting the Silent property on the ResolveAssemblyReference task, meaning that it never attempts to obtain the logging context from the build engine.

Another gotcha here is the fact that versions of various interfaces including IBuildEngine and ITaskItem exist in both the 2.0.0.0 and 3.5.0.0 versions of the Microsoft.Build.Framework assembly. Be careful that you reference the correct version for the task that you’re calling. Normally it will be the 2.0 versions, in which case you’ll need to use explicit file references in your script, adding the directory to the library include path:

#I @"C:\winnt\Microsoft.NET\Framework\v2.0.50727\"
#r @"Microsoft.Build.Framework.dll"

Rather than partially qualified assembly references:

#r @"Microsoft.Build.Framework"

As the latter will resolve to the latest installed version of the assembly, 3.5, which is not what you want. Look out for errors like:

System.InvalidCastException: Unable to cast object of type ‘Microsoft.Build.Utilities.TaskItem’ to type ‘Microsoft.Build.Framework.ITaskItem’.

Alternatives

The Tools|Options MSBuild verbosity setting

Collecting stack traces

One of the most helpful things the heap manager can do for you when investigating memory issues is to capture stack traces for each heap allocation. You can enable the “collect stack traces” heap flag using the gflags GUI or from within WinDbg:

0:006> !gflag +ust
Current NtGlobalFlag contents: 0x00001040
    hpc - Enable heap parameter checking
    ust - Create user mode stack trace database

This means that for each heap block (one located at 0x0bbf7308 in this case), you can see where it was allocated by using the -a (show all information) option:

0:006> !heap -p -a 0bbf7308
    address 0bbf7308 found in
    _HEAP @ a630000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        0bbf7308 0073 0000  [07]   0bbf7310    00380 - (busy)
        Trace: 401c
        7c96d6dc ntdll!RtlDebugAllocateHeap+0x000000e1
        7c949d18 ntdll!RtlAllocateHeapSlowly+0x00000044
        7c91b298 ntdll!RtlAllocateHeap+0x00000e64
        78134333 MSVCR80!malloc+0x00000077

But the obvious problem with this is that the stack trace always stops at malloc. Something’s allocating some memory? You don’t say…

It turns out that this is a well understood and documented issue with the Microsoft VC++ runtime, variously known as msvcrt, msvcr70, msvcr71, msvcr80, msvcr90, etc. Unfortunately they’re all built using the stack frame pointer omission optimisation. Well they’re built with the -O1 (favour small code) option, which implies -Oy. This means that the fast stack-walking algorithm the heap manager uses stops at functions without a return address. The only way to get a decent trace in this situation would be to use the DbgHelp API along with the .pdb files, which would be far too slow to do at each allocation site.

“Fixing” it

So, given that the source for the runtime library ships as part of Visual Studio, maybe it would be possible to build it without the -Oy option?

My first attempt at building it failed miserably with errors like:

NMAKE : fatal error U1073: don't know how to make 'build\intel\mt_obj\startup.lib'

Luckily this excellent page helped me get past this to a point where I could actually get a DLL built.

The next stage was to modify the build scripts to use different compiler switches. This was simply a case of changing line 69 of makefile.sub from:
CFLAGS=$(CFLAGS) -O1
to:
CFLAGS=$(CFLAGS) -O1 -Oy-

I thought I may have to also modify the build scripts to output a version of the DLL with the same name as the file I was replacing, msvcr80.dll, directly, in case there were internal references to the name in embedded manifests. There’s a section at the top of the build script for choosing a name for your private version of the library, but it strongly discourages use of the “reserved” MSVC* names. Luckily it turns out not to be necessary; the DLL is constructed in such a way as to be rename-able without any ill effects. I could build _sample_.dll (the default output name) and simply copy it to the destination directory in the SxS tree (C:\WINNT\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.3053_x-ww_b80fa8ca) and rename it.

Result

Now I get the expected full stack trace (names have been changed to protect the innocent):

0:006> !heap -p -a 0bbf7308
    address 0bbf7308 found in
    _HEAP @ a630000
      HEAP_ENTRY Size Prev Flags    UserPtr UserSize - state
        0bbf7308 0073 0000  [07]   0bbf7310    00380 - (busy)
        Trace: 401c
        7c96d6dc ntdll!RtlDebugAllocateHeap+0x000000e1
        7c949d18 ntdll!RtlAllocateHeapSlowly+0x00000044
        7c91b298 ntdll!RtlAllocateHeap+0x00000e64
        78134333 MSVCR80!malloc+0x00000077
        7816207f MSVCR80!operator new+0x0000001d
        fa92336 leakydll!std::allocator,std::allocator > > >::allocate+0x00000016
        fa9879b leakydll!std::vector >::resize+0x0000005b
        ...
        ...etc...

That’ll make it much easier to work out what’s happening and who’s responsible. Of course, you should be careful with this modified version. Only use it on development machines, and make sure it doesn’t escape into the wild: with great power comes great responsibility.

Form factor

The most obvious reason that this kind of interface works on the iPad is the form factor of the device itself. Think about it: an iPhone is a small (well, smallish) object which has no physical, real-world equivalent. It’s not supposed to “be” anything other than a phone. The iPad, on the other hand, is intended to mimic an actual object, or rather, a variety of objects: a book, a newspaper or an address book for example. The interface for each of these uses reflects this directly: the address book looks like a “little black book”, complete with tabs.

The iPhone uses some of the same metaphors – like the address book for contacts – but the representation is abstract, whereas the iPad UI is literal, using an actual image of a book. The abstraction is necessary on the iPhone in order for it to remain usable with a smaller screen. Compare the iPad and iPhone Contacts interfaces:

The realistic looking address book application on the iPad

iPhone contacts interface

The abstract to realistic continuum has been explored extensively in the world of icons (explained brilliantly here), but less so in the interface as a whole. Obviously some apps don’t have any form of real world equivalent (think System Preferences.app), so they’re likely to stay in a more abstract form.

Display fidelity

Of course, it’s always been possible to represent real objects on the screen, but without a display capable of a decent resolution it probably won’t look real. So another enabler for naturalistic UI is display technology: the more pixels and colour depth the better when it comes to rendering physical things. Realism often relies on subtle lighting and rich textures to achieve the desired effect. Anyone who’s ever attempted to render wood grain using 8-bit colour will appreciate how much easier it is with a 32-bit 1024×768 display.

The need for display quality goes hand-in-hand with a need for more processing power to push the pixels. If the processing “oomph” required to render your interface with sufficient quality means that it’s not responsive enough, then you’ve failed by any measure. The iPad seems to have reached a sweet spot where display capabilities, raw power and form-factor meet to make naturalistic UIs a feasible approach.

Direct manipulation

Up until the relatively recent introduction of direct manipulation technologies like touch-senstive displays, there’s a been a forced physical separation between users and their computer in the form of a mouse and keyboard. A drag operation using a mouse involves disconnected, abstract operations like pointing and clicking, which bear no relation to the object being acted upon. As such, there’s little use in having an extremely realistic representation of something to interact with, because the suspension-of-disbelief has already been broken.

But in the world of touch-based devices, our interactions are much closer to the physical actions we’d make, so having a closer visual representation to go along with it is beneficial. For instance, turning a page in an eBook and actually seeing the page flick over seems consistent when using a natural gesture like flicking a finger across the screen. Otherwise if you had to click a corner of the page and drag it across in a poor imitation of the physical action the visual aspects can quickly start to feel superfluous.

Perhaps given the iPad’s larger interaction surface we can expect to see a wider and more expressive range of gestures supported?

History

Of course, naturalistic UI isn’t really a new thing. It’s in use in quite a few applications already, albeit mostly in desktop apps, where capable hardware is more readily available. In fact, there was much a-twittering (not all of it from Will Shipley) about the similarity between the demo’d iPad apps and the interface of the fantastic Mac application Delicious Library. I’m sure of course – how do they say it in the movie credits – the similarity is entirely coincidental…

Delicious Library UI

iBooks UI

One of the first widespread uses of naturalistic UI elements was in software synths. Guys like Propellerheads and Native Instruments set about creating exact software duplicates of well-known and well-loved bits of mid-80’s audio hardware. The UIs included virtual versions of a huge number of physical elements that had never been used before: including things like sliders, LEDs, toggle buttons, rotary knobs and rocker switches. These interface elements were probably first seen in software-based mixing desks, but the techniques used to make them feel real have gone on to be used in many different ways. Even the lowly push button in both Windows and MacOS got an upgrade by application of some of the 3D effects that they originated.

Where Now?

It will be interesting to see what other types of app are given the iPad realism makeover. It certainly looks like all of the standard Apple apps are being changed, including Notes, iCal, iBooks and Contacts, which were shown in the launch presentation, but how will 3rd-party developers manage? It undoubtedly takes extra effort to make something look convincing when it can be compared to the physical object itself.

Apple has previously managed to get a good level of consistency in its abstract UIs by providing the various interface elements ready-made, but will that approach scale to this kind of UI? No doubt the developers who take great pride in the interfaces now will continue to strive to create apps that sit comfortably alongside Apple’s own, but perhaps there’ll just be a more obvious visual distinction between those guys and the less polished application user interfaces than there has been in the past.

So what’s been keeping me away from WordPress?

• Spending too much time in WinDbg tracking down .NET and COM memory issues.
• Wanting to have a good look at Visual Studio 2010 and .NET 4.0 (though not as much as I’d like, yet)
• Playing lots of Bayonetta, and some Heavy Rain.
• Doing lots of internal demos of the WPF\F# GUI-building toolset we’ve been working on. Should be interesting, specially now Luca’s coming on board.
• Experimenting with iPad development

Hopefully I’ll get the chance to write something about all of this soon.

You can do it without the mouse, too. Just make sure the window is selected in the task bar (perhaps by using the Windows key and then tabbing to the appropraite icon), then hit Alt-Space, Alt-M, and then using the cursor keys.

The only problem is, some application developers choose to change the system menu – the menu visible from clicking the app’s icon in the top left, or by right-clicking the task bar icon – perhaps deciding for some insane reason not to include the standard Move option (as an aside, this breaks one of the fundamental tenets of usability: don’t change – or in this case, remove – existing, established behaviour). If this is the case, you can instead use a handful of lines of Windows API code to access the window and move it programatically. Here’s the entire code of a program that will do exactly that:

#include <windows.h>
#include <tchar.h>
 
int _tmain(int argc, TCHAR *argv[])
{
	if (argc < 2)
	{
		_tprintf(_T("usage: mover windowtitle\n"));
		return -1;
	}
	HWND hwnd = FindWindow(NULL, argv[1]);
	if (hwnd)
	{
		SetWindowPos(hwnd, HWND_TOP, 0, 0, 400,400, 0);
	}
	else
		_tprintf(_T("Unable to find window %s\n"), argv[1]);
	return 0;
}

Compile it up and you can use it as a handy little utility to move arbitrary top level windows, e.g. if you’ve got regedit running:

mover "Registry Editor"

The only tricky thing is finding the exact title of the window to use. Well, you wouldn’t want it to be too easy, would you?