Cornell Box in the evening

What is a “global illumination renderer”?

In short; a global illumination renderer attempts to simulate the way that light behaves in the real world. It bounces rays of light around the scene, noting which surfaces are hit, and how each of them affects the ray. Of course real light travels fast – at about the speed of light, in fact – and our simulated rays can’t match that, so although the resulting effect looks natural, it’s not a very efficient way of getting there. Graphics systems that need to be near real-time (i.e. games), use a variety of short-cuts to try and get the same effect without requiring the same amount of number crunching.

Even this implementation is forced to use a couple of techniques to improve performance, I won’t go into them here, as they’re explained pretty thoroughly on the Minilight site.

Interestingly the algorithm uses a Monte Carlo technique to simulate the random path of the ray through the scene and the same technique is used in finance to model how the price of an asset will change through time.

Translating to F#

Room with sun and light, 800 paths per pixel

Some of the ML constructs caused warnings from the F# compiler as I didn’t use the explicit ML compatibility mode. For instance, the use of the ^ operator to concatenate strings:

warning FS0062: This construct is for ML compatibility. Consider using the '+' operator instead. This may require a type annotation to indicate it acts on strings.

Some other changes included:

• Using dot rather than # notation to access class members
• Using square bracket for array element access (array.[n] rather than array.(n))
• Removing let...and for non-recursive bindings

There was also the opportunity to make use of some of the .NET framework classes, things like System.Drawing.Bitmap to write pixels directly into a bitmap and then save it into a .PNG file. Much easier than trying to find something to read the .PPM file that the original version emits.

Potential improvements

One of the most obvious improvements that could be made is to make use of the parallelism support in F# and .NET. The algorithm itself is easily (if not embarrassingly) parallelisable, given that it iterates over the pixels in the output image, and performs independent operations on each of them. Unfortunately I haven’t had a chance to look at this yet – and the Windows VM I’m using only has a single processor anyway, so on a purely selfish note, I wouldn’t see any immediate performance improvements. It should be as simple as changing the pixel loop to use Parallel.For, or creating an async computation expression.

Something else to note: I err’d on the side of caution when it comes to saving intermediate images, so every iteration it creates and writes the file. This will be doing lots of allocation and strictly unnecessary work, especially for simple scenes.

Given this approach is so completely compute bound it would be interesting to see how moving some of the calculations to a GPU would affect it. It might even be possible to use quotations for some of the inner loops and then generate Nvidia intermediate language (PTX) from them – that’s something I’ve been wanting to do for a while.

I did run the code through the Visual Studio 2010 profiler to get a rough idea of where the time was being spent; it looked like the majority was in the leaf of the calculations where numerical operations where being performed on the vectors, which is consistent with what we’d expect.

Use the source

So, here’s the source. No warranties implied, use at your own risk, your mileage may vary, all errors are mine etc, etc.
minilight_fsharp

_toplayer = [CALayer layer];

Splitting images

Now we’ve got the layers, we need something to but into them. I didn’t want to create lots of pre-processed images for the top and bottom sections of each number, so instead I wrote a routine to split the images in code, drawing each half into an image context and grabbing a UIImage from it. This is then stored in an array for later use. We can get any of the elements from the array and set it as the content of the layer as required.

// The size of each part is half the height of the whole image
CGSize size = CGSizeMake([img size].width, [img size].height/2);
 
// Create image-based graphics context for top half
UIGraphicsBeginImageContext(size);
 
// Draw into context, bottom half is cropped off
[img drawAtPoint:CGPointMake(0.0,0.0)];
// Grab the current contents of the context as a UIImage 
// and add it to our array
UIImage *top = [UIGraphicsGetImageFromCurrentImageContext() retain];			
[_imagesTop addObject:(id)top];
 
// Now draw the image starting half way down, to get the bottom half
[img drawAtPoint:CGPointMake(0.0,-[img size].height/2)];
// And store that image in the array too
UIImage *bottom = [UIGraphicsGetImageFromCurrentImageContext() retain];			
[_imagesBottom addObject:(id)bottom];
 
UIGraphicsEndImageContext();

Now our _imagesTop and _imagesBottom arrays contains all the images we need, let’s get ‘em moving.

Applying animation

The axes and anchor point of a CALayer

_fliplayer.anchorPoint = CGPointMake(0.5, 1.0);

I use the CATransform3DMakeRotation function to create a transform around the X axis using the vector [1,0,0]. For the first part of the motion, we go from 0 radians to pi/2 radians (0 to 90 degrees), using an explicit CABasicAnimation object:

CABasicAnimation *topAnim = [CABasicAnimation animationWithKeyPath:@"transform"];
topAnim.duration=0.25;
topAnim.repeatCount=1;
topAnim.fromValue= [NSValue valueWithCATransform3D:CATransform3DMakeRotation(0.0, 1, 0, 0)];
float f = -M_PI/2;
topAnim.toValue=[NSValue valueWithCATransform3D:CATransform3DMakeRotation(f, 1, 0, 0)];
topAnim.delegate = self;
topAnim.removedOnCompletion = FALSE;
[_fliplayer addAnimation:topAnim forKey:[NSString stringWithUTF8String:k_TopAnimKey]];

Drawing layer content

Unfortunately a CALayer doesn’t have a “back”, so creating the flip layer is not as easy as it could be. We need to add a step in the process to change the image displayed by the flip layer half way through its fall. We can do this in a variety of ways, I’ve chosen to set a delegate object to perform the drawing of the layer contents:

[_fliplayer setDelegate:_layerHelper];

This means our delegate object (_layerHelper) has its drawLayer method called whenever the layer needs to display its contents. Be wary though, this may not be as often as you think! Core graphics aggressively caches the contents of the layer, and will only call your delegate when absolutely necessary. You can force it to happen by calling setNeedsDisplay, which invalidates the contents of the layer:

[_fliplayer setNeedsDisplay];

I encapsulated this in the helper object itself, by providing a custom implementation of the isTop property setter that calls setNeedsDisplay. At least that way callers don’t have to be aware of this requirement.

The drawLayer method itself simply draws one of two images into the context, depending on whether it’s in ‘top’ mode. A slight further complication is that when drawing the bottom part, the image needs to be inverted to display upside down. This is achieved by applying a translation and scale to the context before drawing the image:

CGContextSaveGState(context);
CGContextTranslateCTM(context, 0.0f, r.size.height);
CGContextScaleCTM(context, 1.0f, -1.0f);
 
CGContextDrawImage(context, r, [_imgTop CGImage]);
 
CGContextRestoreGState(context);

Avoiding implicit animation

I’ve tried to avoid using implicit animation here, instead using explicit construction of CABasicAnimation objects, because we need some control of when things happen, and the order in which they happen. Normally, it would be easier to just set some property of the view and let the OS manage animating it. Many properties have default transition that are applied whenever they’re changed, so it takes some additional effort to make them change immediately.

In our example we change the contents of the top half and show the flip layer, but we need that to happen immediately so we disable actions within an explicit transaction that we then commit. This gives us the control we need, and avoids the default behaviour of the transition occurring when the message loop next runs.

[CATransaction begin];
[CATransaction setValue:(id)kCFBooleanTrue
				forKey:kCATransactionDisableActions];
_toplayer.contents = (id)[[_imagesTop objectAtIndex:[self getNextIndex]] CGImage];
_fliplayer.hidden = NO;
[CATransaction commit];

Getting some perspective

Although the images I’ve used to illustrate this post are isometric, what we really want is a front-on perspective view, so that the falling piece seems to stick out. For this we have to set a sublayerTransform on the view’s inherent layer, by setting an individual element on a transform:

CATransform3D aTransform = CATransform3DIdentity;
float zDistance = 1000;
aTransform.m34 = 1.0 / -zDistance;	
[self layer].sublayerTransform = aTransform;

This gives us a much more realistic look.

Putting it together

Wow, so now we’ve got some appropriately sliced images, some layers and some animations. Let’s put them together to get the final effect.

The animation states (last is same as first)

- (void)animationDidStop:(CAAnimation *)oldAnimation finished:(BOOL)flag
{
	[self changeState];
}

• TopToMiddle: show flip layer, and start rotation through 90 degrees
• MiddleToBottom: change image on flip layer, start rotation through further 90 degrees
• ResetForNext: change displayed top and bottom layer contents and hide flip layer

Let’s see how it looks in action:

Form factor

The most obvious reason that this kind of interface works on the iPad is the form factor of the device itself. Think about it: an iPhone is a small (well, smallish) object which has no physical, real-world equivalent. It’s not supposed to “be” anything other than a phone. The iPad, on the other hand, is intended to mimic an actual object, or rather, a variety of objects: a book, a newspaper or an address book for example. The interface for each of these uses reflects this directly: the address book looks like a “little black book”, complete with tabs.

The iPhone uses some of the same metaphors – like the address book for contacts – but the representation is abstract, whereas the iPad UI is literal, using an actual image of a book. The abstraction is necessary on the iPhone in order for it to remain usable with a smaller screen. Compare the iPad and iPhone Contacts interfaces:

The realistic looking address book application on the iPad

iPhone contacts interface

The abstract to realistic continuum has been explored extensively in the world of icons (explained brilliantly here), but less so in the interface as a whole. Obviously some apps don’t have any form of real world equivalent (think System Preferences.app), so they’re likely to stay in a more abstract form.

Display fidelity

Of course, it’s always been possible to represent real objects on the screen, but without a display capable of a decent resolution it probably won’t look real. So another enabler for naturalistic UI is display technology: the more pixels and colour depth the better when it comes to rendering physical things. Realism often relies on subtle lighting and rich textures to achieve the desired effect. Anyone who’s ever attempted to render wood grain using 8-bit colour will appreciate how much easier it is with a 32-bit 1024×768 display.

The need for display quality goes hand-in-hand with a need for more processing power to push the pixels. If the processing “oomph” required to render your interface with sufficient quality means that it’s not responsive enough, then you’ve failed by any measure. The iPad seems to have reached a sweet spot where display capabilities, raw power and form-factor meet to make naturalistic UIs a feasible approach.

Direct manipulation

Up until the relatively recent introduction of direct manipulation technologies like touch-senstive displays, there’s a been a forced physical separation between users and their computer in the form of a mouse and keyboard. A drag operation using a mouse involves disconnected, abstract operations like pointing and clicking, which bear no relation to the object being acted upon. As such, there’s little use in having an extremely realistic representation of something to interact with, because the suspension-of-disbelief has already been broken.

But in the world of touch-based devices, our interactions are much closer to the physical actions we’d make, so having a closer visual representation to go along with it is beneficial. For instance, turning a page in an eBook and actually seeing the page flick over seems consistent when using a natural gesture like flicking a finger across the screen. Otherwise if you had to click a corner of the page and drag it across in a poor imitation of the physical action the visual aspects can quickly start to feel superfluous.

Perhaps given the iPad’s larger interaction surface we can expect to see a wider and more expressive range of gestures supported?

History

Of course, naturalistic UI isn’t really a new thing. It’s in use in quite a few applications already, albeit mostly in desktop apps, where capable hardware is more readily available. In fact, there was much a-twittering (not all of it from Will Shipley) about the similarity between the demo’d iPad apps and the interface of the fantastic Mac application Delicious Library. I’m sure of course – how do they say it in the movie credits – the similarity is entirely coincidental…

Delicious Library UI

iBooks UI

One of the first widespread uses of naturalistic UI elements was in software synths. Guys like Propellerheads and Native Instruments set about creating exact software duplicates of well-known and well-loved bits of mid-80’s audio hardware. The UIs included virtual versions of a huge number of physical elements that had never been used before: including things like sliders, LEDs, toggle buttons, rotary knobs and rocker switches. These interface elements were probably first seen in software-based mixing desks, but the techniques used to make them feel real have gone on to be used in many different ways. Even the lowly push button in both Windows and MacOS got an upgrade by application of some of the 3D effects that they originated.

Where Now?

It will be interesting to see what other types of app are given the iPad realism makeover. It certainly looks like all of the standard Apple apps are being changed, including Notes, iCal, iBooks and Contacts, which were shown in the launch presentation, but how will 3rd-party developers manage? It undoubtedly takes extra effort to make something look convincing when it can be compared to the physical object itself.

Apple has previously managed to get a good level of consistency in its abstract UIs by providing the various interface elements ready-made, but will that approach scale to this kind of UI? No doubt the developers who take great pride in the interfaces now will continue to strive to create apps that sit comfortably alongside Apple’s own, but perhaps there’ll just be a more obvious visual distinction between those guys and the less polished application user interfaces than there has been in the past.

The wow+flutter programme concentrates on “progressive motion graphics”, which can include pretty much anything, while other parts of the festival are more specific: wavelength for music videos, j-star for japanese shorts etc. You can check out the entire list of w+f films are here, but these are my favourites:

Ubik: voxel
I’m a fan of the augmented reality style when it’s done well; and it certainly is here. Striking abstract graphics combine with a haunting location.

Impactist: Parallelostory
A beautifully subtle animation style with a 50’s influenced colour and texture palette.

Bruno Dicolla: Return as an Animal
This featured a striking high-contrast rotoscoped style that looked great on the big screen. Unfortunately the shimmering starfield effect is a little lost in the on-line version. I could almost hear art directors reaching for their iPhones to jot down “must copy this at next available opportunity”.

Christian Schlaeffer: Agent Orange Ready
A dark, disturbing animation in a comic style.

The Lows

Ironically I found the most obviously “digital” films the most disappointing. There was a little too much in the Chris Cunningham inspired glitchy, razor-cut, super-shiny organic/mechanic vernacular. And there was also one film – which I won’t name – that looked shockingly like a Tia Maria commercial. Still, in general the signal to noise ratio was very good.

The Laughs

Some of the most memorable films are those with a punchline; the short film format is a great way of getting a laugh with a simple, funny idea. There’s no risk of stretching it too thin.

Mato Atom: Docking
A laugh-out-loud concept executed nicely and succintly, and it even goes out with an anti-war punchline.

So there you are; just 5 of the 30+ shorts that were shown. If you’re into animation or cutting-edge digital film-making, I’d recommend catching a onedotzero screening wherever you can, either in London or on one of the tour locations.