“The 25th Consumer Electronics Show… showed very clearly that one thing continues to dominate the media and electronics business in the USA. This is not the personal computer – which seems to be causing far more interest in Europe than in America – but the video game.”

So while in the UK we were busy typing in listings from games magazines, in the US Atari, Mattel were creating ready-to-play cartridge based games.

Getting compelling press coverage was made difficult by the fact that it was impossible to get decent screen shots. Think how much of the pages in modern games mags are covered with glossy, full-colour, high-resolution screen shots. Without these, you’re left with nothing but words to describe these visual wonders. Luckily, you didn’t need much of a vocabulary to describe the games of the time; they were either based on existing analogue games, or they were derived from the small pool of original video game ideas:

“Munchie, which is similar to PacMan” … “Cosmic Raiders is a fast-action Defender style games” … “Solar Conqueror is an Asteroids game.”

Of course, some of this “re-purposing” was pukka; Coleco made table-top versions of things like PacMan, Donkey Kong, and Frogger (image above). You can imagine how faithful these arcade conversions actually were…

We’ve made quite some progress in the last 30 years, both in terms of game mechanics and graphics. It’s best summed up by 2 images, I’m sure you can guess which is from 1982, and which from 2012?

Click through these images for the full scans on flickr.

OpenGL

The 3D work I’d done previously on iOS was using OpenGL ES 1.1, so as part of doing this, I also needed to get up to speed with the changes in version 2.0. The move involves replacing the fixed function pipeline approach with the use of shaders crafted in “shader language”, specifically GLSL. This approach is more flexible, but it means you take full responsibility for combining all the relevant matrices (model, projection etc), rather than letting OpenGL do it for you.

There are 2 distinct types of shader programs that operate on either vertices (vertex shaders) or pixels (fragment shaders). The steps required to create shader programs mirror the process of creating “normal” programs: they’re loaded, compiled and linked. Once you’ve got a program loaded, you can initialise it by setting variables of various kinds: attribute, uniform or varying. These are used for vertex and colour data, constants and passing data between vertex and fragment shaders respectively. If you’re interested in learning more, I suggest you check out some of the plethora of internet and dead-tree material on OpenGL. I’ll only be doing pretty basic stuff here.

Bullet

I decided to try using Bullet as the physics engine. It looks very powerful and is widely used in various games and movies; the only bad thing I can say about it is that decent documentation is a bit lacking. I found the best way to get help was googling for API names on the Bullet forum; not a very efficient way of doing things. Maybe I’ve just been spoilt by the copious Apple and MS docs (yeah, yeah).

Building it

Luckily Bullet comes with an Xcode project “in the box”, so with a bit of autoconf magic, it was soon up and building the demo apps. Although there are loads of files in there, the core library itself isn’t too large or complex. As such, I decided to create a separate static library project targeting the iOS simulator (i386) and devices (armv6/7).

API basics

My intention is to use only a small subset of the Bullet API, encompassing rigid bodies and a couple of simple constraint types. For the purposes of the demo, I’m going to create the pretty standard set-up of a large static ground-box, along with a collection of more dynamic objects.

The key part of joining up Bullet with your existing API is the btMotionState class. It enables the API to callback into your code with updated positions at each ‘tick’ of the physics engine, and you can then alter your geometry appropriately. I created a simple class that derives from btMotionState, implements the required pure virtual methods and contains a reference to my world object. Bullet calls the subclassed methods and we pass the transformations on to the object.

class QuadMotionState : public btMotionState {
    Quad *_object;
    btTransform _transform;
public:
    QuadMotionState(const btTransform &initialpos, Quad *node) {
        _object = node;
        _transform = initialpos;
    }
 
    virtual ~QuadMotionState() {
    }
 
    void setNode(Quad *node) {
        _object = node;
    }
 
    virtual void getWorldTransform(btTransform &worldTrans) const {
        worldTrans = _transform;
    }
 
    virtual void setWorldTransform(const btTransform &worldTrans) {
        // Pass the worldTrans to _object
    }
};

Debugging

Now, obviously you will write the perfect code first time, and will have no need for any kind of diagnostics. But I am not that perfect; I need to see the difference between what I’ve told the API to do, and what it’s actually doing. You can do this by creating an object implementing btIDebugDraw and passing it to the (slightly oddly named) setDebugDrawer function on your world:

btIDebugDraw *debug = new DebugDrawHelper();
m_dynamicsWorld->setDebugDrawer(debug);

The most important function on btIDebugDraw is drawLine. It will be called repeatedly to draw axes and bounding boxes in a variety of colours to indicate the object’s state. We can use the Shader we created earlier to give us an easy way of generating these debug visuals. All we need is a very simple shader, so can declare it inline:

program.loadShaders(
		"attribute vec4 position;\n"
		"uniform mat4 projection;\n"
		"void main () {\n"
		"gl_Position = position * projection;\n"
		"}\n", 
 
		"uniform highp vec3 color;\n"
		"void main () {\n"
		"gl_FragColor = vec4(color.r, color.g, color.b, 0.0);\n"
		"}");

Of course, anything more complex than this and you’ll want to put it into a separate file, but this program allows us to pass in the start and end vertex and its colour to draw the line in our implementation of DrawLine:

virtual void drawLine(const btVector3& from,const btVector3& to,const btVector3& color) {
        glUseProgram(program.getProgram());
 
        // Set the projection matrix
        GLint pu = glGetUniformLocation(program.getProgram(), "projection");
        glUniformMatrix4fv(pu, 1, 0, _projection.Pointer());
 
        // Set the colour of the line
        GLint puc = glGetUniformLocation(program.getProgram(), "color");
        vec3 v = vec3(color.getX(), color.getY(), color.getZ());
        glUniform3fv(puc, 1, v.Pointer());
 
        // Set the line vertices
        float tmp[ 6 ] = { from.getX(), from.getY(), from.getZ(),
            to.getX(), to.getY(), to.getZ() };
        glVertexAttribPointer(0, 3, GL_FLOAT, false, 0, &tmp[0]);
        glEnableVertexAttribArray(0);
        glDrawArrays( GL_LINES, 0, 2 );
}

Bullet debug output

Populating the world

So in order to have something nice to look at, we’re gonna need to put some ‘things’ into our world. And in order to make it act like the real world we’ll need to give those things some physical properties.

I created a simple class to render a textured box. This OpenGL object then needs to be represented in the physics worlds by a rigid body object, which in turn contains a collision detection object that closely matches the dimensions of the original object. In fact, in the case of a box, the collision detection object matches exactly, otherwise you’ll need to create an approximation using the more advanced classes. We use a btBoxShape to initialise our btRigidBody instance, along with setting its mass, inertia, and making a link to the state object that I mentioned above.

btRigidBody *addShape(Quad *shape, btScalar mass = 1.0f, float restitution = 0.2f)
{
	// Set the initial transform based on the current location of the object
        btTransform transform;
	transform.setIdentity();
	transform.setOrigin(btVector3(shape->getX(), shape->getY(), 0.));
 
	btCollisionShape* cshape = 
            new btBoxShape(btVector3(btScalar(shape->getWidth()/2),
                                     btScalar(shape->getHeight()/2),
                                     btScalar(shape->getDepth()/2)));
        btVector3 localInertia(0,0,0);
	if (mass != 0.0f)
            cshape->calculateLocalInertia(mass,localInertia);
 
        QuadMotionState *state = new QuadMotionState(transform, shape);
	btRigidBody::btRigidBodyConstructionInfo rbInfo(mass, state, cshape, localInertia);
	btRigidBody* body = new btRigidBody(rbInfo);
 
	body->setDamping(0.85, 0.85);
	body->setRestitution(restitution);
 
        // Add the rigid body to the Bullet physics world
	m_dynamicsWorld->addRigidBody(body);
 
        // Establish the link between our object and the physics object
	shape->setPhysicsObject(body);
	return body;
}

To make things look a bit more polished I added some textures to the quads, along with very simple normal mapping to get some lighting effects. I also added the ability to rotate the view and zoom in and out by using UIGestureRecognizers and changing the camera transform appropriately. The “finishing” touch (yeah, right), is the ability to tap to add a new quad.

The results

Here’s how the final thing looks. It’s very basic, I’ve hardly touched the surface of the Bullet API – there are no constraints in play, for instance – but it’s still kind of fun to play with. It’s running fairly slowly here, what with it being in the simulator, and being recorded. On a physical iPad it’s much snappier.
Let me know if you’d be interested to see some more coverage of some of Bullet’s constraints and hinges features.

Just what do you think you're doing, Dave?

The first bit was easy: I’m “lucky” enough to have one of the older Xboxes, which meant I’d had to get a Kinect with separate power, which is the one required by the SDK. Now all I needed was a Windows machine to develop on.

For all my Visual Studio stuff at home I use a virtual machine, and unfortunately I missed the point in the readme about “Kinect for Windows applications cannot run in a virtual machine”. Doh. That would explain why, try as I might I couldn’t get VirtualBox to detect the device when I plugged it into the host. Whatever I did I ended up with a ‘resource is busy’ error.

I even tried another VM, this time from VMWare. It got further, with the guest seeing the devices, but whenever I tried to run the sample apps the API initialisation call failed. Unfortunately the samples make a bit of a rookie mistake and don’t display the error code associated with the failure; the only way to get the underlying HRESULT is to debug the app. As it turned out the error was the catch-all 80080014, which is attributed to various USB issues.

So, I finally relented and decided to once again set-up Bootcamp, which I’d stopped using a while back: why bother when VMs had done everything I needed? Again I was snookered, this time by the incompatibility of the EFI firmware in my ageing MacBook with the Win7 x64 install disc (which I’d had to actually burn onto optical media – can’t remember the last time I burnt a real physical disk)!

Eventually, I got Win7 x86, Visual Studio and the Kinect SDK installed on the metal, plugged in the sensor and – whoa – the devices were recognised and the drivers installed and… the samples ran!

From this point doing some hacking was pretty straightforward. I set about creating a project that used the skeletal tracking ability from the SDK. The project needs to reference the Microsoft.Research.Kinect assembly, then we can open some namespaces and initialise the API:

open System
open Microsoft.Research.Kinect
 
let nui = Nui.Runtime()
nui.Initialize(Nui.RuntimeOptions.UseSkeletalTracking)

The API uses .NET events to communicate back to the application that some form of data is available. Depending on the options that you specify at init time, any of the skeletal, depth frame or colour data will be returned in the event arguments. This seems like a good place to use F# Async’s event integration: Async.AwaitEvent. We can quite easily write some code that will create an Async task that will repeatedly listen for events:

    let skeleton (nui : Nui.Runtime) =
        let rec loop () =
            async { 
                let! args = Async.AwaitEvent nui.SkeletonFrameReady
                // Do something with the args
                return! loop () 
            }
        loop ()
    // Start the process of listening for events...
    skeleton nui |> Async.Start

This seems to work nicely, although I’m not sure of the overhead involved, perhaps a solution involving Rx would be more appropriate. Anyway, let’s do something with the data we get in the event, like getting all the skeletons that are being tracked, and passing them to a draw function:

                args.SkeletonFrame.Skeletons 
                |> Seq.filter (fun s -> s.TrackingState = Nui.SkeletonTrackingState.Tracked)
                |> Seq.iter draw

All that’s left is doing something cool in draw! Oh, and setting up all the GUI stuff necessary to actually get some pixels on the screen. I went for the WPF approach (as the managed samples do), which involves creating a simple object tree to display a rectangle in a canvas in a grid in a window, and doing a bit of marshalling back from the thread pool (where our Async code runs) to the main GUI thread.

Here’s all the code for possibly the most tedious thing you could do with a Kinect! But hey, it’s a starting point, right? I’m sure at some point I’ll get around to doing something cooler than a couple of small red rectangles with it…

open System
open System.Windows
open Microsoft.Research.Kinect
 
[<STAThread>]
do
    let nui = Nui.Runtime()
    nui.Initialize(Nui.RuntimeOptions.UseSkeletalTracking)
 
    // lifted straight from the sample code
    let getDisplayPosition w h (joint : Nui.Joint) =
            let mutable depthX = 0.0f
            let mutable depthY = 0.0f
            nui.SkeletonEngine.SkeletonToDepthImage(joint.Position, &depthX, &depthY)
            let depthX = depthX * 320.0f; //convert to 320, 240 space
            let depthY = depthY * 240.0f; //convert to 320, 240 space
            let mutable colorX = 0
            let mutable colorY = 0
            let iv = new Nui.ImageViewArea()
            // only ImageResolution.Resolution640x480 is supported at this point
            nui.NuiCamera.GetColorPixelCoordinatesFromDepthPixel(Nui.ImageResolution.Resolution640x480, iv, 
                (int)depthX, (int)depthY, 0s, &colorX, &colorY)
 
            // map back to visible area
            new Point((w * (float)colorX / 640.), (h * (float)colorY / 480.))
 
    // Set-up the WPF window and its contents
    let width = 1024.
    let height = 768.
    let w = Window(Width=width, Height=height)
    let g = Controls.Grid()
    let c = Controls.Canvas()
    let hd = Shapes.Rectangle(Fill=Media.Brushes.Red, Width=10., Height=10.)
    ignore <| c.Children.Add hd
    ignore <| g.Children.Add c
    w.Content <- g
 
    // We simple move the rectangle to where the head is
    let draw (s : Nui.SkeletonData) =
        let point = getDisplayPosition width height s.Joints.[Nui.JointID.Head]
        ignore <| w.Dispatcher.BeginInvoke(Threading.DispatcherPriority.Normal, Action(fun () ->    
            hd.SetValue(Controls.Canvas.TopProperty, point.Y)
            hd.SetValue(Controls.Canvas.LeftProperty, point.X)))
 
    let skeleton (nui : Nui.Runtime) =
        let rec loop () =
            async { 
                let! args = Async.AwaitEvent nui.SkeletonFrameReady
                args.SkeletonFrame.Skeletons 
                |> Seq.filter (fun s -> s.TrackingState = Nui.SkeletonTrackingState.Tracked)
                |> Seq.iter draw
                return! loop () 
            }
        loop ()
 
    skeleton nui |> Async.Start
 
    let a = System.Windows.Application()
    ignore <| a.Run(w)
 
    nui.Uninitialize()

The moral of the story? Don’t be like me, and make sure you actually read the FAQing readme, then maybe you’ll spend more time doing a decent demo and less time plugging and unplugging your Kinect!

But this year, much to my surprise and delight, they introduced a special, Sunday-afternoon screening especially for kids, called “Sprites”; excellent! Now I had an excuse to bring the whole family along too! Was it a co-incidence that Shane Walter, curator of the festival, and his lady wife have had a baby this year…? Maybe…

To be honest, I was a bit worried; my daughters are not the most, err, robust cinema-goers. The youngest spent a good deal of the latter part of Toy Story 3 screaming in terror. Still, I was determined (and intrigued) to see what had been programmed, and thought that if it came to it I’d just have to shuffle embarrassingly out of the cinema with a tearful child.

It was brilliant. I can safely say that all 4 of us (me, my wife and a 3 and 5 year old) really enjoyed the show. It was programmed really well, with a mix of 10+ minute long animated shorts, and shorter, sharper sequences. It kept everyone’s attention very effectively. The longest film was one originally intended as a pilot for a series, an unlikely sounding Argentinian creation called Gluko and Lennon, a quirky, contemporary combination of plucky pink and purple protagonists and a depressed superhero. The amorphous pink Lennon (or was it Gluko) was the youngest one’s favourite character of the show.

Like some of the other films, it had quite an edgy feel, but I think kids nowadays are quite used to it; they’ve been bought up on the Yo Gabba Gabba slightly knowing, ironic style.

There was some very high-quality (I hesitate to say it, but, Pixar-quality) animation on display; and by that I mean both beautifully animated and also very well observed and amusing, even for adults. Lucas Martell’s Pigeon: Impossible was laugh-out-loud funny, and Hi-Sim’s Jun and the Hidden Skies was charming and polished.

There was also more traditional animation on display, which can often feel more disturbing somehow, but which were very unthreatening in this setting. The South Korean Cherry on the Cake was in this style, and featured an interesting angle on family life which was especially relevant to “little people”, though maybe a bit bigger than 2cm.

The psychedelic style was also well represented, from the lullaby style of Chico Jofilsan’s My Little Angel to LookListenFeel, which invoked feelings of a long, lazy summers afternoon. Andy Martin was on hand to introduce his weird and wonderfully-scored Dry Fish in person.

The last short was a minute’s worth of scat kitten. Believe it or not this my 5 year old’s favourite. Ah well, there’s no accounting for taste…

So, it was a huge success. I got to see some fantastic animation on the big screen, the kids got to see things they found funny and fascinating, and we all thoroughly enjoyed ourselves. If you’ve got any interest in animation or film, and a child or two to accompany you, head down down here next year and enjoy it too.

Cornell Box in the evening

What is a “global illumination renderer”?

In short; a global illumination renderer attempts to simulate the way that light behaves in the real world. It bounces rays of light around the scene, noting which surfaces are hit, and how each of them affects the ray. Of course real light travels fast – at about the speed of light, in fact – and our simulated rays can’t match that, so although the resulting effect looks natural, it’s not a very efficient way of getting there. Graphics systems that need to be near real-time (i.e. games), use a variety of short-cuts to try and get the same effect without requiring the same amount of number crunching.

Even this implementation is forced to use a couple of techniques to improve performance, I won’t go into them here, as they’re explained pretty thoroughly on the Minilight site.

Interestingly the algorithm uses a Monte Carlo technique to simulate the random path of the ray through the scene and the same technique is used in finance to model how the price of an asset will change through time.

Translating to F#

Room with sun and light, 800 paths per pixel

Some of the ML constructs caused warnings from the F# compiler as I didn’t use the explicit ML compatibility mode. For instance, the use of the ^ operator to concatenate strings:

warning FS0062: This construct is for ML compatibility. Consider using the '+' operator instead. This may require a type annotation to indicate it acts on strings.

Some other changes included:

• Using dot rather than # notation to access class members
• Using square bracket for array element access (array.[n] rather than array.(n))
• Removing let...and for non-recursive bindings

There was also the opportunity to make use of some of the .NET framework classes, things like System.Drawing.Bitmap to write pixels directly into a bitmap and then save it into a .PNG file. Much easier than trying to find something to read the .PPM file that the original version emits.

Potential improvements

One of the most obvious improvements that could be made is to make use of the parallelism support in F# and .NET. The algorithm itself is easily (if not embarrassingly) parallelisable, given that it iterates over the pixels in the output image, and performs independent operations on each of them. Unfortunately I haven’t had a chance to look at this yet – and the Windows VM I’m using only has a single processor anyway, so on a purely selfish note, I wouldn’t see any immediate performance improvements. It should be as simple as changing the pixel loop to use Parallel.For, or creating an async computation expression.

Something else to note: I err’d on the side of caution when it comes to saving intermediate images, so every iteration it creates and writes the file. This will be doing lots of allocation and strictly unnecessary work, especially for simple scenes.

Given this approach is so completely compute bound it would be interesting to see how moving some of the calculations to a GPU would affect it. It might even be possible to use quotations for some of the inner loops and then generate Nvidia intermediate language (PTX) from them – that’s something I’ve been wanting to do for a while.

I did run the code through the Visual Studio 2010 profiler to get a rough idea of where the time was being spent; it looked like the majority was in the leaf of the calculations where numerical operations where being performed on the vectors, which is consistent with what we’d expect.

Use the source

So, here’s the source. No warranties implied, use at your own risk, your mileage may vary, all errors are mine etc, etc.
minilight_fsharp

_toplayer = [CALayer layer];

Splitting images

Now we’ve got the layers, we need something to but into them. I didn’t want to create lots of pre-processed images for the top and bottom sections of each number, so instead I wrote a routine to split the images in code, drawing each half into an image context and grabbing a UIImage from it. This is then stored in an array for later use. We can get any of the elements from the array and set it as the content of the layer as required.

// The size of each part is half the height of the whole image
CGSize size = CGSizeMake([img size].width, [img size].height/2);
 
// Create image-based graphics context for top half
UIGraphicsBeginImageContext(size);
 
// Draw into context, bottom half is cropped off
[img drawAtPoint:CGPointMake(0.0,0.0)];
// Grab the current contents of the context as a UIImage 
// and add it to our array
UIImage *top = [UIGraphicsGetImageFromCurrentImageContext() retain];			
[_imagesTop addObject:(id)top];
 
// Now draw the image starting half way down, to get the bottom half
[img drawAtPoint:CGPointMake(0.0,-[img size].height/2)];
// And store that image in the array too
UIImage *bottom = [UIGraphicsGetImageFromCurrentImageContext() retain];			
[_imagesBottom addObject:(id)bottom];
 
UIGraphicsEndImageContext();

Now our _imagesTop and _imagesBottom arrays contains all the images we need, let’s get ‘em moving.

Applying animation

The axes and anchor point of a CALayer

_fliplayer.anchorPoint = CGPointMake(0.5, 1.0);

I use the CATransform3DMakeRotation function to create a transform around the X axis using the vector [1,0,0]. For the first part of the motion, we go from 0 radians to pi/2 radians (0 to 90 degrees), using an explicit CABasicAnimation object:

CABasicAnimation *topAnim = [CABasicAnimation animationWithKeyPath:@"transform"];
topAnim.duration=0.25;
topAnim.repeatCount=1;
topAnim.fromValue= [NSValue valueWithCATransform3D:CATransform3DMakeRotation(0.0, 1, 0, 0)];
float f = -M_PI/2;
topAnim.toValue=[NSValue valueWithCATransform3D:CATransform3DMakeRotation(f, 1, 0, 0)];
topAnim.delegate = self;
topAnim.removedOnCompletion = FALSE;
[_fliplayer addAnimation:topAnim forKey:[NSString stringWithUTF8String:k_TopAnimKey]];

Drawing layer content

Unfortunately a CALayer doesn’t have a “back”, so creating the flip layer is not as easy as it could be. We need to add a step in the process to change the image displayed by the flip layer half way through its fall. We can do this in a variety of ways, I’ve chosen to set a delegate object to perform the drawing of the layer contents:

[_fliplayer setDelegate:_layerHelper];

This means our delegate object (_layerHelper) has its drawLayer method called whenever the layer needs to display its contents. Be wary though, this may not be as often as you think! Core graphics aggressively caches the contents of the layer, and will only call your delegate when absolutely necessary. You can force it to happen by calling setNeedsDisplay, which invalidates the contents of the layer:

[_fliplayer setNeedsDisplay];

I encapsulated this in the helper object itself, by providing a custom implementation of the isTop property setter that calls setNeedsDisplay. At least that way callers don’t have to be aware of this requirement.

The drawLayer method itself simply draws one of two images into the context, depending on whether it’s in ‘top’ mode. A slight further complication is that when drawing the bottom part, the image needs to be inverted to display upside down. This is achieved by applying a translation and scale to the context before drawing the image:

CGContextSaveGState(context);
CGContextTranslateCTM(context, 0.0f, r.size.height);
CGContextScaleCTM(context, 1.0f, -1.0f);
 
CGContextDrawImage(context, r, [_imgTop CGImage]);
 
CGContextRestoreGState(context);

Avoiding implicit animation

I’ve tried to avoid using implicit animation here, instead using explicit construction of CABasicAnimation objects, because we need some control of when things happen, and the order in which they happen. Normally, it would be easier to just set some property of the view and let the OS manage animating it. Many properties have default transition that are applied whenever they’re changed, so it takes some additional effort to make them change immediately.

In our example we change the contents of the top half and show the flip layer, but we need that to happen immediately so we disable actions within an explicit transaction that we then commit. This gives us the control we need, and avoids the default behaviour of the transition occurring when the message loop next runs.

[CATransaction begin];
[CATransaction setValue:(id)kCFBooleanTrue
				forKey:kCATransactionDisableActions];
_toplayer.contents = (id)[[_imagesTop objectAtIndex:[self getNextIndex]] CGImage];
_fliplayer.hidden = NO;
[CATransaction commit];

Getting some perspective

Although the images I’ve used to illustrate this post are isometric, what we really want is a front-on perspective view, so that the falling piece seems to stick out. For this we have to set a sublayerTransform on the view’s inherent layer, by setting an individual element on a transform:

CATransform3D aTransform = CATransform3DIdentity;
float zDistance = 1000;
aTransform.m34 = 1.0 / -zDistance;	
[self layer].sublayerTransform = aTransform;

This gives us a much more realistic look.

Putting it together

Wow, so now we’ve got some appropriately sliced images, some layers and some animations. Let’s put them together to get the final effect.

The animation states (last is same as first)

- (void)animationDidStop:(CAAnimation *)oldAnimation finished:(BOOL)flag
{
	[self changeState];
}

• TopToMiddle: show flip layer, and start rotation through 90 degrees
• MiddleToBottom: change image on flip layer, start rotation through further 90 degrees
• ResetForNext: change displayed top and bottom layer contents and hide flip layer

Let’s see how it looks in action:

Form factor

The most obvious reason that this kind of interface works on the iPad is the form factor of the device itself. Think about it: an iPhone is a small (well, smallish) object which has no physical, real-world equivalent. It’s not supposed to “be” anything other than a phone. The iPad, on the other hand, is intended to mimic an actual object, or rather, a variety of objects: a book, a newspaper or an address book for example. The interface for each of these uses reflects this directly: the address book looks like a “little black book”, complete with tabs.

The iPhone uses some of the same metaphors – like the address book for contacts – but the representation is abstract, whereas the iPad UI is literal, using an actual image of a book. The abstraction is necessary on the iPhone in order for it to remain usable with a smaller screen. Compare the iPad and iPhone Contacts interfaces:

The realistic looking address book application on the iPad

iPhone contacts interface

The abstract to realistic continuum has been explored extensively in the world of icons (explained brilliantly here), but less so in the interface as a whole. Obviously some apps don’t have any form of real world equivalent (think System Preferences.app), so they’re likely to stay in a more abstract form.

Display fidelity

Of course, it’s always been possible to represent real objects on the screen, but without a display capable of a decent resolution it probably won’t look real. So another enabler for naturalistic UI is display technology: the more pixels and colour depth the better when it comes to rendering physical things. Realism often relies on subtle lighting and rich textures to achieve the desired effect. Anyone who’s ever attempted to render wood grain using 8-bit colour will appreciate how much easier it is with a 32-bit 1024×768 display.

The need for display quality goes hand-in-hand with a need for more processing power to push the pixels. If the processing “oomph” required to render your interface with sufficient quality means that it’s not responsive enough, then you’ve failed by any measure. The iPad seems to have reached a sweet spot where display capabilities, raw power and form-factor meet to make naturalistic UIs a feasible approach.

Direct manipulation

Up until the relatively recent introduction of direct manipulation technologies like touch-senstive displays, there’s a been a forced physical separation between users and their computer in the form of a mouse and keyboard. A drag operation using a mouse involves disconnected, abstract operations like pointing and clicking, which bear no relation to the object being acted upon. As such, there’s little use in having an extremely realistic representation of something to interact with, because the suspension-of-disbelief has already been broken.

But in the world of touch-based devices, our interactions are much closer to the physical actions we’d make, so having a closer visual representation to go along with it is beneficial. For instance, turning a page in an eBook and actually seeing the page flick over seems consistent when using a natural gesture like flicking a finger across the screen. Otherwise if you had to click a corner of the page and drag it across in a poor imitation of the physical action the visual aspects can quickly start to feel superfluous.

Perhaps given the iPad’s larger interaction surface we can expect to see a wider and more expressive range of gestures supported?

History

Of course, naturalistic UI isn’t really a new thing. It’s in use in quite a few applications already, albeit mostly in desktop apps, where capable hardware is more readily available. In fact, there was much a-twittering (not all of it from Will Shipley) about the similarity between the demo’d iPad apps and the interface of the fantastic Mac application Delicious Library. I’m sure of course – how do they say it in the movie credits – the similarity is entirely coincidental…

Delicious Library UI

iBooks UI

One of the first widespread uses of naturalistic UI elements was in software synths. Guys like Propellerheads and Native Instruments set about creating exact software duplicates of well-known and well-loved bits of mid-80’s audio hardware. The UIs included virtual versions of a huge number of physical elements that had never been used before: including things like sliders, LEDs, toggle buttons, rotary knobs and rocker switches. These interface elements were probably first seen in software-based mixing desks, but the techniques used to make them feel real have gone on to be used in many different ways. Even the lowly push button in both Windows and MacOS got an upgrade by application of some of the 3D effects that they originated.

Where Now?

It will be interesting to see what other types of app are given the iPad realism makeover. It certainly looks like all of the standard Apple apps are being changed, including Notes, iCal, iBooks and Contacts, which were shown in the launch presentation, but how will 3rd-party developers manage? It undoubtedly takes extra effort to make something look convincing when it can be compared to the physical object itself.

Apple has previously managed to get a good level of consistency in its abstract UIs by providing the various interface elements ready-made, but will that approach scale to this kind of UI? No doubt the developers who take great pride in the interfaces now will continue to strive to create apps that sit comfortably alongside Apple’s own, but perhaps there’ll just be a more obvious visual distinction between those guys and the less polished application user interfaces than there has been in the past.

The wow+flutter programme concentrates on “progressive motion graphics”, which can include pretty much anything, while other parts of the festival are more specific: wavelength for music videos, j-star for japanese shorts etc. You can check out the entire list of w+f films are here, but these are my favourites:

Ubik: voxel
I’m a fan of the augmented reality style when it’s done well; and it certainly is here. Striking abstract graphics combine with a haunting location.

Impactist: Parallelostory
A beautifully subtle animation style with a 50’s influenced colour and texture palette.

Bruno Dicolla: Return as an Animal
This featured a striking high-contrast rotoscoped style that looked great on the big screen. Unfortunately the shimmering starfield effect is a little lost in the on-line version. I could almost hear art directors reaching for their iPhones to jot down “must copy this at next available opportunity”.

Christian Schlaeffer: Agent Orange Ready
A dark, disturbing animation in a comic style.

The Lows

Ironically I found the most obviously “digital” films the most disappointing. There was a little too much in the Chris Cunningham inspired glitchy, razor-cut, super-shiny organic/mechanic vernacular. And there was also one film – which I won’t name – that looked shockingly like a Tia Maria commercial. Still, in general the signal to noise ratio was very good.

The Laughs

Some of the most memorable films are those with a punchline; the short film format is a great way of getting a laugh with a simple, funny idea. There’s no risk of stretching it too thin.

Mato Atom: Docking
A laugh-out-loud concept executed nicely and succintly, and it even goes out with an anti-war punchline.

So there you are; just 5 of the 30+ shorts that were shown. If you’re into animation or cutting-edge digital film-making, I’d recommend catching a onedotzero screening wherever you can, either in London or on one of the tour locations.