Gesture recognition in GWT and IT Mill Toolkit: SimpleGesture

SimpleGesture is a pure GWT (and IT Mill Toolkit) implementation of the mouse gesture recognition method described by Didier Brun at bytearray.org (as I understand it). It allows you to register easy to understand (human readable) gestures, and receive events when these occur. Additionally, SimpleGesture can record new gestures, so that you don't have to write them by hand.

Here is a short demo video - I'm using a Wii (as a mouse, no special integration) to control a simple demo application, and you can see me record a new gesture as well:




You can try the demo application live here: http://marc.virtuallypreinstalled.com/SimpleGesture
The demo application is made using IT Mill Toolkit - SimpleGesture is actually a component for IT Mill Toolkit, but it uses GWT on the client side and the actual GWT widget is made to be stand-alone.

SimpleGesture was made using my "10% time" here at IT Mill (yes, we're only half Google, but that's not half bad, if you ask me...;)

Applications
Mouse gestures have their place, and are not always appropriate - for instance, for regular desktop use you might find that keyboard shortcuts are often better. But if the pointing device is your main input device, or you do not have a keyboard, gestures might be the way to go. The video demo, using the Wii, should demonstrate this quite clearly. Also, hitting a small button in the upper left corner with the Wii controller can be quite a challenge, while the gestures work quite well (hello Fitt's Law!)
I'd imagine there are a few uses for touch -based interfaces as well... ;-)

Technical details
I had used the bytearray mouse gesture library for a Flash project before, and I love how simple the method is, and yet it works very well, so I decided to try to implement it in pure GWT.

The method works by assigning a number depending on which direction the mouse moves in (moving right will produce "0", moving down will produce "2", and so on), and comparing the resulting string of numbers to the registered gestures. The gesture with the lowest Levenshtein distance (and below a set threshold), is considered a match.

For instance, the "rotate clockwise" gesture used in the demo looks like this: "7001"
How is that for simplicity?

SimpleGesture uses the Levenshtein Distance method from Jakarta Commons (getLevenshteinDistance() from StringUtils) on the client-side, without modifications, compiled using GWT. I think this shows one of the strengths of GWT - the ability to make use of existing Java code in the browser.

Use it!
Feel free to use and improve SimpleGesture - it uses the Apache License, just as IT Mill Toolkit and Jakarta Commons. Hopefully you'll send me some patches... :-)

If you're using IT Mill Toolkit, you can get the whole demo project from SVN.

To use it in "plain" GWT, (w/o IT Mill Toolkit), just get the GWT stuff from SVN (or just ISimpleGesture.java). ISimpleGesture a GWT Widget, and there are only two IT Mill Toolkit specific things (which you can remove): "implements Paintable", and the "updateFromUIDL()" -method.

Note that SimpleGesture currently uses EventPreview and listens to the whole window, which is not ideal - event preview is somewhat problematic, and it would be nice to be able to use different gestures in different parts of the window. This is one area that should be improved (patches, anyone?), and it could be made more configurable as well.

Feedback
Please feel free to comment here, discuss at the IT Mill Toolkit forums, or email me at marc.englundÄTitmill.com