Friday, April 24, 2009

Gesture recognition in GWT and IT Mill Toolkit: SimpleGesture

SimpleGesture is a pure GWT (and IT Mill Toolkit) implementation of the mouse gesture recognition method described by Didier Brun at bytearray.org (as I understand it). It allows you to register easy to understand (human readable) gestures, and receive events when these occur. Additionally, SimpleGesture can record new gestures, so that you don't have to write them by hand.

Here is a short demo video - I'm using a Wii (as a mouse, no special integration) to control a simple demo application, and you can see me record a new gesture as well:




You can try the demo application live here: http://marc.virtuallypreinstalled.com/SimpleGesture
The demo application is made using IT Mill Toolkit - SimpleGesture is actually a component for IT Mill Toolkit, but it uses GWT on the client side and the actual GWT widget is made to be stand-alone.

SimpleGesture was made using my "10% time" here at IT Mill (yes, we're only half Google, but that's not half bad, if you ask me...;)

Applications
Mouse gestures have their place, and are not always appropriate - for instance, for regular desktop use you might find that keyboard shortcuts are often better. But if the pointing device is your main input device, or you do not have a keyboard, gestures might be the way to go. The video demo, using the Wii, should demonstrate this quite clearly. Also, hitting a small button in the upper left corner with the Wii controller can be quite a challenge, while the gestures work quite well (hello Fitt's Law!)
I'd imagine there are a few uses for touch -based interfaces as well... ;-)

Technical details
I had used the bytearray mouse gesture library for a Flash project before, and I love how simple the method is, and yet it works very well, so I decided to try to implement it in pure GWT.

The method works by assigning a number depending on which direction the mouse moves in (moving right will produce "0", moving down will produce "2", and so on), and comparing the resulting string of numbers to the registered gestures. The gesture with the lowest Levenshtein distance (and below a set threshold), is considered a match.

For instance, the "rotate clockwise" gesture used in the demo looks like this: "7001"
How is that for simplicity?

SimpleGesture uses the Levenshtein Distance method from Jakarta Commons (getLevenshteinDistance() from StringUtils) on the client-side, without modifications, compiled using GWT. I think this shows one of the strengths of GWT - the ability to make use of existing Java code in the browser.

Use it!
Feel free to use and improve SimpleGesture - it uses the Apache License, just as IT Mill Toolkit and Jakarta Commons. Hopefully you'll send me some patches... :-)

If you're using IT Mill Toolkit, you can get the whole demo project from SVN.

To use it in "plain" GWT, (w/o IT Mill Toolkit), just get the GWT stuff from SVN (or just ISimpleGesture.java). ISimpleGesture a GWT Widget, and there are only two IT Mill Toolkit specific things (which you can remove): "implements Paintable", and the "updateFromUIDL()" -method.

Note that SimpleGesture currently uses EventPreview and listens to the whole window, which is not ideal - event preview is somewhat problematic, and it would be nice to be able to use different gestures in different parts of the window. This is one area that should be improved (patches, anyone?), and it could be made more configurable as well.

Feedback
Please feel free to comment here, discuss at the IT Mill Toolkit forums, or email me at marc.englundÄTitmill.com

4 comments:

Jouni said...

Wow! It's not only the gestures that are amazing on themselves, but the application that you came up with for them, photo manipulation on the Wii, is brilliant.

I can see my mother editing her vacation photos with this in the living room :)

Now what sort of a gesture would trigger a color balance editor...

Phil said...

This is really nice.
Took me a while to figure the icons at the top showed some of the gestures, but thats a nice way of showing them too, maybe they could be shown with some form of list showing the gestures and new ones would be added with a picture of the gesture too. I failed miserably to get it to work on a touch ipod (probably me), but it would be real nice if it did.

emarc said...

Thanks, Jouni!

Phil, good suggestions! I've been thinking about making a more 'standard' gesture-binding ui in the demo (i.e. a list), but I have not worked up the energy for it yet ;-)
And you're right: the iPhone/iPod Touch do not work, nor does the Nokia Safari-based browser. The reason is that these touch-devices handle the mousemove event a little different (due to the touch implementations), so it would need some special event-handling. It's on my TODO. I'm still using GWT 1.5 too, so I'll probably convert to 1.6, start using the new event handling, and implement iphone support - at some point :-)

Best Regards,
Marc

Flavio said...

I was going to look ur demo but:

exception

javax.servlet.ServletException: java.lang.OutOfMemoryError: Java heap space
com.itmill.toolkit.terminal.gwt.server.ApplicationServlet.service(ApplicationServlet.java:598)
javax.servlet.http.HttpServlet.service(HttpServlet.java:717)