Gesture Recognition in Flash

This is an older experiment, but I think it’s still fairly interesting, and I wanted to post it to the blog so that it has a home on my site. This system demonstrates converting user gestures into machine readable information – in this case, it simulates Palm graffiti recognition. Click and drag to write graffiti characters on the input pad, and the system will convert them into characters on screen. Supports almost the full range of graffiti inputs including letters, numbers, uppercase, backspace and punctuation.


Here’s a graffiti guide to help you out – the dot shows the starting point of the stroke. It also supports caps shift (upwards stroke), caps lock (two upwards strokes), and punctuation shift (single tap).

This algorithm was also put to use by Random Media on the Shinobi website. For a full explanation of the algorithm, you can click here.

The cool thing is that this is all based on a very generic pattern recognition engine that I wrote, which can be used to identify patterns in any kind of linear 2 axis data.

Grant Skinner

The "g" in gskinner. Also the "skinner".

@gskinner

16 Comments

  1. As always, really nice, mate.

    but… Somehow all the things I did turned out wrong at the text field.

    probably you made a chart on how to draw those letters to come up right, like the palm has.

  2. Added a graffiti guide. I haven’t had any problem drawing any of the characters – it’s at least as accurate as my old Palm Pro (though that’s not saying much). Then again, it is trained to my graffiti input.

  3. Very nice. I got all the letters fine. I’ll be taking a look at this soon.

    Cheers.

    +LA

  4. Really nice!

    The things that can be done with flash never stop amazing me.

  5. A more reliable algorithm may go like this:

    Record the position of the mouse at different intervals. Create two sub-graffities, x against t and y against t. For example, A maps to rising line in x versus t and a up and down line in y against t.

    Normalize those and do a fit to find the good letter. Because it uses speed instead of position, and because x and y are functions of t instead of things doing loops and such, it should be faster and more reliable.

  6. Someone actually released a system based on discrete gestures similar to what you described. It was less processor intensive, but far less accurate – this is because it relies on the accuracy of independent strokes, rather than a comparison across an entire character.

  7. Hello, very nice !

    The only problem is the url

    http://www.gskinner.com/playpen/graffiti.html

    isn’t working yet… Any idea ?

    Ciao, r0main

  8. i have a palm pilot and this is crap!!!!!!!

  9. I also would like to see the http://www.gskinner.com/playpen/graffiti.html page. Thanks!

  10. Harsha Hegde June 1, 2005 at 2:36pm

    Really sweet!! Excellent work indeed!!

  11. very nice

    good idea

  12. Why can’t I seem to get numbers to work?

    Also I would like to see the link fixed to the http://www.gskinner.com/playpen/graffiti.html page

    Cheers!

    Nice work by the way

  13. gesture recognition with webcam.

    First engine for recognition with webcam

    small video presentation for my program

    http://www.robotdimitron.com/oblici

    Autor: Dimitron

  14. First engine for gesture recognition using webcam with hand control( no color detection )

    http://interaktivnamatematika.com/oblici/

    I am from Macedonia where is born Alexander Macednien, its small country with bigs ideas.

    Thank you Grant Skinner for pioner idea and realisation of this program

  15. Hi Grant,

    I’m really interested in looking at the source for this but it seems to have disappeared. Can you let me know whether or not you plan to post it again?

    Thanks as always,

    Dave

  16. Nice Flash Example. Do u have more like that ?

Leave a Reply

Your email address will not be published. Required fields are marked *