Projects » multitouch hand tracking tabletop

Multi-User Multitouch Tracking System

Working with Lumen Digital Studio in Wellington, New Zealand, I developed a computer-vision based multitouch hand tracking tabletop system, capable of tracking many hands simultaneously, for interactive culture & heritage projects. It is currently used in projects for Te Puia, Te Papa Tongarewa Museum of New Zealand, and Tourism Australia.

The system tracks the movement of multiple hands on a table surface in realtime, using an overhead infrared camera and custom software built using the open-source OpenCV computer vision library. It is able to consistently track as many hands as will fit around the table, and can also determine at which edge of the table each person is standing, so that any content triggered on the table can be properly aligned and/or delivered to the edge of the table, appearing in the right place for that person.

The tracker runs as an application that communicates with a 3D engine (or any other software) using OpenSoundControl packets.

Video

Download a video (512×384 h264/mpeg-4, 7MB, playable with QuickTime or VLC) which demonstrates the tracker in action at Te Puia. Developed in close collaboration with the Maori cultural guides of the institution, the interactive tabletop interface offers a Maori interpretation of land as a context for stories. Stories are associated with particular points on the landscape, which can be called up by touching the table surface at that point.

There are more videos of the handtracker being used in different applications below. These links will take you to the relevant section on the Lumen Digital Studio website.

for Te Puia:

for Tourism Australia:

Other applications:

  • DJ Table
    dj table
    A simple and fun example of the hand tracking technology in action.
last updated Thursday 23rd October 2008