Multitouch Unplugged

Introduction – Hardware choices

At the beginning of the project, we were given the following pieces of hardware  :

  • A multitouch Table from Polytouch (32″)
  • 2 piezo microphones
  • 2 RFID Readers
  • 1 Kinect camera
After a first brainstorming, we all decided to use only one RFID Reader, and the 2 Piezo microphones.
In our setup, the piezo is hidden behind a metal part. We have two metal parts, fixed on two wooden tables.

The scenario

The most difficult part in this kind of project, is to come up with an idea for an intuitive, innovative and funny scenario.

Our the scenario is a game, taking place in a bar, A virtual bar environnement was created, using custom textures and assets aplied on planes in the Unity Game Engine.This is a quick and easy way to setup our scene. In the foregroud, you san see the bar, the beer tap (coming soon), and of course, beer glass. In the background we have some tables. The frame on the wall displays random pictures fetched from the internet.

 

You can see our first prototype just here :

As seen earlier, there are two table. The first oneis  for the bartender, and the second for the waiter. The RFID Reader is used to order beer.

In a first time, the waiter must to check with several RFID Tag, hidden between two bar coasters (with some pictures from severals Belgian beer manufacter).

The Bartender receives a todo list, with the ordered beer. The glass is filled by some “tapping” from the bartender. When the glass is full, the bartender does a “Smash” move on the table, he then makes the virtual beer slides on the bar with a natural gesture.

The Waiter must stop the beer, before it falls down from the bar, after that, the cycle can restart.

We can increase the scenario with some sound, for each action.

Gestures recognition

The piezo microphones works like a usual microphone. It sends an electrical signal, with a diffrent frenquency spectrum  for each gestures. We can recognize 4 gestures :

  • Smash : to stop the beer and place it on the bar, before sliding
  • Tapping : to fill up the beer glass
  • Scratching : to make the glass slide on the bar
  • Knocking : not used !

First, a PureData Patch is used to filter some frequency band and analyses the amplitude in the matched band. If we detect a combinaison of severals filters, we can identify if the gesture is a smash, a tap, or something else…

When a gesture or action is identified, we use UDP message (through a local loopback interface) to send them to our Unity application, and trigger the corresponding animations.