Archive for August, 2008

Multi-Touch Network Gateway

I’ve recently been developing a software gateway for transmitting multi-touch data across a network. There are a few immediate uses for this:

  1. Dedicating a machine for processing the input, thereby freeing up resources that can be used for running a multi-touch application (though at the expense of some network latency).
  2. Allowing multiple applications to utilize the multi-touch data simultaneously.
  3. Support for non-.NET environments to access the multi-touch input. This includes entirely different operating systems (of course the transmitting environment hosting the gateway is still Windows and .NET).

I’ve included support for the gateway within the XML configuration file, and it’s likewise supported from within the multi-touch XNA component. This means that existing XNA applications written against the framework do not need any modifications in order to use the gateway. By setting the useLocalSurface value to “false” and specifying the gateway element, you can easily toggle between retrieving data directly from the multi-touch surface or from a remote surface. These Xml settings are displayed below.


The gateway uses the Open Sound Control specification to efficiently transmit multi-touch points (85 bytes per point). I’ve made my C# implementation of OSC available as a stand-alone package, and it’s also included with the Bespoke Multi-Touch Framework download. I adopted the OSC specification since it was being employed by the Touchlib multi-touch library, and the reactable reacTIVision software system. The reacTIVision folks have developed the TUIO protocol (which sits on top of OSC) for transmitting tangible and multi-touch interactions, and though I didn’t implement the TUIO protocol directly, I’ve definitely followed their lead.

The gateway’s application-level protocol is extremely simple. There are two types of message that are transmitted: Point messages and Alive messages. Point messages relay the specifics of a single multi-touch interaction point. Alive messages contain the IDs of the set of active points. Point messages are always packaged together within an OSC bundle, but Alive messages are communicated within the point bundle and as stand-alone messages (sent for periods of time with no surface interaction). The specifics of each message type are shown below.





Point messages are transmitted only when surface interactions are detected. Once a point has been transmitted, it’s corresponding message will only be resent when the point has moved. This makes the system efficient in how much data it sends every frame.

Since the underlying protocol is UDP, it’s quite possible that messages will not be received by the client, or might be received out-of-order. The C# client implementation that I’ve included, drops out-out-order packets by checking the timestamp within the Point message. Additionally, Alive messages are useful in determining which points are genuinely active, versus those that might be hanging around because of a dropped packet.

Another interesting note on the gateway is its support for multiple types of transmission: unicast, multicast, broadcast, and local broadcast. With unicast, I’ve implemented a subscription service, again via OSC, and the gateway transmits heartbeat messages to verify that the client is still interested in receiving data. Each client (multiple simultaneous clients are supported) sends an OSC message with an address pattern of “/gateway/subscribe” and will begin receiving Point and Alive messages. The client must also listen for “/gateway/heartbeat” messages and respond with the same address pattern. The gateway transmits heartbeats every 10 seconds and each connection has a 25 second time-to-live. So the client has the opportunity to respond to two heartbeat messages before the TTL expires and the gateway stops sending packets to that client. Every heartbeat response refreshes the TTL to the original 25 seconds.

Unicast is perhaps the most efficient of the transmission methods, when you only have a few clients connected, because the gateway will only transmit to those specific clients and won’t send any data if no clients are subscribed. However, multicast, broadcast, and local broadcast are the easier clients to implement as they don’t require subscribe or heartbeat messages. Multicast and broadcast are in the same vein, in that they transmit packets to multiple destinations (I’ll leave it to Wikipedia to describe the differences in more detail). As for local broadcast, this is my own word for continuously transmitting data to the loopback address ( It seemed like a better way to address, what I felt to be, a common utilization of the gateway — the gateway running on the same machine as the client, without forcing the client to subscribe and respond to heartbeat messages. Just keep in mind that any of the non-unicast methods will continuously transmit data with no knowledge of what clients are listening.

The transmission mode, IP address, and port that the gateway uses can all be specified on the command-line. With no command-line options specified, the defaults are: local broadcast,, and 5101. The command-line options are:


The gateway runs as a Windows service, which means that you have to specify these command-line options through the Windows registry (unless you run the gateway from within the debugger, which bypasses the service startup). Edit the ImagePath key within HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\Bespoke.MultiTouch.OscGateway.Service, appending your any of these options to the command string. Alternately, it is trivial to transform the gateway utility into a non-Windows service. All of the functionality is in fact housed in the Transmitter class within the Bespoke.MultiTouch.Framework.Gateway.Osc namespace.

For anyone looking to implement this protocol on their own, I hope this description has been helpful. If you’re just interested in the functionality, use the included C# client or Receiver class, or the XNA multi-touch component which directly supports remote access. The gateway is included in the release of the Bespoke Multi-Touch Framework

If you have any questions, please let me know.


Bespoke Multi-Touch Framework Version 4.1

I’m pleased to announce the release of the Bespoke Multi-Touch Framework version 4.1. The primary additions in this release are:

  • Improved Calibration
  • Mouse Emulator
  • Open Sound Control (OSC) Gateway

I’d previously blogged about the improved calibration and the mouse emulator and will have a post out shortly on the gateway. There’s also a change in how hit testing is performed. In the 4.0 release, hit testing was done through the FtirFrame class. This functionality has been moved to the FtirPointCollection class, with instances exposed through the FtirFrame.FtirPoints and FtirSurface.CurrentFtirPoints properties.

If you have any questions of comments on this release, please let me know.


Wii on PC – Interview with Laura Foy

My advisor and I were recently at the Microsoft Faculty Research Summit where we presented a series of video games developed with the Bespoke 3DUI XNA Framework. While there I was very pleased to meet Laura Foy from Channel 10. She’s posted the following video of our interview.

Thanks Laura, very cool stuff!


Bespoke 3DUI XNA Framework

I’d like to announce the release of the Bespoke 3DUI XNA Framework — an open-source software library for developing games and simulations using 3D user interfaces.

I’d published the library last month, but hadn’t announced it on this blog. The full page on the library goes into much more detail including a series of videos for games that have been developed using the framework.


Improved Multi-Touch Calibration

I’ve updated the calibration system for the Multi-Touch Framework to use a 4-point system adapted from Johnny Chung Lee’s Wiimote Whiteboard project. This is the calibration used in the videos for the Mouse Emulator. I’ll be publishing the code for this shortly, but until then here’s a brief demonstration of the accuracy of the new system.