Bespoke Multi-Touch Framework version 4.2

I’m pleased to announce the release of the Bespoke Multi-Touch Framework version 4.2. After several weeks with the framework off-line, downloads are once again available,  This new release changes the license from the Microsoft Public License to BSD. Software changes include:

  • Changed the OSC transmission type for point IDs from byte[] to string. This was necessary as the flosc gateway doesn’t currently support byte arrays.
  • Added background filtering to the image processing pipeline (can be helpful in reducing IR noise).
  • Added user-defined filter points for explicitly removing interaction points.

If you have any questions concerning this release, please let me know.

Paul

Bespoke 3DUI XNA Framework Questionnaire

In an effort to evaluate and improve the Bespoke 3DUI XNA Framework, I’ve posted a brief questionnaire (7 questions). If you are using the framework, I would greatly appreciate your feedback.

Thank you!

Paul

Multi-Touch Network Gateway

I’ve recently been developing a software gateway for transmitting multi-touch data across a network. There are a few immediate uses for this:

  1. Dedicating a machine for processing the input, thereby freeing up resources that can be used for running a multi-touch application (though at the expense of some network latency).
  2. Allowing multiple applications to utilize the multi-touch data simultaneously.
  3. Support for non-.NET environments to access the multi-touch input. This includes entirely different operating systems (of course the transmitting environment hosting the gateway is still Windows and .NET).

I’ve included support for the gateway within the XML configuration file, and it’s likewise supported from within the multi-touch XNA component. This means that existing XNA applications written against the framework do not need any modifications in order to use the gateway. By setting the useLocalSurface value to “false” and specifying the gateway element, you can easily toggle between retrieving data directly from the multi-touch surface or from a remote surface. These Xml settings are displayed below.

SP32-20080816-172328

The gateway uses the Open Sound Control specification to efficiently transmit multi-touch points (85 bytes per point). I’ve made my C# implementation of OSC available as a stand-alone package, and it’s also included with the Bespoke Multi-Touch Framework download. I adopted the OSC specification since it was being employed by the Touchlib multi-touch library, and the reactable reacTIVision software system. The reacTIVision folks have developed the TUIO protocol (which sits on top of OSC) for transmitting tangible and multi-touch interactions, and though I didn’t implement the TUIO protocol directly, I’ve definitely followed their lead.

The gateway’s application-level protocol is extremely simple. There are two types of message that are transmitted: Point messages and Alive messages. Point messages relay the specifics of a single multi-touch interaction point. Alive messages contain the IDs of the set of active points. Point messages are always packaged together within an OSC bundle, but Alive messages are communicated within the point bundle and as stand-alone messages (sent for periods of time with no surface interaction). The specifics of each message type are shown below.

SP32-20080817-111221

 SP32-20080817-110916

SP32-20080817-110934

SP32-20080817-111836

Point messages are transmitted only when surface interactions are detected. Once a point has been transmitted, it’s corresponding message will only be resent when the point has moved. This makes the system efficient in how much data it sends every frame.

Since the underlying protocol is UDP, it’s quite possible that messages will not be received by the client, or might be received out-of-order. The C# client implementation that I’ve included, drops out-out-order packets by checking the timestamp within the Point message. Additionally, Alive messages are useful in determining which points are genuinely active, versus those that might be hanging around because of a dropped packet.

Another interesting note on the gateway is its support for multiple types of transmission: unicast, multicast, broadcast, and local broadcast. With unicast, I’ve implemented a subscription service, again via OSC, and the gateway transmits heartbeat messages to verify that the client is still interested in receiving data. Each client (multiple simultaneous clients are supported) sends an OSC message with an address pattern of “/gateway/subscribe” and will begin receiving Point and Alive messages. The client must also listen for “/gateway/heartbeat” messages and respond with the same address pattern. The gateway transmits heartbeats every 10 seconds and each connection has a 25 second time-to-live. So the client has the opportunity to respond to two heartbeat messages before the TTL expires and the gateway stops sending packets to that client. Every heartbeat response refreshes the TTL to the original 25 seconds.

Unicast is perhaps the most efficient of the transmission methods, when you only have a few clients connected, because the gateway will only transmit to those specific clients and won’t send any data if no clients are subscribed. However, multicast, broadcast, and local broadcast are the easier clients to implement as they don’t require subscribe or heartbeat messages. Multicast and broadcast are in the same vein, in that they transmit packets to multiple destinations (I’ll leave it to Wikipedia to describe the differences in more detail). As for local broadcast, this is my own word for continuously transmitting data to the loopback address (127.0.0.1). It seemed like a better way to address, what I felt to be, a common utilization of the gateway — the gateway running on the same machine as the client, without forcing the client to subscribe and respond to heartbeat messages. Just keep in mind that any of the non-unicast methods will continuously transmit data with no knowledge of what clients are listening.

The transmission mode, IP address, and port that the gateway uses can all be specified on the command-line. With no command-line options specified, the defaults are: local broadcast, 127.0.0.1, and 5101. The command-line options are:

SP32-20080817-121140

The gateway runs as a Windows service, which means that you have to specify these command-line options through the Windows registry (unless you run the gateway from within the debugger, which bypasses the service startup). Edit the ImagePath key within HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\ Services\Bespoke.MultiTouch.OscGateway.Service, appending your any of these options to the command string. Alternately, it is trivial to transform the gateway utility into a non-Windows service. All of the functionality is in fact housed in the Transmitter class within the Bespoke.MultiTouch.Framework.Gateway.Osc namespace.

For anyone looking to implement this protocol on their own, I hope this description has been helpful. If you’re just interested in the functionality, use the included C# client or Receiver class, or the XNA multi-touch component which directly supports remote access. The gateway is included in the 4.1.0.0 release of the Bespoke Multi-Touch Framework

If you have any questions, please let me know.

Paul

Bespoke Multi-Touch Framework Version 4.1

I’m pleased to announce the release of the Bespoke Multi-Touch Framework version 4.1. The primary additions in this release are:

  • Improved Calibration
  • Mouse Emulator
  • Open Sound Control (OSC) Gateway

I’d previously blogged about the improved calibration and the mouse emulator and will have a post out shortly on the gateway. There’s also a change in how hit testing is performed. In the 4.0 release, hit testing was done through the FtirFrame class. This functionality has been moved to the FtirPointCollection class, with instances exposed through the FtirFrame.FtirPoints and FtirSurface.CurrentFtirPoints properties.

If you have any questions of comments on this release, please let me know.

Paul

Wii on PC – Interview with Laura Foy

My advisor and I were recently at the Microsoft Faculty Research Summit where we presented a series of video games developed with the Bespoke 3DUI XNA Framework. While there I was very pleased to meet Laura Foy from Channel 10. She’s posted the following video of our interview.



Thanks Laura, very cool stuff!

Paul

Bespoke 3DUI XNA Framework

I’d like to announce the release of the Bespoke 3DUI XNA Framework — an open-source software library for developing games and simulations using 3D user interfaces.

I’d published the library last month, but hadn’t announced it on this blog. The full page on the library goes into much more detail including a series of videos for games that have been developed using the framework.

Paul

Improved Multi-Touch Calibration

I’ve updated the calibration system for the Multi-Touch Framework to use a 4-point system adapted from Johnny Chung Lee’s Wiimote Whiteboard project. This is the calibration used in the videos for the Mouse Emulator. I’ll be publishing the code for this shortly, but until then here’s a brief demonstration of the accuracy of the new system.

Multi-touch Mouse Emulator

I’ve added a mouse emulator to the Bespoke Multi-Touch Framework. The emulator is UI independent and passes mouse messages to the operating system. I’ve included a GUI that runs in the Windows systray and allows the emulator to be started/paused/stopped and for various settings to be adjusted. The gestures are:

Single point
Press and hold – Mouse move
Press briefly – Left click

Two points
Two brief touches next to each other. Move the mouse and double click
Press and hold right point and tap to the left. Left click
Press and hold left point and tap to the right. Right click
Press and hold left point and drag right point vertically– Mouse wheel
Press and hold bottom point and drag top point horizontally – Alt-tab (not actually a mouse command, but seemed useful)

Below are a couple of demonstration videos. The first is just on the Windows desktop, and the second video is a brief game of Starcraft.

 

The Bespoke Multi-Touch Framework

I’m pleased to announce the initial release of the Bespoke Multi-Touch Framework. You can download the release from here. There are a two packages available: source+pre-compiled-binaries; and source-only.

6 sample applications are included:

  • HelloWorld (the canonical intro app and almost one-liner for interacting with the API – outputs to a text console)
  • CalibrationDemo
  • InkDemo (Pen-style interaction on a multi-touch surface)
  • ParticleTrails (presents smoke-like particle effects at each interaction point)
  • SurfaceSimon (2D XNA version of the classic 80s memory game – demonstrates multi-touch hit testing)
  • SurfaceCommand (3D XNA application demonstrating a simple real-time strategy (RTS) style multi-touch interface)

The samples (except for HelloWorld) employ the XNA presentation layer and share a common set of key-commands, such as:

V – Toggle display modes (there are 5 display modes, 4 debug + 1 normal)
C – Calibrate
Esc – Exit the app

This release is admittedly light on the documentation. I’m actively working on the doc and will be updating it soon. I’ve also left out WinForms and WPF sample applications, but these presentation layers should work without issue. The release is labeled 4.0 (instead of 1.0 for a typical intro release) because I’ve been working on this for awhile and have done a few internal releases. Aside from these few disclaimers, the code should at least compile cleanly and hook to a DirectShow capable video camera. Be certain to configure the Bespoke.MultiTouch.Framework.config file to match your multi-touch surface.

I hope you find the framework useful. If you have any questions please let me know.

Paul

Multi-Touch Montage

Below is a video montage demonstrating many of the features of the Bespoke Multi-Touch Framework.

« Previous PageNext Page »