Multi-Touch Ink Demo

I’ve recently added Ink support to the Bespoke Multi-Touch Library. This allows for single-touch, pen-based/writing-style interaction through a multi-touch surface. This video demonstrates the collection of ink stroke data for training and classifying arbitrary symbols.

Multi-Touch Calibration & Particle trail Demo

Below are a couple of videos that demonstration some functionality of the Bespoke Multi-Touch Framework.

The calibration system compensates for differences in alignment between the camera, surface, and projector. This algorithm takes a series of sample points to calculate offset vectors and interpolates the remaining screen coordinates. The system generates a lookup table for quick (no run-time calculation) conversion between camera and screen points, and the calibration table can be serialized for automatic loading.


The next video is a demo app that produces 3D particles at every multi-touch interaction point. This is an extremely simple application, and is included in the framework package — coming soon!

Robotic Control with the Nintendo Wiimote

In the last post I discussed a gestural control system using the Nintendo Wiimote. An interesting application of this system is for controlling unmanned vehicles. My work with the ACTIVE lab (Applied Cognition and Training in Immersive Virtual Environments) often involves research in robotics, and a colleague and I have just finished a paper on using the Wiimote as the input device. There’s quite a bit of interesting technology behind the scenes here, but I think I’ll let the couple of videos below do most of the talking. Here’s a screenshot of the GUI that’s running on my laptop (but isn’t really visible in the videos):

The GUI provides telemetry for latitude, longitude, altitude, speed, heading, and streaming video. The software also allows the user to connect to each of these services independently and provides appropriate feedback on which services are active.

Wiimote Gesture Recognition

The Nintendo Wiimote is a remarkable, inexpensive device for exploring 3D user interaction. With 3 accelerometers, the Wiimote allows motion control and communicates wirelessly using the BlueTooth communication protocol. Open-source libraries (such as Brian Peek’s Wiimote.NET library) allow access to the Wiimote from a PC.

Many of the games on the Wii, use the Wiimote in a very simplistic way — mainly looking for thresholds on an axis to indicate an action. You can see this behavior if, for example, you swing the Wiimote above your head and maybe the golf club still swings as if you’d moved the controller correctly. Accurate gesture control — precise recognition of hand/arm movement — has been largely missing from most Wii games.

A colleague and I have devised a set of 29 features that can be used to classify the stream of Wiimote XYZ accelerometer values into a recognizable motion. Using a machine learning algorithm, we can train an arbitrary set of gestures which can then be mapped to actions.

In the video below, I present a graphical user interface for training and classifying gestures, providing output through sound, animation, and even a tactile belt. Importantly, the underlying recognition system is an API — allowing pre-trained data to be used by any application. I’ve used the system now in a couple of XNA video games (in fact the animation in the demo here is presented through XNA), in a Windows app, and even to control a robot (future post).

SurfaceSimon Redux

With the initial work on the Multi-Touch table I had issues with coupling between the acrylic and touch points. You had to press quite hard to frustrate the internal reflection of the infra-red light. The issue here is that we need to change the interface that the light is encountering from acrylic-to-air, to acrylic-to-finger. However, when you press on the acrylic directly, micro air gaps exist between your finger and the acrylic, and at those points the interface hasn’t changed. To remove those micro air gaps, you can press really hard or moisten your fingertips a bit. Neither of these solutions is particularly great, and an alternative exists — to form a compliant surface between acrylic and your touch points. I’d attempted this several months ago with a silicone rubber product called Ecoflex. Unfortunately that material wasn’t usable when placing the projection material (Rosco Gray) on top of the rubber. I suspect that the EcoFlex didn’t cure properly, or was just the wrong type of silicone rubber. At the time I was under a time crunch, and just resorted to no compliant surface, and tracing paper taped to the bottom of the acrylic as a projection diffuser.

Over the past few weeks I’ve been addressing this issue (with the help of new mechanical engineer we hired at the lab) and I’m very pleased with the results. This time around, we used SORTA-Clear 40, another silicone rubber, and after a few attempts have much better response out of the table. Here’s a video of the latest effort, with a bit of commentary.


Multi-Touch Videos

Last semester I built a Multi-Touch display and a corresponding framework using .NET and XNA. I have much to say on the subject, but just wanted to post a couple of videos today.

This first demo application is called SurfaceSimon. It’s a simple implementation of the classic ’80s memory game Simon. While the game itself is really single-touch (unless I implement a multi-player version) it demonstrates the basic functionality of the library.

The next video is a bit more involved, and demonstrates hit-testing (the button presses) and point tracking (labeling a touch point from one frame to the next). This application is called SurfaceCommand and heads in the direction of a real-time-strategy game interface.

Head Tracking in C# and XNA

Over the summer, I took a virtual reality class and examined a number of interesting technologies like augmented reality and 3D tracking. As a final project I wrote an XNA application that explored multiple input sources. I’d already written XNA components for the mouse, keyboard, and the XBox 360 gamepad, but I wanted to explore 3D head tracking and the Nintendo Wiimote. The next article will focus on the Wiimote (and I’m also writing a classifier for training and detecting 3D gestures for the Wiimote — also a future post), but this article is on 3D head tracking.

3D head tracking locates and follows your head in 3D space — mapping your head movements and position to some computer application. Common uses for head tracking include: virtual reality input, games, and attention monitoring (watching where you’re looking for potential feedback in case your eyes should be “on the road”). Full-body tracking is commonly used for motion capture (mocap).

A variety of technologies exist for head tracking, including: magnetic sensing, acoustic (ultrasonic), optical detection, and hybrids of these types. There are pros and cons to each technology, but I think that optical tracking has a lot of positives (and there are ways to mitigate the significant negative of visual occlusion) and this is the type of hardware I used for my project. Specifically, I employed the TrackIR from NaturalPoint. This is a fantastic piece of hardware at a very reasonable price ($179.95). The TrackIR (shown below) uses Infra-Red light , a 46-degree field of view, and has a seriously fast 120 fps sample rate.


The TrackIR is made of two components — the camera (above left) and a set of three points that either emit or reflect IR. The camera sits on top of your monitor and houses a set of IR LEDs that can be used with the retro-reflective track clip (above  right) that attaches to a baseball/golf hat. Alternately, you can use the TrackClip Pro which emits light from three LEDs and clips on over your headphones. The camera uses these three points to track your head’s position, yaw, pitch, and roll and makes this data available through a C++ API. The hardware is great, but I was completely sold by the API. Available as a download on the NaturalPoint website, the API is exposed as a set of COM interfaces. This makes it very simple to access from .NET languages like C#. Moreover, the documentation is easy to read and comprehensive. Seriously, I can’t say enough about the quality of the TrackIR product and the great experience I’ve had with the folks at NaturalPoint. The product performed beautifully, the API was excellent, and getting support couldn’t have been easier. They have a number of other products, including a line of mocap cameras (I’m looking at the OptiTrack FLEX:V100 for a Multi-Touch surface I’m working on — more about that later).

My first step in using the API was to recreate a basic version of a sample application included with the TrackIR. The screen shot below shows some basic functionality, including a slider for adjusting the light filter threshold (useful for operating in noisy IR environments), the image captured by the camera, and output for the yaw/pitch/roll and X/Y/Z values associated with the three-point head clip.

There’s also a button and combo box for turning the camera’s LEDs on/off. That’s a pretty cool feature of the camera/API that I needed for an application where I didn’t want the built-in IR LEDs to illuminate. I created an enumeration called TrackIRComponent.LED that maps the 4 types of LEDs on the camera (IR illumination, Green, Red, and Blue — these last three are for status and the like).

This simple app serves as a testbed for the C# TrackIRComponent I wrote. The component is a wrapper to the underlying COM API that exposes what I found to be the most useful data. Specifically, the class looks like this:


The Start() and Stop() methods turn the camera on and off and the DrawFrame() method puts a camera frame to a window handle (which you can get from a Form or Panel instance through the Handle property). The Update() method should be called every frame to get the latest data. The component updates the HeadVector property with each Update() call also — and it’s this HeadVector that provides the X/Y/Z and Yaw/Pitch/Roll data. A sample update loop (typically called in a thread) looks something like:

There’s almost nothing special you have to do to use this component in XNA — and it’s easier than using it with Windows Forms because the TrackIR is meant to be polled, and XNA provides the Update() loop for exactly such a purpose. Assuming you have some actor instance that has a TrackIRComponent, and you want to rotate the actor with your head, you could do something like:

Note, that the GameSettings.TrackIRGain property is used to modulate the head tracking data (in my case I set this to 0.02 so the space ship I’m steering doesn’t turn like crazy when I move my head a little).

Anyhow, I think this post is becoming awfully long, so I’m going to leave a more detailed discussion, of the simple XNA game that I created to make use of the TrackIR and the Wiimote, to the next post. But before I do — here’s the source code  to the TrackIRComponent and the testbed application. If you have any questions or comments, just shout.


Square Wave Blogging

I’ve noticed a trend in my blogging pattern — specifically, that it resembles a square wave. The last few months have been very busy for me, with the Fall semester at UCF has demanded all of my attention. I’m taking three interesting classes on Human Computer Interaction, Machine Learning, and Pen-Based User Interfaces. This should produce a run of articles, that I’ll be posting over the coming weeks and months. However, I want to start with a two-part article on Head Tracking and the Wiimote using C# and XNA. This work came from a Virtual Reality course I took over the summer. I’m posting those articles next. Hopefully this is the start of another blogging duty cycle :)


Andii – Phase II

It’s been quite a while since I’ve posted anything on the Andii navigating robot project. For most of 2006, as often happens with hobby projects, I shelved the robot. I picked it up a couple of times to present to the Orlando .NET Users Group, and for a Seminole Community College (SCC) open house event. But otherwise, the project was dormant.

So, when I started work at the University of Central Florida in the Modeling & Simulation program, I had an opportunity to continue this effort. This latest iteration picks up where I’d left off, and adds Video and GPS telemetry and a much more complete Windows client.

The software is broken into three primary components: the client, server, and shared interfaces. The server runs as a Windows service, and is composed of three independent “servers”, for vehicle control, and video and GPS telemetry. The client can connect to the each of these servers individually or at the same time.

The shared interfaces are for GPS telemetry and vehicle control using .NET Remoting. The video server offers up video data using a custom TCP/IP server instead of .NET Remoting because of the nature of the data (the heavy resource consumption for capturing and transmitting the video).

The service software employs the BrainStem.NET library for servo control, as well as two external software libraries for GPS and video management. The GPS library is GPS.NET from GeoFrameworks (~ $279) and the video library is DirectShow.NET (open source). The DirectShow.NET library wraps the Microsoft DirectX library DirectShow for use in .NET applications. The client software also makes use of DirectX for joystick input (DirectInput) and the GPS.NET library for speed and compass controls.

Here’s a screenshot of the client. If anyone’s interested in the source code, let me know and I’ll make it available (though you’ll need the GPS.NET controls).

Color Matching (XYZ to RGB)

Last semester I wrote a color matching application for a graphics class (using C# and XNA — XNA only for rendering the output texture, which is a kind of overkill, but no matter). I’ll let Wikipedia describe the CIE color space, the eye, color vision and so on — I definitely wouldn’t do these topics justice. But the general idea is, that in order to display a sample of light on a computer monitor, we have to convert the spectral values to RGB values.

For this project, I was supplied spectral values for the Macbeth Color Checker. These were stored in a comma-delimited text file with wavelengths samples between 380 and 780 nanometers in 5nm increments. Using the CIE 1964 color space (also stored in a text file and in 5nm increments between 360 and 830nm), and NTSC chromaticity coordinates:

R 0.67 0.33
G 0.21 0.71
B 0.14 0.08
White Point 0.313 0.329

we can go from spectrum samples –> XYZ coordinates –> RGB.


The output is a reproduction of the color checker as a set of 24 textures. The coding is pretty straight forward, and I’ve included it here . Kind of an obscure topic, but i hope it comes in handy for someone :)


« Previous PageNext Page »