Over the summer, I took a virtual reality class and examined a number of interesting technologies like augmented reality and 3D tracking. As a final project I wrote an XNA application that explored multiple input sources. I’d already written XNA components for the mouse, keyboard, and the XBox 360 gamepad, but I wanted to explore 3D head tracking and the Nintendo Wiimote. The next article will focus on the Wiimote (and I’m also writing a classifier for training and detecting 3D gestures for the Wiimote — also a future post), but this article is on 3D head tracking.
3D head tracking locates and follows your head in 3D space — mapping your head movements and position to some computer application. Common uses for head tracking include: virtual reality input, games, and attention monitoring (watching where you’re looking for potential feedback in case your eyes should be “on the road”). Full-body tracking is commonly used for motion capture (mocap).
A variety of technologies exist for head tracking, including: magnetic sensing, acoustic (ultrasonic), optical detection, and hybrids of these types. There are pros and cons to each technology, but I think that optical tracking has a lot of positives (and there are ways to mitigate the significant negative of visual occlusion) and this is the type of hardware I used for my project. Specifically, I employed the TrackIR from NaturalPoint. This is a fantastic piece of hardware at a very reasonable price ($179.95). The TrackIR (shown below) uses Infra-Red light , a 46-degree field of view, and has a seriously fast 120 fps sample rate.
The TrackIR is made of two components — the camera (above left) and a set of three points that either emit or reflect IR. The camera sits on top of your monitor and houses a set of IR LEDs that can be used with the retro-reflective track clip (above right) that attaches to a baseball/golf hat. Alternately, you can use the TrackClip Pro which emits light from three LEDs and clips on over your headphones. The camera uses these three points to track your head’s position, yaw, pitch, and roll and makes this data available through a C++ API. The hardware is great, but I was completely sold by the API. Available as a download on the NaturalPoint website, the API is exposed as a set of COM interfaces. This makes it very simple to access from .NET languages like C#. Moreover, the documentation is easy to read and comprehensive. Seriously, I can’t say enough about the quality of the TrackIR product and the great experience I’ve had with the folks at NaturalPoint. The product performed beautifully, the API was excellent, and getting support couldn’t have been easier. They have a number of other products, including a line of mocap cameras (I’m looking at the OptiTrack FLEX:V100 for a Multi-Touch surface I’m working on — more about that later).
My first step in using the API was to recreate a basic version of a sample application included with the TrackIR. The screen shot below shows some basic functionality, including a slider for adjusting the light filter threshold (useful for operating in noisy IR environments), the image captured by the camera, and output for the yaw/pitch/roll and X/Y/Z values associated with the three-point head clip.
There’s also a button and combo box for turning the camera’s LEDs on/off. That’s a pretty cool feature of the camera/API that I needed for an application where I didn’t want the built-in IR LEDs to illuminate. I created an enumeration called TrackIRComponent.LED that maps the 4 types of LEDs on the camera (IR illumination, Green, Red, and Blue — these last three are for status and the like).
This simple app serves as a testbed for the C# TrackIRComponent I wrote. The component is a wrapper to the underlying COM API that exposes what I found to be the most useful data. Specifically, the class looks like this:
The Start() and Stop() methods turn the camera on and off and the DrawFrame() method puts a camera frame to a window handle (which you can get from a Form or Panel instance through the Handle property). The Update() method should be called every frame to get the latest data. The component updates the HeadVector property with each Update() call also — and it’s this HeadVector that provides the X/Y/Z and Yaw/Pitch/Roll data. A sample update loop (typically called in a thread) looks something like:
There’s almost nothing special you have to do to use this component in XNA — and it’s easier than using it with Windows Forms because the TrackIR is meant to be polled, and XNA provides the Update() loop for exactly such a purpose. Assuming you have some actor instance that has a TrackIRComponent, and you want to rotate the actor with your head, you could do something like:
Note, that the GameSettings.TrackIRGain property is used to modulate the head tracking data (in my case I set this to 0.02 so the space ship I’m steering doesn’t turn like crazy when I move my head a little).
Anyhow, I think this post is becoming awfully long, so I’m going to leave a more detailed discussion, of the simple XNA game that I created to make use of the TrackIR and the Wiimote, to the next post. But before I do — here’s the source code 
to the TrackIRComponent and the testbed application. If you have any questions or comments, just shout.
Paul
Comments(0) 
