Wiimote Gesture Recognition

The Nintendo Wiimote is a remarkable, inexpensive device for exploring 3D user interaction. With 3 accelerometers, the Wiimote allows motion control and communicates wirelessly using the BlueTooth communication protocol. Open-source libraries (such as Brian Peek’s Wiimote.NET library) allow access to the Wiimote from a PC.

Many of the games on the Wii, use the Wiimote in a very simplistic way — mainly looking for thresholds on an axis to indicate an action. You can see this behavior if, for example, you swing the Wiimote above your head and maybe the golf club still swings as if you’d moved the controller correctly. Accurate gesture control — precise recognition of hand/arm movement — has been largely missing from most Wii games.

A colleague and I have devised a set of 29 features that can be used to classify the stream of Wiimote XYZ accelerometer values into a recognizable motion. Using a machine learning algorithm, we can train an arbitrary set of gestures which can then be mapped to actions.

In the video below, I present a graphical user interface for training and classifying gestures, providing output through sound, animation, and even a tactile belt. Importantly, the underlying recognition system is an API — allowing pre-trained data to be used by any application. I’ve used the system now in a couple of XNA video games (in fact the animation in the demo here is presented through XNA), in a Windows app, and even to control a robot (future post).

2 Comments so far

  1. […] Paul Varcholik has been experimenting with some very interesting technologies such as training input gestures for the wiimote.  It would be cool to see this extrapolated to a more general application.  This could be used to […]

  2. Darshana Perera on September 5th, 2011

    Dear Sir,

    Asking help for a Virtual Reality game development based on XNA that uses C# and Wii remote

    About me
    I am a student from Sri Lanka following a Computer Systems and networking degree

    About project
    I am currently developing a virtual reality cricket batting training software/equipment. There I am using XNA (3.1) as my game environment and using C# as language.
    By this equipment I am going to give the user an actual cricket bat with attached Wii remote (with motion plus) in it. A virtual bat (designed using 3D max) in the XNA environment should move according to the actual bat. Then in the XNA environment I am generating a virtual ball and when it comes to the user, user can see it using a head-up display or any other display and user will play the shot using his actual bat with Wii remote according to the movement of the actual bat virtual bat in the XNA will move and hit the virtual ball.

    What I have done so far?
    Up to now I have created the virtual environment (with basic graphics) and kind of virtual bat. Also few types of virtual ball deliveries. By using the keyboard I can control the virtual bat (3D bat in the XNA environment) and hit the generated virtual ball and if the bat hit the ball it will successfully projectile according to the shot.

    My main questions
    Is it possible to use Wii remote to control (rotate, swing, tilt) a 3D object in XNA using C# language.
    Is it possible to use Wii remote use as a head tracking unit in XNA environment that uses C# language, in a first person shooter game like game.

    My scenarios
    I have try out Mr. Johnny Lee’s program that uses Wii remote for head tracking, it work about 80% for me. I have some problems with it. When I use it in a lighted environment with florescent light it will not work properly, I thought that it because florescent make some IR rays and make Wii remote to work not properly. Anyway up to 80% is ok for me.
    I have another question to ask from you I try to integrate Mr. Johnny Lee’s coding with the XNA C# code in my project but Mr. Johnny Lee’s code is very long and large so it makes lot of difficulties when integrate it. So I would like to ask that is it possible to use Mr. Johnny Lee’s code segment in XNA environment and how can I do it, can you please help me where I can find kind of tutorials for that or some sample codes.
    I am trying to use the WiimoteLib (Version 1.8) that compatible to Wii motion plus, by using that I can now able to see the coordinates and other details of the Wii remote by using the windows form application. Means successfully run the given WiimoteLib.
    But I faced lot difficulties when I try to use the WiimoteLib in the my coding environment, actually I cannot connect the Wii remote to the my C# coding successfully.
    And next I want to map the coordinates that I see in the WiimoteLib to the virtual bat in the XNA environment and move the virtual bat in the XNA environment when I move the Wii remote. (Actually developer of this library Mr. Brian peek mailed me that when I asked about this it kind of not possible thing by using his library)
    So I would like to ask what the other ways that I can do it.

    From where, I can get some sort of sample codes, learning materials, tutorials related to control a 3D object in XNA environment (C#) using Wii remote.

    Thank you for your consideration and valuable time.

    It’s a great pleasure if you could send me a positive answer.

    Please help me on this sir I am now in big trouble, because I will miss my project without this part, so please help me sir.

    Yours sincerely,
    Darshana Perera.

Leave a reply