Wiimote Interactions for Freshmen Engineering Education

Abstract --The Nintendo Wii revolutionized the gaming industry, but its Wii Remote is the true star. This innovative controller has gained a large fan base dedicated to harnessing the power of the Bluetooth device in developing a wide range of PC-based applications. An application of particular interest is the ability to communicate with the Wiimote via MATLAB. The freshman engineering track at the University of Notre Dame contains an eight-week MATLAB programming module that can be a challenge for new programmers. The high level of interaction provided by the Wiimote, along with the novelty of the device, may be able to facilitate the learning process.

Introduction

The University of Notre Dame offers a year-long course sequence that is divided into four half-semester sections designed to introduce incoming students to a variety of engineering disciplines. One of these sections, EG111, is a fast-paced course dedicated to teaching the fundamentals of programming with MATLAB. Through lectures and laboratory projects, the students are taught the fundamentals of loops, conditionals, and input/output. The class culminates with a final project of developing an interactive game similar to Lunar Lander, a popular arcade game from Atari. The course, as a whole, is very unconventional and ever-evolving, and this project continues the trend by incorporating use of the Wiimote into the existing curriculum to enhance the students' experience in a way that is both fun and educational.

The Nintendo Wii Remote (Wiimote) was the first of its kind, containing three accelerometers, an infrared sensor, twelve buttons, and built-in speaker, vibration-producing motor, and LEDS that are triggered by events in some of the Wii's games [1]. This interactive functionality provides broad appeal and has attracted gamers of all ages to the Wii gaming console.

Furthermore, because the Wiimote communicates with the console via simple Bluetooth signals [2], it became apparent that the Wiimote could be reverse engineered and made to communicate with any Bluetooth-enabled computer.

It was only a matter of time before Wiimote libraries were created for nearly all programming languages and operating systems. Notable examples include wiiuse, a C library; a C# managed library, WiimoteLib; and wiiusej, written for Java. After testing several of the existing libraries and using them to create small, simple applications using the Wiimote, it became clear that the C# library, WiimoteLib, easy to use and understand, would best suit this project's purposes [2]. WiimoteLib provided a well-documented, comprehensive foundation of functions for the Wiimote.

This paper details the following efforts to incorporate the Wiimote into introductory computer science courses:

• Development of a wrapper, WiiLAB, for the WiimoteLib and explanations of how this class is able to communicate the Wiimote data to MATLAB.
• Creation of a MATLAB class that utilizes WiiLAB to collect data from the Wiimote. The class includes all the necessary data structures to store the Wiimote's information, as well as methods for communicating with and controlling the Wiimote.
• Construction of a set of global functions that access the Wiimote class and allow individuals of all programming backgrounds and experience levels to use WiiLAB.
• A variety of demonstration applications that highlight the key features of the Wiimote class and global function set and how to use them.
• A wiki page that provides an open-source outlet to document the project.

Wiimote

In addition to the usual features you would expect to find in a video game controller, including buttons, a directional pad, and a joystick (found on the Nunchuk extension), the Wiimote contains a number of extra features that set it apart. The Wiimote also contains a three-axis accelerometer, infrared camera, LEDs, speaker, and a small motor capable of producing vibrations.

The Wiimote transmits its data over a standard Bluetooth frequency, with each byte of data corresponding to a specific part of the Wiimote. Through the efforts of many individuals and their contributions to the WiiLi [1] and WiiBrew [3] projects, the Wii remote has been largely reverse engineered, making it possible to both send data to the Wiimote and interpret most of the data received from the Wiimote. Because transmission is through a standard Bluetooth signal, communication with the Wiimote is attainable using any computer and compatible Bluetooth adapter.

Although much of the Wiimote has been successfully reverse engineered, there are some aspects that remain a mystery. The embedded speaker, for example, is not yet functional in any Wiimote library. Due to limitations in computers' rates of transmission and the unknown data format of the Wiimote, the speaker is currently unable to emit any meaningful audio [1]. A series of crackling noises and high-pitched beeps are the current limitations of sound production, so speaker functionality has not been included in this project.

The Wiimote's features allow for such tasks as motion capture and gesture recognition. These unique capabilities have facilitated the Wiimote's expansion beyond the video game industry. Significantly, the Wiimote has slowly begun to impact the academic community, as teachers of all levels are realizing the educational potential of the versatile device.

WiimoteLib

The WiimoteLib, created and maintained by Brian Peek, is a .NET managed library for using the Wiimote from .NET applications [2]. The library provides support for all of the successfully reverse engineered Wiimote features: buttons, accelerometer, infrared camera, rumble, and LEDs. Along with these standard Wiimote features, the library also supports all Wiimote extensions, including the Nunchuk, Classic Controller, Guitar, and the new Wii Fit Balance Board. The WiiLAB project communicates with the Wiimote through Peek's existing library. Most of the library is being used as written, but there are a few custom features that have been added out of necessity.

The first major change involved the WriteReport and WriteData methods, which send data to the Wiimote to control features such as rumble, LEDs, and configuration. The existing methods included a Thread.Sleep(100) command that halted execution for 100 milliseconds. These were included because it was possible for the Wiimote to enter an unresponsive state during certain combinations of reading and writing data. With the sleep commands, these methods performed significantly below desired levels. After running a series of tests on the Wii remote and experimenting with various pause lengths, the 100-millisecond pause was able to be reduced to 10 milliseconds.

The OnRead method of the library was the focus of another considerable change. Previously, an unhandled IOException occurred when the Bluetooth adapter was disconnected while the library was running. MATLAB could not handle this error, resulting in a crash and the loss of any unsaved data. To remedy this situation, a ConnectionState data type was added to the library. This structure has three states: Connected, NotConnected, and ReadError. The necessary logic to catch the IOException was also incorporated, and now the ConnectionState is set to ReadError when applicable to notify WiiLAB of the error and allow it to take the appropriate actions.

WiiLAB

The first challenge in completing this project involved finding a way to make the W!iimoteLib accessible from MATLAB. WiimoteLib, a managed .NET library, provides no direct access to MATLAB, which uses unmanaged code. However, the .NET environment supports this interaction through a COM callable wrapper (CCW). The creation of a CCW for the WiimoteLib exposed the methods of the library to all COM visible applications, allowing MATLAB to connect to the library. This wrapper took on the name WiiLAB, and in time the entire project became known by the catchy alias.

MATLAB presented another obstacle with its inability to interpret objects. The WiimoteLib includes many custom data types for the information obtained from the Wiimote; passed directly, the data would be indecipherable by MATLAB. Therefore, specific methods for accessing each of the Wiimote's data elements were incorporated in the wrapper. These methods convert the data structures from the WiimoteLib into arrays that are consistent with the structure and indices of the data being stored. The retrieval of button, acceleration, and infrared camera data, as well as control the rumble and LEDs, is supported.

The only method of maintaining a constant stream of the Wiimote's most current data is polling. The Wiimote continuously sends data to the WiimoteLib, which parses and stores the data within its various structures as it is received. WiiLAB is unable to pass this information directly to any dependent program, so the most recent data values can only be obtained by continuously polling the WiimoteLib data through WiiLAB.

MATLAB Class

Accessing the COM server through MATLAB allows a connection to be established with the WiiLAB wrapper. With this connection, it is possible to call any of the methods from WiiLAB through the instance of the server. To keep this communication organized, a MATLAB class was created for the Wiimote. For each of the Wiimote's features, the class contains properties to store the data upon retrieval from WiiLAB. Also contained in the class are methods for each of the WiiLAB functions. The Wiimote class was written so that it is not necessary for the user to have any knowledge of the background tasks, such as accessing a COM server and the required syntax. An instance of the COM server is created when the class's constructor is called and is then used in the subsequent methods to access the WiiLAB functions.

In addition to the WiiLAB functions, the class has a few custom functions that provide helpful features not originally included. The first is a set of graphs that, when called, plot the three-dimensional acceleration of the Wiimote, with a subplot for each axis. Every change in acceleration triggers a command to add the new values to the existing graphs. Another feature is an acceleration indicator. This method creates a rod and ball graphic that can be attached to any user-created graphic object. This indicator serves as a visual representation of the Wiimote's acceleration and provides a more intuitive view of the data created by the accelerometers. Finally, a method was added to collect the acceleration data from a Wiimote over a given period of time. This method samples the acceleration data at a predetermined rate for a user-specified length of time and returns an array of acceleration data for each axis. These values can then be studied and plotted.

The MATLAB Wiimote class is supplemented with a number of global functions that call the various class methods. The global functions maintain a strict call and return structure for accessing the Wiimote's data, effectively masking the syntax associated with the Wiimote class and allowing users with any level of programming experience to use the Wiimote. With the global functions, a simple function call to getWiimoteAccel(), for example, is all that is necessary to access the appropriate data from the Wii remote.

Demonstration Applications

Several demonstration programs have been created to exhibit the Wiimote's capabilities and provide example usage of the global functions. The demos were written for use in an educational setting, but their fun, interactive nature appeals to the casual user, as well.

First, the bouncing ball application is a simple demo that allows the user to lift and throw a ball that bounces realistically as it hits the bottom of the figure window. A position graph is also created to allow users to study the ball's movement.

The inverted pendulum is a classic problem involving a pendulum with its mass located above its pivot point, making it inherently unstable. The goal is to keep it balanced upright, which can be done by applying torque at the pivot point or by moving the pivot point horizontally, often by being mounted on a small, motion-capable cart. The WiiLAB Inverted Pendulum application makes use of the former balancing method, with the Wiimote's X-axis acceleration used as a representation of applied torque. The user must continually twist the Wiimote either clockwise or counter-clockwise to attempt to balance the inverted pendulum. A bit of simulated gravity adds to the realism and difficulty.

Also included is Pong, the classic video game reminiscent of a tennis match. In this version, each player controls his paddle with the Wiimote, tilting it up or down to move the paddle the corresponding direction and attempt to keep the ball in play.

A segented disk resembling a game spinner is featured in a spinning disk application. Once again utilized is the Wiimote's acceleration, perhaps its most versatile feature, being used in this case to determine the disk's rotational velocity. The user grabs the edge of the disk and swings the Wiimote the desired direction to watch the disk spin and see its acceleration values displayed.

Finally, a series of acceleration graph classes can be called to supplement any application and allow the user to track the acceleration along each of the Wiimote's three axes.

Walkthroughs for most of these demonstration programs can be found on the project's wiki. The code for each is broken down into its component parts and described in detail to provide examples and inspiration for users desiring to write their own WiiLAB program.

NetScale Wiki

The wiki page that has been diligently assembled as part of the NetScale Laboratory's TWiki is perhaps the most important part of this project. Others' ability to use, understand, and build off of the work is vital to the project's success, and the wiki page provides a channel for accomplishing these goals. The wiki contains extensive documentation of all the work that has been put into this project.

Included are tutorials detailing every step of the process, from installing the necessary programs and establishing a connection with the Wiimote to writing one's own program with WiiLAB. Documentation is provided for a stand-alone C# application that can verify both correct installation of WiiLAB and the operability of up to four Wiimotes. There is also a comprehensive list of known actions that affect Wiimote-Bluetooth pairing. All the important code that has been written, including the Wiimote class and global functions for MATLAB, are included on the wiki, along with basic usage guides and examples. A Getting Started guide, complete with skeleton code, has been included for novice programmers, while a quick function reference list makes it easy for more experienced users to get started and is also helpful to users of any skill level who need a simple reminder.

The thorough documentation on the wiki has made the project completely open-source. Others are free to either use the code in its current form or modify it to fit their needs; WiiLAB was designed to be suitable for use in a variety of applications. The easily accessible online format of the code and documentation puts WiiLAB within reach of everyone.

Related Work

Since its release in 2006, programmers with varying levels of expertise have been hard at work harnessing the power of the device for a wide range of applications.

Perhaps the most well-known are the efforts of Johnny Chung Lee. He has developed a virtual reality program that can track the location of the user's head and adjust the viewpoint appropriately. He has also written a program to track and respond to finger movement, as well as one that can turn any surface into an interactive whiteboard [4].

A pair of programmers with access to an industrial robot spent a Saturday writing a simple pattern recognition algorithm that allowed them to turn the robot into a ferocious tennis player. They attached a tennis racquet to the robot's arm, and a swing of the Wiimote was matched with a pre-programmed tennis swing which the robot replicated. Just for fun, they also decided at one point to replace the racquet with a sword [5].

Many others have used the Wii remote to program their own robots, create music, control PC media players, develop demonstrations for physics classes, and do nearly anything else one could imagine.

Conclusion

The potential of the Wiimote is only beginning to be discovered. The Wii remote is an accessible and affordable sensor array because it uses a standard Bluetooth communication system. This has allowed anyone with a computer to use the Wiimote and learn from it. The Wiimote and MATLAB have successfully been fused, and the results documented, to create a platform for users of all experience levels to easily access.

WiiLAB will premiere this fall at the University of Notre Dame. Its use will at first be limited to lecture demonstrations and optional student projects, but it will hopefully expand and become a more integral part of the programming module of the introductory engineering course as its value and efficacy are realized. The Wiimote has to potential to increase motivation to study computer science and to enhance the classroom experience. Ideally WiiLAB will inspire teachers at other institutions and pave the way for the Wiimote's educational potential.

Acknowledgments

The authors would like to thank, first and foremost, the open-source community, including contributors to the WiiLi and WiiBrew wiki pages. Their time and dedication to reverse engineering the Wiimote has provided great insights into the inner workings of the device, without which this project would not have been possible. The authors would also like to thank Brain Peek and the various people who have assisted him in developing the WiimoteLib, a managed C# library for the Wiimote, which provided a strong base for this project. Dr. Aaron Striegel, University of Notre Dame, provided invaluable guidance and support, in addition to developing the initial project concept. Finally, the project was funded in part by the National Science Foundation through REU site grant CNS-0754933.

References

[1] (2008, July). WiiLi [Online]. Available: http://www.wiili.org

[2] B. Peek. (2007, Mar. 14). Managed Library for Nintendo's Wiimote [Online]. Available: http://blogs.msdn.com/coding4fun/archive/2007/03/14/1879033.aspx

[3] (2008, July). Wiimote. WiiBrew. Retrieved July 2008, from http://wiibrew.org/wiki/Wiimote.

[4] J. C. Lee. Projects: Wii [Online]. Available: http://www.cs.cmu.edu/~johnny/projects/wii/

[5] A. Rasmussen. WiiBot [Online]. Available: http://www.usmgarage.com/usmgarage/WiiBot.html

[6] Nintendo. (2008). Wii Controllers. Nintendo. Retrieved June 2008, from http://www.nintendo.com/wii/what/controllers\#remote

• wiilab_report_draft02.pdf