Full Motion .org

Preliminary Analysis: Project Natal (Part 2)

noreply@blogger.com (Maginomicon) — Sun, 14 Feb 2010 11:02:00 +0000

This is part two of an ongoing preliminary analysis of Project Natal (preliminary in that the Natal device hasn't been released to the general public yet). If you haven't read part one yet, it's best that you do so now. Keep in mind that everything in this preliminary analysis consists of educated guesswork based only on what the public has been told.

This article answers some questions raised by readers about part one, discusses how the Natal device works, some issues that arise because of how the Natal device works, and the current ideal controller Project Natal would need to resolve those issues.

Clarifications

Readers of part one have raised concerns about how controlling the game camera by tilting your head might be too sensitive or awkward to be practical. I obviously did not describe that facet of the controls well enough. The following should answer any concerns you may have.

Microsoft is listing facial recognition as one of the features of the Natal device. Facial recognition requires a high enough scanning resolution to distinguish between human faces, which in general are all extremely similar. Additionally, since they appear to be planning for facial recognition to replace the traditional profile password system, it is obvious that a lot of work is going into making this system as fluid and accurate as possible. It would make sense therefore that Natal will be capable of tracking the player's face in 3D in real time. The Natal device can use this ability to compare the angle of the player's face relative to the television screen.

Imagine someone behind you starts talking to you. You turn around, but you don't stop turning once the person's in your peripheral vision. No, you keep turning until they are in the center of your vision. It's not just being polite, it's a natural reflex. This same reflex applies to acquiring targets in real combat. When you notice an enemy, you don't use your peripheral vision to verify your target and fire, you turn your whole head until the enemy is in the center of your view and then fire. This translates easily to first-person shooters. The targeting reticle is in the center of the screen not just because it's the center of the screen, but also because it's the center of your view.

This reflex also applies in videogames. Say in a console first-person shooter an enemy appears on screen. Even if you're more than 4ft away from the screen, odds are your first reaction isn't to focus on the enemy using your peripheral vision. Your first reaction is to turn your head (perhaps only slightly) to look at that enemy. Then you push a joystick to start turning the camera. Once the enemy is in the center of your vision, you stop turning your head and let go of the joystick. There's no reason the Natal device can't use that reflex to its advantage.

Imagine the same scenario, except when you on reflex start to turn your head towards the enemy, the game's camera turns with you simultaneously. Once the enemy is in the center of the screen, your head (and the camera) stop turning. The motion is the same natural reflex you used before, the only difference is the enemy is in the center of your vision faster because the camera turned with you. If the system's too sensitive, you merely change the size and shape of the face-tracking "dead zone".

(NOTE: This does not have anything to do with aiming a weapon. Your hands should control that.)

How the Natal Device Works

Popular Science published an article on how Natal calculates your body's position in 3D space. It can be found HERE.

Step 1: As you stand in front of the camera, it judges the distance to different points on your body. In the image on the far left, the dots show what it sees, a so-called "point cloud" representing a 3-D surface; a skeleton drawn there is simply a rudimentary guess. (The image on the top shows the image perceived by the color camera, which can be used like a webcam.)

Step 2: Then the brain guesses which parts of your body are which. It does this based on all of its experience with body poses—the experience described above. Depending on how similar your pose is to things it's seen before, Natal can be more or less confident of its guesses. In the color-coded person above [bottom center], the darkness, lightness, and size of different squares represent how certain Natal is that it knows what body-part that area belongs to. (For example, the three large red squares indicate that it’s highly probable that those parts are “left shoulder,” “left elbow” and “left knee"; as the pixels become smaller and muddier in color, such as the grayish pixels around the hands, that’s an indication that Natal is hedging its bets and isn’t very sure of its identity.)

Step 3: Then, based on the probabilities assigned to different areas, Natal comes up with all possible skeletons that could fit with those body parts. (This step isn't shown in the image above, but it looks similar to the stick-figure drawn on the left, except there are dozens of possible skeletons overlaid on each other.) It ultimately settles on the most probable one. Its reasoning here is partly based on its experience, and partly on more formal kinematics models that programmers added in.

Step 4: Once Natal has determined it has enough certainty about enough body parts to pick the most probable skeletal structure, it outputs that shape to a simplified 3D avatar [image at right]. That’s the final skeleton that will be skinned with clothes, hair, and other features and shown in the game.

Step 5: Then it does this all over again — 30 times a second! As you move, the brain generates all possible skeletal structures at each frame, eventually deciding on, and outputting, the one that is most probable. This thought process takes just a few milliseconds, so there's plenty of time for the Xbox to take the info and use it to control the game.

-- Jill Duffy, PopSci.com, 2010-01-07

Two issues immediately jump to mind.

1. The 3D-box primitives for the hands are probably as precise as we're going to get for Natal. That means no hand gestures being recognized by the game as was previously rumored (keep in mind there's no reason the flat RGB images of your hands couldn't be transmitted live or analyzed by image recognition software).

2. Natal has no idea what orientation your palm is relative to your arm, it only perceives your hand as a cube attached to the arm block. It does not seem to care what rotation your hand is relative to your arm. This means we probably won't be fighting with swords, pistols, or any other one-handed precision weapons (Two-handed weapons can still be used because of the "invisible line" that can be drawn between the player's fists by the depth sensor).

Natal Controller Update

As I advised in part one of this preliminary analysis, Project Natal needs a controller to solve some apparently inherent design problems. The following is an update to the "ideal Natal controller":
Given the new information about how Natal works, it should be a...

one handed controller
with a tilt-sensor,
a joystick,
rumble, and
two buttons (at least).

Why one-handed?

Project Natal's tagline is "No controller required". We should not interpret this to mean "No controller required ever." There are certain things you just can't do without a controller. That said, if we're going to use a controller, it needs to be one-handed and lightweight.

There's a rumor that Microsoft expects developers to create games that use both the Natal device and the standard Xbox 360 gamepad. Nintendo learned the hard way that you do not tell people to use a motion control system without physically tying any controllers involved to the player. What's worse here is that a fair percentage of Xbox360 owners are playing their games on expensive HDTVs. Don't believe it's a problem? Check out WiiHaveAProblem.com The Xbox360 gamepad doesn't even have a loop for a wrist strap, so the notion that we're going to be swinging around a heavy and bulky Xbox360 controller is ridiculous.

Why a tilt-sensor?

As I mentioned previously the system only perceives your hand as a fixed cube attached to the arm block. The human wrist however can rotate in any direction. A one-handed controller with a basic tilt-sensor (no more complicated than the one in the PS3 controller) would tell the system how your wrist is rotated relative to your arm.

LOOKING BACK: The Wii game "Zelda: Twilight Princess" included in its settings screens a process for enhancing pointer accuracy by telling the system the physical size and location of your TV screen relative to the Wii sensor bar. This same process can be used by the Natal device (which already knows where your body is in 3D space) to calculate where your TV is relative to your body.

How exactly would it use this information? Knowing the orientation of your wrist relative to your arm would allow the system to know for example where you're pointing a sword, or what angle you're pointing a pistol relative to your arm. Knowing both where the pistol is (in your hand) and what direction it's pointing (20° down from your arm) will allow the system to draw an imaginary line between your hand and where on-screen the pistol is pointing.

Why a joystick?

I've already described in great detail in part one of this preliminary analysis why Natal's controller needs a joystick. Essentially, the Natal device on its own has no way to allow the the player move or strafe controls without using awkward foot position or hand position systems. For some genres they work (such as racing games, which only care about going forward or backward) but for many they won't. It's that simple.

Why rumble?

Over a year ago, I wrote about a "rumble-based haptic interface" on my design blog:

In an ordinary videogame, everything you can interact with is either assigned to a hotkey or visible on-screen.

In Full Motion, that isn't always be the case because not everything you can interact with is always on-screen.

So, why not have the controller rumble when you "mouse over" an object not visible on screen? It could even rumble a bit more when you "select" the object. Both of these would serve as a sort of tactile "screen" for object selection.

-- me, FullMotion.info, 2009-02-11

LOOKING BACK: When I originally posted what's quoted above, it seemed as if the Wii MotionPlus would feature one-to-one motion tracking (IIRC, Nintendo even stated such at E308). The Wii MotionPlus is out now, and no games have been released (not even by Nintendo) at time of writing that indicate that the Wii MotionPlus is capable of even temporary one-to-one motion tracking. The only time we've ever seen proof that the device can conceivably pull it off was in a video from July 2008 by AiLive, who developed the device.

Any motion control system featuring one-to-one motion tracking can take advantage of a rumble-based "sense of touch", including Project Natal.

Why two buttons?

If the Natal device can't distinguish fingers, we'll need some way to interact with the game world with more dexterity than that of a teddy bear.

Using Natal, Natural User Interfaces can be designed based on the most basic tactile functions we learn as infants. First we learned the sense of touch, then we learned how to grab things, then eventually we learned how to interact with objects we grabbed.

We can bind these elementary functions to a game controller.

"touch" = rumble
"grab" = button 1
"interact" = button 2

Just like the above elementary tactile functions cover all actions you could do in real life, game controller bindings of them can cover all actions you could do in a game.

LOOKING BACK: An example of how a game might use this "touch-grab-interact" system from back when I thought the Wii would be capable of Full Motion can be found HERE.

Preliminary Analysis: Project Natal (Part 1)

noreply@blogger.com (Maginomicon) — Mon, 21 Dec 2009 14:50:00 +0000

From what I've seen, there's an inherent design flaw with Project Natal.

Ever played a hover-sim, such as the Descent franchise? A hover-sim gives you complete control over your position and orientation in 3D space. That is, it gives you all six degrees of freedom (DOF) for camera control.

Move (forward/back)
Strafe
Slide (up/down)
Yaw
Pitch
Roll

In games that use a first-person camera, such as a modern FPS, you absolutely need these:

Move
Strafe
Slide (jump/crouch)
Yaw
Pitch
~~Roll~~

Project Natal, without a controller, only gives you these inherently:

~~Move~~
~~Strafe~~
Slide (move head up/down)
Yaw (turn head left/right)
Pitch (look up/down)
Roll (tilt left/right)

Because we have limited space in our gaming room, the camera can't follow our lateral (move and strafe) movements as much as would be necessary for a modern FPS.

If you haven't already seen them, watch the Project Natal Promotional Trailer and Basic Gameplay Demos. You'll notice that for each of the "core" game examples, only a fixed linear camera (either still or restrained, relative to the character) is used.

Promotional Trailer
- Martial Arts Game - restrained camera
- Racing Game - still camera
- Godzilla Game - still camera
- Soccer Game - still camera
- Skateboarding Game - still camera
Basic Gameplay Demos
- Ricochet (Dodgeball Game) - still camera

It seems the people that designed Natal forgot that our TVs can't move with us. It sounds too obvious to miss, but think about it. In the godzilla game, every time the player had to move the character forward, he had to take a physical step forward. He started the scene at the couch and ended up in the middle of the room. Starting to see the problem?

Example Controller-Free Solutions

Without a controller, you only have slide, yaw, pitch, and roll control through the Natal device. For games where either camera control is irrelevant, it doesn't need "strafe", or it doesn't need both "move forward" and "move backward", Project Natal works fine as-is.

Unless we want all other games to be on-rails, we need analog move and strafe controls. Since the space in our gaming room is limited, our options are also limited. We can either have:

Foot-based controls, where the locations of our feet on the floor (relative to some central position, akin to a DDR pad) control move and strafe, or...
Hand-based controls, where the locations of our hands in the air (relative to the player's body) control move and strafe

Each of these options would essentially prohibit us from using that part of our body for game-specific controls.

Example Controller-Based Solution

Let's see... We need some kind of one-hand controller with a joystick that'll handle the move and strafe functions. It may also need a button (or an analog trigger) or two.

Sound familiar?

I know. You're thinking, "What? The Wii nunchuck? That's a Nintendo controller! How's that going to help?" As I see it, Nintendo got the nunchuck controller right, but for the wrong reasons. Those familiar with the "Grab" haptic interface I designed know it uses force-feedback for "touch" (rumble), a button for "grab" (the "B" button), and a button for "interact" (the "A" button). For Natal, a wireless nunchuck equivalent could do all this. However, it wouldn't need motion sensors, as Natal already knows where your hand (which is holding the nunchuck) is located in 3D space.

Naturally, a second nunchuck-like controller would be needed for the player's other hand, but mostly for the buttons. The second joystick would be free for game-specific functions (like how the D-pad is used in Wii games that use both the Wii remote and nunchuck).

There's now a part two to this ongoing preliminary analysis. It can be found HERE.

First press shots of Sony's PS Wand

noreply@blogger.com (Maginomicon) — Thu, 24 Sep 2009 14:12:00 +0000

Sony has finally released a picture of the final (or near-final) design on their motion controller.

[image credit]

Unfortunately, it still appears to be lacking a joystick or D-Pad. The analog trigger is nice, but without a joystick or D-Pad it will (like Natal) be unable to give the player complete control of the camera. Yaw, Pitch, and Roll can be done to a limited extent with face tracking (your eyes are in your head, not your hands), but without some kind of instant-access device the player has no fine control of character X/Y movement.

The only other way around this problem I can think of at the moment without changing the technology to include a joystick would be to map the spatial locations of either the hands or head to those functions. (ex. It would be like using an invisible DDR pad in place of a D-Pad... which to me sounds rather awkward and unintuitive)

PS Wand Science and Origins Explained

noreply@blogger.com (Maginomicon) — Wed, 09 Sep 2009 16:35:00 +0000

Sony has released more detailed videos explaining the science and origins of the PS Wands.

Part 1:

Part 2:

I just hope the final version has a joystick on there. (We've yet to see a close-up of the device) You can't have complete camera control otherwise. Natal has the same problem. Looking around (camera yaw, pitch, roll) can be handled well enough through face tracking, but character lateral movement (camera X, Y, and to a certain extent Z) is simply impossible without some kind of control being instantly accessible to the player's hands (thumbs).

07-31 Presentation

noreply@blogger.com (Maginomicon) — Sat, 01 Aug 2009 02:55:00 +0000

Today I gave a tech-oriented presentation on Wii Full Motion.

It can be found online HERE.

A lot of it is stuff you've seen before, but some of it is new.

Sony's Wand Prototype

noreply@blogger.com (Maginomicon) — Thu, 04 Jun 2009 16:04:00 +0000

Sony also announced its own one-to-one motion tracking system which works in conjunction with the PlayStation Eye.

Although no official name was given for the system, its controllers consist of a short stick with a ball on the end, so for lack of a better word we're calling it a PlayStation Wand.

Microsoft Announces Project Natal

noreply@blogger.com (Maginomicon) — Mon, 01 Jun 2009 22:26:00 +0000

Microsoft has announced "Project Natal".

Promotional Trailer:

Basic Gameplay Demos:

Milo Demo:

04-29 Presentation

noreply@blogger.com (Maginomicon) — Wed, 29 Apr 2009 23:24:00 +0000

Today I gave another, more refined Full Motion presentation than the one from earlier this month.

It can be found online HERE.

Linking Cap

noreply@blogger.com (Maginomicon) — Sat, 11 Apr 2009 16:26:00 +0000

A "Linking Cap" -- as I call it -- is a 3IRLED cap that has one-to-one motion tracking built in.

What a Linking Cap does is link together each device's relative location.

If we had such a cap, we would eliminate the need for a stationary head tracker Wii remote.

A "one armed" or "onemote" Full Motion system is where one hand holds a WiiMot+ (a.k.a. "Wii remote augmented with a Wii MotionPlus") and the other hand holds an ordinary nunchuck accessory.

In a "one armed" Full Motion system, this would mean you only need one Wii remote and one Wii MotionPlus (in addition to the Linking Cap).

Your in-hand WiiMot+ would acquire its relative data as usual and link with the Linking Cap by simply pointing its IR camera at the cap's three IRLEDs. Since the Linking Cap has built-in one-to-one motion tracking, it doesn't need the IR head tracker in order to tell the system its location. The only thing that matters is where the TV and floor are relative to the player. After linking, the system knows the relative locations of "TV to WiiMot+", "Floor to WiiMot+", and "WiiMot+ to Linking Cap". That's all the system needs for a "one armed" Full Motion system.

For a "two armed" or "twomote" Full Motion System, the only difference is two in-hand WiiMot+ have to link to the Linking Cap.

04-10 Presentation and Facebook Group

noreply@blogger.com (Maginomicon) — Fri, 10 Apr 2009 18:13:00 +0000

Earlier today I gave a presentation about Full Motion.

It is available online HERE.

A Facebook Group about the Full Motion Project has been set up. It can be found HERE.

Rubber Wedge Wii Remote Stand

noreply@blogger.com (Maginomicon) — Sat, 14 Mar 2009 02:03:00 +0000

I had an epiphany last night while studying for a midterm.

You know the papercraft Wii remote stand I made, right? It looked like a sawed-off tin can and was held together with tape?

I realized a way to make a Wii remote stand that's slightly less cheap, has no moving parts, is ridiculously simple to set up, and will never break.

NOTE: These conceptual drawings are not drawn to scale. The four heights shown (see Fig. 2) are simply examples, and should not be misinterpreted as heights commercial versions would use.

You simply prop this wedge up under the front of the Wii remote.

This Wii remote stand has four distinct heights at which the Wii remote can be angled. (See: Fig. 1)

The triangle's altitude (height 'A')
The triangle's legs (heights 'B' & 'C')
The prism's depth (height 'D')

The stand is coated with a high-friction material such as rubber and is heavy enough not to slide around on smooth surfaces.

You lay the Wii remote on the stand upside-down because:

Unlike its underside, the top of the Wii remote is flat, allowing it to sit on the wedge easier.
The Wii Remote (even without a Wii MotionPlus accessory) always knows what direction is down.

We recommend placing the Wii remote so that the edge of the stand is between the D-Pad and 'A' button to keep it from sliding off. Additionally, if you put a sleeve on the Wii remote (we recommend Nintendo's official Wii remote sleeve) it should further reduce the chances of it sliding off.

To coarse-adjust the angle, use a different edge of the stand.

To fine-adjust the angle, move the back end of the Wii remote back and forth as needed.

Mapping Yourself and Your Gaming Room

noreply@blogger.com (Maginomicon) — Thu, 27 Nov 2008 09:47:00 +0000

A while back, I discovered that a method for increasing pointer accuracy used in the setting screens for the Wii version of Zelda: Twilight Princess could be used to tell the game where your screen is in 3D space relative to the sensor bar. Then, if the system knew where the sensor bar was in 3D space relative to the Wii remote, it would know where the screen is relative to the Wii remote.

However, I've just realized there's a much easier way to tell the system where your TV is in 3D space.

All you need to do is touch at least three corners of the TV with the tip of a Wii MotionPlus-equipped Wii remote. (WiiMot+) The accelerometer in the WiiMot+ could even be used to automatically detect when it touches an object.

This method can be used to map out your entire gaming room virtually. For example, once you map out the location of your screen, you could tell the system how low the floor is relative to the screen by simply placing the WiiMot+ on the ground. Any stationary obstacles in your gaming room (such as a couch or standing speakers) could be mapped out in the same way.

NOTE: Once the system knows where the screen is in 3D space relative to the WiiMot+, the system becomes capable of knowing when the WiiMot+ is pointing at the screen and where, without using the sensor bar. (although the sensor bar would probably still be useful for verifying the pointer's location)

All this virtual mapping may be neat to think about, but this information isn't particularly useful to the system until you also tell it where you are in 3D space. That is, until you give the system a "Body Frame Approximation". Once it knows this, it's possible for it to render your entire body virtually and track it in real time. Useful proportions might include such measurements as: how high your waist is above ground, the height of your knees above ground, the width of your shoulders, and the distance between your shoulder and your elbow. All of these measurements can be mapped in the same way the TV screen was.

There's a problem though. Even if the system knows where the hand holding a single WiiMot+ is in 3D space relative to the screen and/or the floor, it can't make any guesses where the rest of your body is. You could potentially move your body in any direction while keeping the in-hand WiiMot+ stationary. The system would need to know the locations of all relevant "independent appendages" (e.g. both hands) and one "fixed appendage." (e.g. your head) Luckily for us, "3IRLED head tracking" and "twomote" fulfill these requirements perfectly.

NOTE: In order for the tracking to be accurate, each WiiMot+ must be calibrated to an initial position like I described in my last post before any other objects or body proportions are mapped. This is only necessary if head tracking or twomote are used. With head tracking, the system needs to know the head tacker's location relative to the TV. Likewise, with "twomote" the second WiiMot+ needs to know where it is relative to the mapping done with the first WiiMot+. Technically, one way to get around this problem could be to map the second WiiMot+ as an object using the first WiiMot+.

NOTE: Technically, you could move your elbows and legs around to a limited degree while keeping the two in-hand Wii MotionPlus-equipped Wii remotes and your head stationary, but not enough to affect gameplay.

How's that for an immersive experience?

Tracking Hands

noreply@blogger.com (Maginomicon) — Tue, 18 Nov 2008 00:38:00 +0000

Currently, the Wii MotionPlus (WiiMot+) is capable of tracking its own movement in 3D space one-to-one. However, by conventional means it can only track this movement relative to the location it was at when the WiiMot+ was powered on. That is, if you held the WiiMot+ three feet above the ground, turned it on, made a motion, turned it off, turned it on again a foot higher and made the same motion, the WiiMot+ would not be able to tell the difference.

With 6DOF head tracking though, it is possible to tell the WiiMot+ exactly where it is located in 3D space regardless of when and where you turn the device on.

As you're aware from my previous posts, a 3IRLED 6DOF head tracker is capable of determining the exact position and orientation of the player's head in 3D space. Since the system knows the location of your head relative to the head tracker, it's possible to tell the WiiMot+ in-hand where its position in 3D space is by virtue of the fact that the in-hand Wii remote's IR camera has the same capabilities as the head tracker.

It's all a matter of relative perspective. As we know, the reason IR head tracking works at all is because conventionally the Wii remote detects the stationary sensor bar's IRLEDs as "moving" according to its own perspective and discerns where it's pointing from that.

The above image shows a stationary head tracking Wii remote and an in-hand Wii remote equipped with the WiiMot+. Both are pointing at a 3IRLED hat. The two boxes shown below the Wii remotes represent the approximate positions of the hat's three IRLEDs from the perspective of the corresponding camera. Because the system knows where your head is in 3D space courtesy of the head tracking Wii remote, any in-hand Wii remote can see the three IRLEDs on your hat and make the same calculations as the stationary head tracker to determine the location of your head in 3D space relative to that in-hand Wii remote. Since that in-hand Wii remote now knows your head's location relative to itself, it can determine it's own location relative to your head.

NOTE: The in-hand Wii remote only needs to be pointed at the IR hat during calibration.

Presto! The in-hand MotionPlus-equipped Wii remote now knows an initial position to start measuring one-to-one motion from. Naturally, this method can also be applied to "twomote".

Nintendo's Mixed Signals

noreply@blogger.com (Maginomicon) — Wed, 08 Oct 2008 22:40:00 +0000

The two devices I described in the last post are cheap to produce and sell, so why hasn't Nintendo announced head tracking yet?

Emailing them for comment on the discovery of IRLED head tracking prompted the expected self-promoting response:

Greetings,

At this time, there has not been [an] announcement concerning using such a feature. As the worldwide leader and innovator in the creation of interactive entertainment, and because of our history of pushing the limits of gameplay and design, you can always count on Nintendo offering the most imaginative products.

In the meantime, keep checking our website's "What’s New" section (http://www.nintendo.com/whatsnew) for the latest information and announcements.

Sincerely,

Dale Thompson
Nintendo of America Inc.

As we've seen with the WiiMot+, Nintendo's proven itself to be overly-secretive with its hardware development. It may be that Nintendo is well aware of IRLED head tracking and is working on such a feature, but Johnny Lee's comments on the removal of the Boom Blox head tracking easter egg imply the opposite:

"I have been talking to some Wii game developers and they've said that if [a game] requires too much movement on the player's part, Nintendo asks them to pull it."

I understand to a degree why Nintendo may be limiting player motion, but head tracking is -- as Darwin would say -- the next logical step after coming down from the trees immersion-wise. Gaming culture should at the very least be ready to get up off the couch and play, (at least that's what Nintendo would have us believe with games like Wii Fit) so what's with the mixed signals?

As I see it, this is where gaming is going and if Nintendo doesn't get on the ball with head tracking then Microsoft and Sony will. (Technically, Sony already has, and already has the "stand" peripheral covered) Personally, I'd prefer Nintendo because the Wii remote although costing about the same as a PS3 Eye has more functionality besides its IR camera, meaning that you could already own an extra Wii remote and just use that or buy one for the head tracking and as an added bonus use it as a controller.

Wii MotionPlus + 6DOF Head Tracking

noreply@blogger.com (Maginomicon) — Tue, 19 Aug 2008 12:40:00 +0000

Let's recap.

Simple Head Tracking
As-is, Johnny Lee's simple head tracker can allow the player to lean, crouch, and jump without using any buttons or joysticks. In his setup, you still cannot look (up/down) or turn. (left/right) Even if you bind the handheld Wiimote to a "look margin" similar to the one used in Metroid Prime 3, there isn't enough practical function in it to convince developers to integrate head tracking into their games. With just leaning, crouching, and jumping, it's just not worth requiring players to buy additional hardware, even if it's just another Wiimote and a pair of infrared LED glasses.

6DOF Head Tracking
With Manuel Unternaehrer's 6DOF head tracker, not only can you do everything Johnny Lee's original program could do, but you can also look and turn. In effect, the head tracker becomes its own pointer, and the "look margin" from Metroid Prime 3 that was previously bound to the handheld Wiimote can be reassigned to the head tracker. Because the handheld Wiimote is no longer bound to a "look margin", it is free to be pointed anywhere on screen. Apparently however, this still doesn't offer enough immersion to warrant much attention.

Wii MotionPlus
Then the Wii MotionPlus was announced. Combine the hands-free pointer of the 6DOF head tracker with the one-to-one motion tracking of the Wii MotionPlus, and you get an unbound Wiimote that you can point not just anywhere on-screen, but anywhere off-screen as well. Imagine the level of immersion you could get out of that system. For example, if you know there's an enemy behind you, you can point the Wiimote over your shoulder and fire. Or if you know there's an enemy right outside your peripheral vision you could fire in that direction instead of turning to look. (without head tracking, you wouldn't be able to do this because the wiimote would have to be pointing at the screen at all times in order to look and turn)

There are a couple of distinct possibilities that are now open to us.

"Twomote"
If we only need bindings for moving and strafing now, (because looking and turning is assigned to the head tracking "pointer") we could technically bind those functions to a Wiimote's D-pad. If we do that, then we don't need the nunchuck anymore, and can finally start using two Wiimotes, one in each hand. (which I call "Twomote" Style)

Balance Board Centering
Until the price of the Wii Balance Board goes down, it is my opinion that developers should have this hardware be completely optional to full motion gameplay. Once the price drops suffeciently though, it is possible that developers could expect the player to use the Balance Board not just for its inherent features, but also as a "default" spot on the floor for the player to return to for centering. (really though, if the inherent features aren't used, players would be better off using something else to mark their "default" spot)

Head Tracking Kit
The cost to manufacture, market, and buy the necessary additional hardware required for head tracking -- namely the IRLED headgear and wiimote stand -- can be minimized. As you can see at the end of Manuel Unternaehrer's 6DOF head tracking video, the 3 IRLEDs are arranged around the rim of an ordinary baseball cap. Those LEDs could potentially be attached to clips, meaning Nintendo wouldn't have to sell goggles or glasses, they could just sell the 3 IRLED clip set and ask the player to attach them to a baseball cap they already own. (Nintendo could also sell their own caps separately or bundled without having to modify the cap in the slightest)

As for the wiimote stand, I personally have designed and successfully tested a wiimote stand made entirely out of posterboard which features simple construction, four pre-defined angles, and supports the weight of the wiimote beautifully.

Obviously, a commercial version would probably use laminated cardstock or something stronger but just as cheap, and use tabs instead of tape to hold it together.

Nintendo has no excuse now.

Wii MotionPlus and AiLive's LiveMove 2

noreply@blogger.com (Maginomicon) — Thu, 31 Jul 2008 03:23:00 +0000

While the promise of 1:1 Wii Remote control is a tantalising one in theory, so far we've seen it working on...Wii Sports 2. Uh, thanks. With the company that's supposed to bring smiles doing nothing of the sort, it's instead left to the MotionPlus' creator, boring tech company AiLive, to show us a video that, despite being shot in 10 minutes with one of the techs and a shitty handycam, really gets us all excited about the add-ons potential. You'll want to skip to around the 1:50 mark. -- Luke Plunkett, Kotaku

As you can see, the WiiMot+is fully capable of doing everything it claimed it could do.

Headtracking (6DOF) with Wiimote

noreply@blogger.com (Maginomicon) — Sat, 12 Jul 2008 23:38:00 +0000

A modification of Johnny Lee's method has been discovered by Manuel Unternaehrer which solves the problems I mentioned previously, creating a true 6DOF head tracker.

It's official thread on WiimoteProject.com is here.

(I realize that the person is sitting down in the video but he just as easily could of been standing up in front of a TV.)

How does it work?

The Wii remote depends on the positions of two IRLEDs relative to eachother in order to perceive depth by measuring the apparent distance it sees between them. A third IRLED, positioned to make a triangle, will move differently when looking and turning than it would if it were moving in distance from the screen. For example, if you look down, the IRLED in the middle would move closer to and pass through the "line" between the other two IRLEDs. Likewise, If you look left or right, the middle IRLED would appear to "lean" closer to one of the other two LEDs.

You can demonstrate this to yourself like so: Hold up three fingers, with the middle one closer to you than the other two. Then bend your wrist towards you, (to simulate "looking down") away from you, (to simulate "looking up") and twist it left and right. (to simulate "turning left and right") Your fingertips positions as they would be seen in 2D act exactly the same as the three IRLEDs on the cap. (the Wii remote's IR camera can only see in 2D)

What does the third IRLED give us?

With this head tracker, not only can you do everything Johnny Lee's original program could do, but you can also look and turn. In effect, the head tracker becomes its own pointer, and the "look margin" from Metroid Prime 3 that was previously bound to the handheld Wiimote can be reassigned to the head tracker. Because the handheld Wiimote is no longer bound to a "look margin", it is free to be pointed anywhere on screen.

Head Tracking for Desktop VR Displays using the Wii Remote

noreply@blogger.com (Maginomicon) — Sun, 23 Dec 2007 01:29:00 +0000

The first step towards full motion has been accomplished.

There are some limitations to this method.

Johnny Lee's method, which I call "simple head tracking", allows the player to lean, crouch, and even jump as buttonless inputs. However, it can not distinguish where on screen the player's head is pointing. This can be proven in a normal remote-in-hand setup by going to the sensitivity settings in the Wii Menu and changing the angle at which the Wii remote sees the sensor bar. (say, 45 degrees to the left) You'll notice that the two IRLEDs on the sensor bar (from the perspective of the Wii remote) appear to be closer together, which it interprets as the player being further away from the screen than when you were standing the same distance directly in front of the sensor bar. It can't tell the difference. For the same reason, simple head tracking also can not distinguish looking up from looking down. The Wii remote can only see the X-Y coordinates of the two IRLEDs, not which direction they're pointing.