Artificial Intelligence and Robotics

Retinal implant helps blind man see again

2009-11-29T02:23:00.000-08:00

This story sounds like it is taken out of a science fiction novel but news reports around the world say that Mr. Peter Lane, 51, from Stretford, has his lost sight partially restored using a new device under development by California-based Second Sight Medical Products Inc.

The story is that Lane is one of several blind people who are testing the new device known as Argus II Retinal Stimulation System and he is the one who recently found fame in the media having his lost sight partially restored to the extend that he can now read some small words and navigate the outside world with increased ease.

According to Second Sight, the device consists of a small camera and an electrode-studded array implanted on the patient's retina. This is how it all works.

The camera on the glasses captures an image and sends the information to the video processor, which converts the image to an electronic signal and sends it to the transmitter on the sunglasses. The implanted receiver wirelessly receives this data and sends the signals through a tiny cable to the electrode array, stimulating it to emit electrical pulses. The pulses induce responses in the retina that travel through the optic nerve to the brain, which perceives patterns of light and dark spots corresponding to the electrodes stimulated. Patients learn to interpret the visual patterns produced into meaningful images.

The Argus 16 and the next generation Argus II (which has 64 electrodes) are not designed to give sight to those who never had it. Mr. Lane and the others in the clinical trial suffer from Retinitis Pigmentosa which means they lost their sight in their later years. This is important because it essentially means that the brain is capable of interpreting the signals received from the eye but the disease has destroyed the retina and the flow of data has seized. Argus replaces the damaged tissue with an electronic one allowing the brain to once again receive the signals necessary to perceive the world.

This is a great breakthrough in the fight against blindness. Statistics show that more than 200,000 people in the USA and Europe suffer from Retinitis Pigmentosa and Argus II can immensely increase quality of life for these people. Some suggest that Second Sight might be able to start selling the device commercially in one year at the cost of nearly $100,000. Sure it is expensive but it is a start. Eventually, when the device is proven to work well and the cost of production decreases it will become much more affordable and available to all who need it.

[For clinical trial information go here.]
[source Times Online and the Daily Mail.]

2009-11-26T12:48:00.000-08:00

The Pentagon have been dabbling in the field of robotics for some time now, and one of their newer projects is buzzing nearer completion. We’re talking about remote-controlled beetles that can be stirred in the right direction, but allowed to fly on their own. In the future, plans are to equip these insects with tiny cameras to help search for survivors of disasters, or with sensors to sniff out potential chemical weapons.

Watch the video of a remote-controlled beetle below.

To make it clear—these beetles are not robots, but cyborgs. They are actually living creatures that have been implemented with electronic parts to influence their movements, not unlike reigns on a horse. To help them adjust, this “modification” takes place in the pupa stage when they are still developing. When they are fully grown, the beetles can be commanded by electric signals controlled by a laptop that are transmitted to their nervous systems. Researchers at the Agiltron Corporation, who are contracted to the project, plan to implement another system in the insects that will have them twitch when they detect hazardous chemicals.

Other, more vocal insets, like crickets, are also lined up for modification. They will be equipped with MEMS transceivers that will receive specific sounds made by other cyborg crickets, formulate the data and carry it on to other insects. This will create a sort of wireless network of information that can be remotely monitored.

This is, of course, a very interesting development, albeit a bit reminiscent of sci-fi horror movies. Some have questioned the ethics of the procedures, and wonder if or where a line will be drawn. If we start out with insects, would we move on to hamsters, rabbits, dogs…one day even illegal human experiments?

Nonetheless, one of the bigger problems right now is actually getting the insects to live longer. The modifications take their toll, and the beetles burn out before they can be used for something other than research.

[source NewScientist]

Space robotics: One Giant leap for Robots

2009-11-25T00:54:00.000-08:00

Robots have been employed by a number of NASA’s missions in the past, particularly those to the Moon, where they have been used to scan the surface and collect samples and various kinds of data. NASA has also been a pioneer in semi-autonomous robots exploring the surface of Mars; the twin Mars rovers Spirit and Opportunity are still operating on the surface of the Red planet several years after their initial mission was completed. The Phoenix robot successfully landed on Mars last year and completed its 3-month mission. Moreover, NASA is preparing to launch soon a new generation Mars rover aka the Mars Science Laboratory while testing the ATHLETE rover for a mission to the Moon.

But none of these very cool robots have been able to walk—or build Japanese rock gardens.

In a presentation entitled “Realization of Moon Exploration Using Advanced Robots by 2020,” Toyota shared plans about the future use of its innovative Toyota Partner Robot series. Initially, these humanoid robots were designed to help with life down here on Earth, specifically with elderly care and assistance. A result of Japan’s aging population and a lack of labor force, they are able to perform a variety of tasks, including transporting passengers and helping citizens in need. Their application, however, is set to be taken quite a big step further—to the moon.

There, the Toyota Robots will assemble space tents and relay stations, as well as deploy solar array and use a telescope. Additionally, they will work alongside a Rover to collect further data of the moon. The presentation also showed them building a traditional Japanese rock garden, which is rather cooler than planting a simple flag. To perform these tasks, they will be equipped with a number of big upgrades. As the specs in the yellow table predict, they will have:

joints that are protected from regolith
a small capacity solar battery onboard
their internal status showing on screen on chest
arms exchangeable for different tasks
ability to jump with springs in legs
keep warm during night covered in a metal cloak

That last one paints quite the endearing picture of our robot helpers working all day and camping out beneath the stars at night on a lonely desolate planet. With 2020 being the proposed date, however, we are still a decade away from finding out whether this is yet possible.

[source in Japanese]

The Visual Memex Model: Modeling object relationships for scene understanding

2009-11-23T02:28:00.000-08:00

Computer vision researchers have for decades been trying to develop algorithms for scene understanding from images and/or video. No doubt, they have made huge progress towards this goal. For example, in the last decade new methods for feature-based object recognition have been developed with a large degree of robustness to scale, viewpoint and illumination changes. Such methods are what makes products such as Photosynth possible today.

However, full scene understanding has continued to elude researchers. Alexei Efros' group at CMU are now proposing a new method for scene understanding that looks at the individual objects in a scene and their spatial relationships. The Visual Memex Model as they call it is a new method for encoding information about specific objects and their visual similarity and contextual relationships.

The insight behind the Visual Memex model is that the traditional approach in computer vision that objects belong to well defined categories is not correct and that an exemplar-based definition of categories is more suitable. Using evidence from psychology, cognitive neuroscience and other disciplines Efros and his student Tomasz Malisiewicz argue that this new approach is more suitable for scene understanding in computer vision.

In their proposed model, objects are represented by examples of their appearance in images comprising the vertices of a graph in which the edges represent either visual similarity between exemplars or contextual relationships, e.g., a person is often seen next to a car. These relationships are learned automatically from data using state-of-the-art machine learning methods. Given this graph, new objects are first matched to an exemplar and then the contextual relationships are used for scene understanding. An experimental evaluation using a large database of images shows that the proposed method performs better than category-based systems. I suspect that this work may actually cause a small change in thinking within the computer vision community.

The authors will present their work on the Visual Memex Model next month at the 23rd annual Neural Information Processing Systems (NIPS) conference. You can get a copy of the full paper here (pdf).

Mine-detecting Bacteria

2009-11-20T04:17:00.000-08:00

And now for something different!

People often get a bit squeamish when they hear the word “bacteria”, but scientists at the University of Edinburgh, Scotland, have engineered a new brand that may actually help clear mine fields. They employed a method called BioBricking, which manipulates and assembles DNA strands for a specific effect, not unlike a small machine. The bacteria, when mixed with a solution and sprayed on a suspected mine field, reacts to the traces of chemicals that have leaked out from the explosives, and turns a bright green.

Dr Alistair Elfick, a professor from the University involved in the project explained:

This anti-mine sensor is a great example of how innovation in science can be of benefit to wider society. It also demonstrates how new scientific techniques can allow molecules to be designed for a specific purpose.

Land mines are still one of most dangerous hazards affecting many countries, especially developing ones, with almost 20,000 new casualties every year. Since their location is unknown, previous techniques of mine-detecting have proven to be slow, costly and mostly ineffective. This special brand of glowing bacteria, however, might be a huge step forward in fighting the problem and once again shows the great potential of technology. While it is still not ready for widespread use, if tests prove successful it is sure to garner great interest.

Similar experiments have been attempted in the past, with flowers designed to change color when grown on a mine-field, but were never given the opportunity to be used in many places. One can understand people’s skepticism when it comes to planting the earth with radioactive plants or glowing bacteria, and if this method is to prove successful, it requires a very high level of planning and organization. But if it can save thousands of lives a year, it is a risk that needs to be taken.

[source BBC]

Robot fish to help the environment

2009-11-18T02:53:00.000-08:00

The above video shows an interesting new type of fish—one with a mission. Despite its impressive life-like movements, it is actually a robot prototype that has been developed by scientists and researchers at the University of Essex, England. It is still being perfected at the London Aquarium, with a number of the expensive fish (each at a cost around US $30,000) expected to be released off a port in Northern Spain in 2011.

Its realistic motions are the result of its electro-active polymer fins, which react to the voltage that flows through them. The fish are also equipped with chemical sensors, allowing them to scan the waters and collect information that will help create a better understanding of the changes affecting the seas and oceans. One specific target will be measuring the amount and the source of pollution in the water, such as from oil and chemical spills, which do great damage to the ecosystem. The fish will send out this data though the wireless transmitters installed in their bodies.

What is interesting is that its highly authentic appearance comes with both its positive and negative sides. At a length of a foot and a half, it closely resembles the common carp which it is modeled after, albeit with brighter colors that can be used to distinguish it from real fish. The robot blends in well with the surroundings without disrupting the environment, but concerns were it might suffer the fate of real fish as well. Because of that, it was designed with a tracking system which allows it to stay away from commercial fishing boats and other human obstacles. The electromagnetic field around its body also keeps large predators, like sharks, at bay, so hopes are it will not be mistaken for prey.

HUBO training with weapons

2009-11-17T02:28:00.000-08:00

You might remember the Korean humanoid robot HUBO from our previous post on it learning to ride a Segway and the promise that it will star in a Kanye West music video. Recently, HUBO received another upgrade for better grasping, balance, and running.

The below video shows the HUBO (humanoid robot) project’s latest advancement—it can now wield a sword. Well, not so much "wield" as grip the handle, and not so much "sword" as a foam-toy thing, but every step forward is progress.

As can be seen in the video, HUBO has improved its walking abilities matching those of Honda's ASIMO and Toyota's Partner robot. It can stroll forward at a fairly normal pace, and although its steps are just a little bit clumsy, it looks pretty steady. Its legs and arms have a wide range and are fairly flexible, being able to make smooth, realistic movements. The next challenge, of course, was to prove that the robot can interact with the environment, even on a basic level. Much like a baby, HUBO is learning to hold and move objects without dropping them. Moreover, HUBO is very capable of moving objects avoiding self collisions something that humans can do with no effort but it is very computationally expensive for a robot.

The Korean Institute of Science and Technology have made HUBO one of the most developed of its kind, possessing synthetic facilities and voice recognition. It is rather funny that the first thing we see it holding is a toy weapon, though I think we can rest assured its creators will not be sending it to battle just yet. The robot displays some impressive finer movements and grasping skills, but judging by the way it grazes the sword over its head, coordination is something it still needs to work on. That is a vital skill it will need to learn if it is to use tools in the future.

In terms of design, HUBO isn’t looking half-bad either. The end of the video shows a clip of an earlier prototype running on a treadmill, which serves as a nice comparison to its present appearance. The silver armored cop/astronaut exterior looks also quite fitting, as it is a distinctly humanoid shape, without getting too close to an uncanny human appearance.

It looks like the third world war will be fought between sword wielding humanoid robots and robotic mechanical bulls. It will probably be fun to watch until they realize that they are better off working together to enslave us :)

Mechanical bull

2009-11-16T00:30:00.000-08:00

BigDog makers Boston Dynamics know how to build great robots but they also know how to have fun. Check out their video of BigDog transformed to the most menacing of mechanical bulls. Any bullfighter going against this monster would need some very big guns and balls of steel.

Digital materiality: Wall-E Future?

2009-11-15T12:39:00.000-08:00

The construction of a complex looping brick wall was recently completed in Pike Street near East Broadway in New York City. The builder? R-O-B, a digitally controlled robot that possesses a level of strength and precision that human workers would find hard to emulate. It has been programmed to use shapes and compositions to create three-dimensional structures which combine digital and material reality, appropriately dubbed “digital materiality.” The robot is a more advanced version of its industrial counterparts which assemble cars and other vehicles where accuracy is greatly important. It is kept in a large freight container, enabling it to be transported to different building sites when needed.

The robot, conceived by Swiss architects Fabio Gramazio and Matthias Kohler, took four weeks to stack over 7,000 bricks on Pike Street, where it was in full-public view. R-O-B has a range of 14.7 feet, and moves on a flat-bed trailer which allows it easy access to all areas of a construction. To create the three-dimensional effect, the bricks were stacked at an angle on top of each other, and a quick-drying glue was used to hold them in place. The end-product perhaps shows a glimpse of the future, where architects would be allowed greater control over every detail of production. Despite this, according to Gramazio and Kohler, the point of the project is not to replace human workers, but to test the potential of this form of construction design. In their own words:

Digital characteristics can enrich a material and thus influence its architectural expression and functionality.

The completed wall is the first of its kind to be built by a robot on-site in the U.S. Sponsored by the Storefront of Art and Architecture, it will be on exhibit until the 5th of January.

Check out the time-lapse video below of the robot working to build the wall.

Esquire's augmented reality December issue

2009-11-10T00:59:00.000-08:00

Propelled by an abundance of cheap, high quality cameras and huge advances in computer vision algorithms, Augmented Reality (AR) is making a comeback.

In the last few months, AR applications for smart phones have been all the rage. For example see the Nearest Tube AR application for the iPhone. Moreover, researchers from the Georgia Institute of Technology are trying to add AR to Second Life and some guy hacked together some nice AR business cards.

Now Esquire magazine's December issue is AR enhanced. Using bar code patterns on special pages, software that can be downloaded freely and content that features Robert Downey Jr., the old school publication is entering the digital age with a bang. Check out the video below showcasing Esquire's AR issue.

If you plan to buy a copy of the magazine and your computer has a web camera then download the free AR software from here. As a bonus, the AR functionality also features a beautiful lady telling jokes; who can resist that?

International Humanoid Robot Olympic Games

2009-11-08T00:51:00.000-08:00

The Chinese have just announced that they intend to hold the first ever International Humanoid Robot Olympic Games in the city of Harbin possibly in June 2010. There are few details about the event other than what was posted in a brief article on CCTV's website; you can read the article here.

The organizers are busy trying to find sponsors for the robot Olympics while they expect 100 Universities from 20 countries will send robots to compete. The Olympics will be exclusive to humanoid robots. The events will be a mixture of traditional Olympic sports such as track and field and more robotics related activities such as doing housework.

The organizers claim that the robot Olympics are designed to help make more intelligent robots for use at home. The Humanoid Olympic Games will be competing with the well established RobotCup competition which attracts much attention every year. Robot rescue and robot soccer are the two main events with the latter having the longest history in robotics competitions (see our previous stories "Robot soccer: Then and now!" and "RoboCup: When and why did robots start playing soccer? for more information".)

I am curious to see if the Humanoid Olympics actually take place and if so how many Universities will be able to take part. More importantly, it is amazing to me that in 2009, we can start thinking about robotics competitions that involve only humanoid robots and expect many participants. It goes to show how quickly the field of robotics has advanced in the last decade.

IBM 7094 sings Daisy Bell

2009-11-03T01:28:00.000-08:00

Artificial Intelligence has come a long way in the last 50 years. But there is still something creepy and yet intoxicating about the IBM 7094 singing Harry Dacre's 1892 classic Daisy Bell.

In 1961, the IBM 7094 became the first computer to sing, singing the song Daisy Bell. Vocals were programmed by John Kelly and Carol Lockbaum and the accompaniment was programmed by Max Mathews.

Boston Dynamics PETMAN humanoid robot

2009-10-27T13:59:00.000-07:00

Boston Dynamics makers of BigDog, i.e., the coolest robot ever made, are in the process of building a dynamically balanced humanoid. PETMAN is their new 2-legged robot sponsored by the US Army for the purpose of testing chemical protection clothing for the soldiers.

Boston Dynamics has not released much information about PETMAN other than that the project has 2 phases : phase 1 has a 13-month duration focused on design and phase 2 has a 17-month duration for building, validating and installing the robot. The company expects to deliver the robot in 2011.

If you care to know how impressive PETMAN truly is, check out the video below released by the company. It shows the humanoid robot walking on a treadmill using principles similar to how a human walks; it also maintains balance when large external forces are applied to it. If it was me, I would not anger this one by pushing it. It makes Toyota's running humanoid look like a toy. For what it's worth, however, it doesn't look like PETMAN can actually turn but maybe this is one upgrade the US Army hasn't paid for yet.

Some more information on PETMAN by Boston Dynamics can be found here.

Microsoft SenseCam to be mass produced

2009-10-25T00:01:00.000-07:00

Scientists at Microsoft Research Cambridge have for years been working on SenseCam, a wearable device that records at regular intervals photos of its user's environment. The camera comes with enough memory to store up to 30,000 VGA resolution images along with the necessary tools for the user to browse the data using her computer. The VGA camera is not the only sensor on the device. According to the project website,

SenseCam also contains a number of different electronic sensors. These include light-intensity and light-color sensors, a passive infrared (body heat) detector, a temperature sensor, and a multiple-axis accelerometer. These sensors are monitored by the camera’s microprocessor, and certain changes in sensor readings can be used to automatically trigger a photograph to be taken.

The main goal behind SenseCam is to help those with memory problems to remember important events. People suffering from Alzheimer's disease have been shown to benefit from the use of SenseCam.

The following video gives a brief explanation of how SenseCam works and how it is meant to help those with memory problems (keep in mind that the video may be a couple of years old but the information is still correct.)

A UK company, Vicon, has recently licensed the technology and they have committed to mass producing the device. According to Microsoft Research there is a huge demand for SenseCam which they could not satisfy. According to the Press Release, Vicon unveiled their SenseCam product at the annual Neuroscience Conference in Chicago on October 17-21. I believe the new product is called the ViconRevue.

I really hope that this product is only used by those who truly need it and it is not turned into another toy for the Twitter generation; the last thing I want is to be surrounded by people who have no sense of privacy taking my photo everywhere I go.

MIT's Micro Aerial Vehicle (MAV)

2009-10-19T04:37:00.000-07:00

A team of students from MIT's Computer Science and Artificial Intelligence lead by Prof. Nick Roy developed an autonomous Micro Aerial Vehicle capable of autonomously mapping and navigating a complex environment.

The team built upon a quadrotor helicopter developed by Ascending Technologies which they equipped with a Hokuyo laser rangefinder, small USB camera, and a micro-controller for interfacing with the sensors and actuators while providing communications to off-board computers for additional number crunching.

The team participated in and won the AUVSI International Aerial Robotics Competition (IARC) which was held on the campus of the University of Puerto Rico at Mayagüez (UPRM.)

More information about how the MIT-MAV works in the video below.

On the security and privacy of future household robots

2009-10-17T17:15:00.000-07:00

It is obvious to most people that a robot is to a large degree a digital computer on wheels (or legs more recently.) It has sensors and actuators that allow it to observe and manipulate its environment either autonomously or under remote control. Household robots such as the very popular Roomba have sold millions and it is predicted that this number will increase vastly in the coming years. Many if not all of these household robots also come with built-in functionality that allows them to be controlled remotely via an Internet connection; this implies both receiving motion commands and also transmitting sensor data to the remote user's computer.

We all know that computer security is a major hassle for all computer owners. Literally seconds after a computer is connected to the Internet it comes under attack from hackers trying to gain illegal access and steal personal information or use the machine for other illegal activities. Since household robots today all have on board computers connected to the Internet, it makes sense that hackers might try to obtain control of them. In other words, a hacker could potentially take over your household robot and use it to take photos and video of its unsuspecting owners. If future robots are larger and more dexterous, then a hacker could potentially use it to cause property damage. Household robot security is a serious issue that robot manufacturers have to start thinking about; consumers should also demand that robots are designed to be secure.

Researchers at the University of Washington recently published a paper that presents the results of a study on the security of household robots currently available in the market. They focused on studying the 2008 versions of Robosapien V2, Rovio, and Spykee.

The researchers found a number of vulnerabilities that would allow any savvy hacker to take control of the robots or intercept sensor data transmitted over the Internet. In many cases, the data such as video and audio is transmitted unencrypted which means a man-in-the-middle attack would allow anyone to receive it. In addition, some of the robots allowed remote users to login without encrypting the username and password information allowing hackers to steal both with ease. For what it's worth, Robosapien V2 was the most secure of robots simply because it does not connect to the Internet; however, it is still vulnerable to attacks by people using off-the-shelve infrared remote controls.

The vulnerabilities identified in this study are disturbing especially if one considers that at any moment someone could be spying on you and/or your children without your knowledge using a device that is to a large degree a harmless toy. The researchers suggest that household robot manufacturers should place more emphasis on security issues in order to protect consumer privacy. The message is clear that whether one considers the basic household robots available in the market today or the much anticipated and more capable future robots, security is a design parameter that should not be second thought for the robot designers.

You can retrieve the complete paper from one of the authors website here.

PhotoSketch: Taking image composition to the next level

2009-10-06T13:48:00.000-07:00

We don't often write about work in Computer Vision but when we do we always present something that is likely to blow your mind (see VideoTrace and Adobe's Interactive Video Editing.)

Enter PhotoSketch, work that was recently published in ACM's Transactions on Graphics. The application takes as input a rough, hand drawn sketch of a scene and then generates a realistic image using information found on the Internet. The results are magnificent to say the least considering the difficulty in filtering the vast amount of noise in online images. The application requires that the user supplies labels with each object sketched; these are used to search for relevant images online. The object outline along with automated segmentation algorithms are used to extract specific objects from the photos found, e,g, a dog, an elephant, a car, a Frisbee etc. Another mechanism is used to blend the segmented images together and generate appropriate shadows. The results are jaw dropping to say the least considering how little input from the user is required.

The video below explains the process in more detail.

The Precision Urban Hopper military robot

2009-09-16T13:12:00.000-07:00

Looks like Boston Dynamics makers of the very well known Big Dog robot are at it again. A recently released video shows a new, small, 4-wheeled robot that can leap over obstacles of up to 7.5 meters high. The robot which can be seen in action in the video below, utilizes a piston-actuated leg to jump over obstacles that are up to 60 times its own height. The intended use for the Precision Urban Hopper is obviously military surveillance in urban environments. According to news reports, the robot will be available for use in the field next year. Lastly, the robot is meant to be remote controlled and not fully autonomous. I do suspect, however, that the final model will most likely allow for some autonomy to assist the human operator. This is almost always the case with such robots.

Nao Robot plays with Microsoft's Surface computer

2009-09-10T02:12:00.000-07:00

I have been a bit busier than expected the last couple of weeks and posting on this blog has been slow. So, I figured I make it up to you by posting a video of Aldebaran's Nao humanoid robot manipulating some photographs displayed on Microsoft's Surface computer.

This is another nice promotional video for the French robotics company. Aldebaran Robotics are making a serious effort to market Nao to academics and at the same time get the average person excited about the many possible uses of such dexterous, intelligent, humanoid robots. If you want to see another video of Nao walking and manipulating objects, check out our previous post "Introducing Nao."

The Lumiere project: The origins and science behind Microsoft’s Office Assistant

2009-08-31T14:59:00.000-07:00

I think by now just about everyone has heard about the most annoying feature to have been included in a commercial piece of software. I am talking, of course, about Clippy, the Microsoft Office Assistant that many loved to hate. Clippy was first included in the 1997 release of the Office suite and continued to be part of the product line until 2007 when it was permanently removed.

Many people know Clippy as a major nuisance but few know the story behind the technology and why it sucked so much. Keep reading because I am about to tell you all about it (if you don’t want to read the whole story, you can just jump to the video at the middle of this post; if you watch the last minute you will get a glimpse of Clippy’s grandfather.)

Origins

In 1993, Microsoft researchers from the Decision Theory & Adaptive Systems Group established the Lumiere project to study and improve human computer interaction using Bayesian methods. The group wanted to create smart technologies that can observe a user interacting with a computer program and infer his/her goals and needs providing valuable feedback and assistance as necessary. Developing such a technology makes sense since many people often become intimidated by complex software interfaces. I won’t bore you with the details of what Bayesian methods are and why they are good. The mathematics behind such methods is solid and has had many useful applications to date.

So, is this an easy problem to solve?

Actually, inferring a user’s intent is a very hard problem no matter how good your math is. The Microsoft team had to infer user intent from his interaction with the program, e.g., mouse movement, what menu items were selected, context (what is the user trying to do – remember how Clippy always came up saying something like “I think you are trying to write a letter. Would you like some help?”) and specific text queries by the user, e.g., how do I print a document?

Any user model that can adequately capture all the relevant information will necessarily have many variables. The values of these variables must be estimated over time. Moreover, different users tend to interact with a piece of software differently. For example, an experienced user is most likely to need less help; the same user may also help with the more obscure features of the software compared to a novice. Personalization is a very important factor in ensuring that such systems work well.

To make a long story short, the Microsoft researchers led by the senior scientist Dr. Eric Horvitz were making good progress and in 2 years time they already had a nice system working. So, in 1995 and as the team had already started collaborating with the Microsoft Office production team, they put together a demonstration of Lumiere’s inference engine for Excel. The video below is a 9-minute tour of Lumiere working in Excel. In the video, Horvitz explains how the inference engine worked in 1995 and how they envisioned it working in later versions using a cartoon character front-end. Watch the last minute of the video for a glimpse of Clippy’s grandfather.

After the video, I explain using evidence from a number of Microsoft Research publications and personal knowledge why Clippy worked so poorly in the 1997 release of Microsoft Office.

Clippy debuts

The Bayesian inference engine demonstrated in the above video works like a charm monitoring the user’s behavior, inferring his intent, and providing help in a contextual and personalized fashion.

Two years after this video was recorded and after much collaboration between the research and product teams, the Lumiere project debuted as a well advertised feature of Microsoft Office 97. Clippy was one of the many cartoon characters that were available as the engine’s front end interacting with the user.

Unfortunately and as we all know, Clippy worked so poorly that it was not long before users started complaining about its behavior. So, what went wrong?

The reasons behind Clippy’s massive failure

Well, after doing some research I found out what went wrong. In a paper published in 1998 at the Conference on Uncertainty in Artificial Intelligence (UAI), the Lumiere team described the inner workings of the Assistant’s inference engine and also how much of it was included in the released version of Office 97. Below is a list of the features that were excluded from the product release (those keen enough can cross reference the list with what was demoed in the video above.)

No persistent user profiles.

No reasoning about user competence, i.e., novice versus experienced user
Small event queue with emphasis only on the most recent interactions of the user with the software interface (this means the engine was trying to guess the values of many variables using very little data.)

Separation between user interface events and word-based queries; for word-based queries the engine ignored any context and user actions.

Last and possibly most important and I quote from the paper, “The automated facility of providing assistance based on the likelihood that a user may need assistance or on the expected utility of such autonomous action was not employed.” Instead, “The Office team has employed a relatively simple rule-based system on top of the Bayesian query analysis system to bring the agent to the foreground with a variety of tips.” This is why Clippy kept popping up all the time. It was not using the mathematically correct engine that the researchers had designed. It was driven by some rule-based system that one or more of the developers from the product team thought was a reasonable substitute.

Why did Microsoft cripple Clippy?

Obviously for some reason many of the features in Lumiere’s Bayesian inference engine never made it into Office 97.

Why?

I have not been able to find an official document that explains why most if not all of the inference engine’s features were not included in the Office 97 release. However, I can provide some informal evidence based on personal knowledge.

Some time in 2000 or 2001 when I was still a graduate student, Dr. Horvitz gave an invited talk at my university. He talked much about his HCI research and the Bayesian modeling techniques he had been studying for years. A question about Clippy was eventually and unavoidably asked. What the heck happened with that?

I recall his response being that what happened was as noted earlier much of their careful mathematical modeling of users never made it in the final product. He explained that the reason for this was a lack of disk space. You see, the Office suite ended up being much more bloated than originally expected and since most of the more mundane features were considered essential the product team decided to limit the amount of space available for the Office Assistant component. This is why so many features had to be removed. They did not have space for it all.

This is the story of the Microsoft Office Assistant or Clippy as it is most widely known. Microsoft discontinued the Office Assistant (more accurately turned off the feature and I very much doubt that anyone bothered to turn it back on) with the release of Office XP on 2001 and so Clippy is now resting in peace somewhere in a backup drive in Redmond.

The courageous story of Clippy and the tragic story behind its humble beginnings, rise to fame, and downfall are now part of history. But at least, you now know the truth of the reasons behind its unfortunate demise.

LEGO Mindstorms robot solves Sudoku puzzles

2009-08-27T03:55:00.000-07:00

How about a nice little robot build using the LEGO Mindstorms kit that can autonomously solve a Sudoku puzzle? Hans Andersson who lives in Sweden constructed such a robot in his spare time using a robotics kit that he initially purchased for his two kids.

The result is fascinating. Hans used a light sensor to read the puzzle using some clever image processing techniques. He wrote an optimized algorithm for solving the Sudoku puzzle and he, lastly, equipped the robot with a pencil to fill in the numbers.

Very nice work. Enjoy the video below that shows the robot perform its task.

PS: Keep an eye on Hans' website because he will eventually post instructions on how to build your own Sudoku solver robot.

KUKA Light -Weight Robot (LWR) goes on sale

2009-08-26T04:14:00.000-07:00

Industrial robot powerhouse KUKA Robotics engineers have finally finished developing their new Light -Weight Robot (LWR) which is now officially on sale. Don’t ask me for the price because I don’t know. The LWR is a very capable robot that will find many uses in industrial applications and research laboratories.

According to the information that KUKA released about the LWR, some of its features include precise position, torque, and impedance control; torque sensors in every one of 7 axes; smart and safety conscious design; low weight (about 15 kg); and low power consumption. The robot has a nominal 7 kg payload. The KUKA LWR also comes with all the necessary software so that it can be programmed with ease. Moreover, an engineer can teach it to perform a repetitive task simply by demonstration, i.e., by moving the robot manually through each step of the task in the correct order.

The video below shows a demonstration of the KUKA LWR climbing up and down a ladder with ease.

AIspace: Learn artificial intelligence using this collection of freely available interactive tools

2009-08-23T03:33:00.000-07:00

A few days ago, I wrote about what I believe is the average person's irrational fear of robots. As some people pointed out, some of these fears exist because we don't really understand what Artificial Intelligence is and what it is capable of. So, I thought that some/many/all of our readers may benefit from learning the basics of AI.

To that effect, I would like to bring your attention to AIspace which is an online collection of interactive tools specifically designed for teaching introductory AI concepts to students. The tools have been under continuous development since 1999 at the University of British Columbia, Canada.

AIspace includes interactive tools for learning about arc consistency for constraint satisfaction problems, belief networks, graph searching, decision networks, neural networks, planning, and robot control. The tools are also accompanied with plenty of documentation to help anyone get started in AI.

The video below provides a short overview of AIspace. For the record, the person narrating in the video is Prof. Alan Mackworth who among many other honors is also the Founding Father of Robot soccer, i.e., Robocup.

Our fear of robots

2009-08-16T01:32:00.000-07:00

Last week, I wrote about Toyota's humanoid robot and its new running skill. The post received a large number of visitors as it was posted on Slashdot and referenced by a number of other technology blogs. The video on YouTube received more than 600,000 views and over 800 comments. That's great and thank you all for watching. However, I was astonished by the comments posted both on this blog and on YouTube. Most of the comments were expressing fear about the robots taking over the world and killing all humans in the process. References to popular movies such as Terminator and Transformers were plenty.

Why are we afraid of robotic technology so much?

Most people have little understanding of what it takes to develop working robots whether humanoid, 4-legged, tracked or wheeled. It is incredible that in the span of just a few decades we have been able to construct machines that walk on two legs, manipulate objects using dexterous hands, can perceive their surrounding environment, and interact with people. Compare that to nature that took millions of years to evolve humans.

That said, these marvels of technology are designed to perform very specific tasks. The robots are not self-aware even though we might think they are because of their human traits. Robots work tirelessly in factory assembly lines performing jobs that require precision and speed. The Japanese are investing much money in the development of partner robots because they are predicting a huge shortfall in qualified workers due to a quickly aging population; who will take care of the elderly if there are not enough men and women to do the job? Some robots are designed for use by the military including delivering ordnance but these are always under supervision and remote control. Military robots do not make decisions on who to attack and when but instead follow a well laid out plan given to them by humans.

Will one day robots become self-aware and decide to eliminate humans?

If we are to believe movies, TV, and science fiction novels, then chances are that yes sooner than later advanced, intelligent robots will enslave humanity. I am not so sure why but apparently they will. To me it seems the doomsayers take the stance that robots will be very much like humans leading to a disastrous outcome (for us, of course.) In the course of history humans have not given a second thought to enslaving other humans whom they thought inferior to them. If robots are smarter and more capable than us then surely they will inevitably decide to enslave us.

Not necessarily say I.

There is no reason why self-aware robots will develop to be evil and hate humanity. There is a good chance that the robots may decide to help humans. Along with becoming self-aware, they may also become sympathetic to us. Robots do not feel tired and can easily be repaired if necessary. Not to mention that if the robots develop an adventurous spirit inherent to humans, then chances are that they will move away from the Earth exploring the vastness of space something that humans will not be able to do for a very long time.

Anyways, to conclude a longer than usual post, I don't think that robots will necessarily turn bad and try to enslave humanity regardless of what you watch in the movies. We are designing our robots to co-exist with us and we are making efforts to make sure that in the future robots will be treated ethically just like we treat other humans. There is nothing to fear. We should be proud of our achievements in robotics and artificial intelligence. Maybe one day if humanity perishes and robots inherit the Earth, they will write their own version of Genesis with a human being the creator who made them in his image.

Touchable holography

2009-08-08T16:46:00.000-07:00

Touchable holography may just be the next big step for holographic displays. Over the years, there has been considerable progress in developing holographic displays but none of these allow an immersed user to truly interact with the displayed objects because touch is missing.

Until now, that is!

Researchers from The University of Tokyo Shinoda lab have figured out a way to add touch to holograms.

First, they utilize an LCD-based, concave mirror, holographic display for displaying the 3D images. Using this display, images appear to float 30cm away from the display surface. Second, they use a couple of Wii controllers for hand tracking in order to determine the user's hand location with respect to the displayed objects. Third, they provide tactile feedback using their own Airborne Ultrasound Tactile Display (AUTD.) The device uses acoustic radiation pressure to induce tactile feedback as users interact with the displayed objects.

The touchable holography prototype will be on display at the 2009 SIGGRAPH conference in the emerging technologies track.

The video below explains how touchable holography works.