Not only did the robot need eyes, he also needed to be able to recognize what he was looking at. The objective of the interactive was to have young children track down specific objects on a surface. To give them a prompt and to make the video of the surface a bit more interesting, a target was to be overlayed on the video pointing to where the object was. This meant that the Rovio would need to distinguish between the unique objects on the driving surface. So what set the objects apart from each other?

Shape and size: Using these characteristics and constructing custom object (similar to facial) recognition using opencv and haartraining the different objects could be “seen” by the rover. This ended up being a balance of accuracy and speed, of which neither was good enough to use in the application.

Position: Using the rover’s position, both location and orientation, the specific object could be determined from its coordinates. The numbers coming back from the rover, through the beacon, ended up being too cryptic and not reliable enough to give specific positioning information, especially for moving a precise onscreen target. There was also no way to get vertical movement information.

Color: Pairing a unique color with the object to be identified could allow color detection to run the brain of the rover. This ended up being the most streamlined and accurate approach to identifying object in the interactive. Using a vibrant red, green and blue gave enough differentiation between the objects to bring up specific prompts for each target. Running the color detection on the video stream and then creating a box around the identified color gave a specific position in the video frame. The target could then be positioned in the center of the box, and exceptions could be made for outlying false positives. Color scraps could be disregarded and focus could be on heavy concentrations of the specific colors. Below you can see the blue box (hidden when not in debug mode) surrounding the green colored image.

Lighting greatly impacted the color detection on the objects. Above is a picture of two of the colored boxes blacked out by the background window light. The objects needed to be lit from a reliable, consistent source of light. This light would have to be pointed on the object or what was done in this case from within the object. The targets were lit with colored light to attempt to nullify lighting differences due to sunny or cloudy days (the interactive being in a heavily windowed area). Below are images against the window areas. First, the transparent gels of the three different colors (the red needed some help and has a light behind it). Second is a stronger light competing with the self lit color cone.

Recipe for robot eyes:
Wowwee Rovio
Red 5 video server
Flash netstream/video objects
Color detection

A year ago I published a post about widgets coming to the television. A year later I have widgets on my TV. Now that I have experienced the widgets first hand, there is not much change to my feelings about the basics of the previous post.

There is still no bridge between the cable content coming over the wire and the widgets on screen. Widget content cannot be populated by on-screen cable content. This leaves out advertising tie-ins and related content that would be automatically populated.

There are however widgets for pulling video content from online. This enables the content to interact with additional features. The main downside to the online content is the timeliness of the content in comparison to traditional counterparts. There is very little to no live television content being delivered through the widget interface, or anywhere online. Any online movie rentals or purchases are delayed by at least a month compared to in-store movies.

HD is another problem area for online content. There are allot of bits to move for HD content and until broadband infrastructure is in place to handle the content, a coaxial cable is the only option.

Online content delivery still needs time to develop and find a viable business model. Timely content will not be allowed to move to an online market unless it promises not to threaten the more profitable traditional models, and until timely content is delivered people will still want cable/satellite and physical dvd rental.

1. The interactive needed a custom interface so the built-in web interface (and keyboard controls) would not work.
2. The interactive needed to be accessible to children as young as 3 years old.

The custom interface was built with ActionScript3 and so to move the rover, the api was used. With the api, the rover could be sent movements via http requests. These api requests needed to be mapped back to user input and so the web interface was recreated in with as3. The ASDW keys were mapped back to specific movement requests via the api. This is a specific piece of code initiated by a keypress that gets sent to the rover:

switch(e.keyCode){
//forward
case 87: //W key
drive = 1; //1 maps to forward
speed = 1;
request.data = "Cmd=nav&action=18&drive=" + drive + "&speed="+speed; //action 18 is manual drive mode
loader.load(request);
break;
}

The drive variable signals the command/direction for the rover (forward, backward, rotate right) and the speed variable is how fast the command should be executed by the rover. Sending one of these http requests would move the rover a fraction of an inch. Mapping each key to a specific movement and sending multiple commands successively reproduces the motion of the built-in interface. This does not remedy our problem of simplifying the interface for 3 year olds. This is where the custom hardware comes in.

Our custom hardware consists primarily of a joystick but also includes some auxiliary buttons for game functionality which I will describe in a following post. The hardware came from Suzo Happ and also included a UGCI board for USB interface. The UGCI board had an included key-mapping utility which was used to map the joystick and button contacts to key presses. A constant connection with the joystick in any direction sent a repeating key press rather than a key up and down.

With key presses mapped to the joystick control and to the rover’s api through actionscript, the rover could now be driven with custom joystick controls. Now if only our rover could see…

Over the past months at work I have been taking a consumer mobile webcam (robot) and converting it to custom interactive experience. Over the next couple of weeks I will be talking about some of the finer point of taking an off-the-shelf WowWee Rovio (which comes with some fine built-in features) and transforming it into something much more. I will be talking about the following modifications in the coming weeks:

• Custom hardware controls
• Processing the video stream in Flash
• Object detection
• Docks and Beacons

This is on the side of the Rovio box:

I will be proving this incorrect.

First the condition must be set. This would be the circumstances under which the assessment is performed. This could be further broken down for software into the current state of the application and the action that the user takes to initiate the test. In regards to instruction this could be the classroom conditions or the specific format for the test (worksheet, one-on-one, with a behavioral prompt).

Next is the specific behavior that is to be observed. This should be something that is both observable and measurable. While the tests that we are performing are higher level tests (functional/usability), this could be applied simpler unit tests that would be measured by simple logic (does 2 + 2 evaluate to be 4?). Seeing that a video plays without visual stuttering would be observable. This behavior could however use some clarifying in terms of what the visual stuttering would be defined as, or would be left to subjective opinion, which may at some point be acceptable. In the educational realm, the behavior could be a students verbal response to a behavioral prompt or responding to a question with a correct answer.

The final relevant similarity is the criteria used to measure success. In most cases with software testing, the only acceptable criteria is pass or fail, but in some circumstances this could be measured with a success rate/percentage.

Given [condition] the application/student will [behavior] at [criteria]

At it’s basic level both of these tests are about quantifying behavior and removing subjectivity as much as possible. Someone unfamiliar with the inner workings of the application or the student should be able to record the same results as someone who is familiar with the subject.

Social networking on the web is just an extension of social networking in person. You are connecting with other people for a variety of purposes: business, social, common interest. As the internet came along we adapted these same ideas to the tools of the time. Chat rooms and email became the tools of socializing. As the web became more of a marketplace of ideas it became more social. Specific tools to socialize became available adding yet another avenue to network with others.

Because social media is so prevalent it is a hot place to be in terms of a business. Advertising dollars are continuing to be thrown at the web and at social media and there is growing marketplace on the internet and especially within social media outlets. Why? This is where the people are gathering and talking, why not talk about you?

One of the tools of social media is Twitter. Twitter is a micro-blogging platform limiting posts (tweets) to 140 characters. This character limitation has its roots in cell phone communication but Twitter has grown since it’s cell platform start. Twitter is a give and take, where you can follow people and have them follow you. Signing up for Twitter is fairly straight forward. Click on the sign up link, enter your information and start tweeting. If you are a business these micro messages would reflect the message you are trying to deliver about your company. Are you representing an individual at your company or are you the official marketing voice of your company? The answer to this may determine the content of the tweet. Twitter is also good for following others and keeping an ear to the ground about your business’s buzz. Are customers talking up your service or criticizing you for an area they found lacking. Responding to good or bad can more than make up for the original offense. Being able to head off some bad word of mouth can turn a negative advertisement into a good one. How do you find these people to listen to? Search is a great option to find people in your area, industry or people who are talking about you. There are some directories that may also be helpful (wefollow.com) or just using the basic search or “find people” functionality on Twitter may be enough to get started. You may also find that by putting out quality information people may start to find you, and if they are not spammers or bots you may very well want to follow them back. If they are interested in your business you are interested in them.

Another tool to connect is Facebook. Facebook was started on the Harvard campus by Mark Zuckerburg and quickly took root on other college campuses and then around the world. Facebook now has 300 million active users and 50% of those users log on any given day. How does a business take advantage of the networks built on Facebook? Usually through a Fan Page. A fan page is a brand or business landing page on Facebook. The sign up process is similar to Twitter. If you want to build a fan page you can use an existing personal Facebook account or create one fresh. In either case you use the create a page link on the Facebook’s home page. From here you can indicate you type of business and the name of the fan page. When you are logged in you need to add some details to the profile and an profile picture to gain a bit more footing as people search and gain information about your business. Add as much information to the profile as possible to enhance the value of your page. Status updates are similar to Twitter’s tweets.

So why do people share information on the social networks? As with anybody we do things for ourselves. We are promoting a self image, whether it be humorous, helpful, emotional, friendly or any number of different characteristics. So sharing something that triggers on emotions is key to encouraging social marketing. A funny video will get shared around as people want to make others laugh.

This advertisement for Dove may get shared as people want to encourage a healthy self image:

The strength of the emotion and impact of the message increases the sharing potential. These are videos but text or still images can be just as viral. Oprah has been a catalyst for coupon sharing with a KFC grilled chicken promotion and more recently a 50% off Payless shoe coupon that was good for two days only. This type of information relies on getting shared through social tools. People also get a great feeling of helping their fellow man by sharing this information around.

Individuals like to help people by sharing information and that is good for business too. Participating in the community shows that you are a valuable resource and full of expertise. Offering tips or how-to’s is the equivalent to a physical coupon in that you are giving some information away for free in the hopes of gaining a paying customer. Listen to your community and share back. Dell has done this in an ironic way by offering advice about social media to small businesses on Facebook. They took their own advice.

LinkedIn is another social tool more designed for professionals within business rather than businesses themselves. This may be a good way to gain credibility with people within your field. There is a similar simple sign up process and a free or subscription level of service.

To utilize these networks to their fullest you will want to connect them. Use your business website to advertise your membership in each of these platforms. Know that to get the most out of social media networking you need to put time in. Leaving a Twitter account dormant could be just as damaging as not having one at all. This may indicate to customers that you are not listening or contributing anything to the community. So communicate with sincerity and with the intention of advancing your business’s place in the community. This is a legitimate marketing venture.

And if you wondering the results of your labor there are a few ways to track statistics. Facebook fan pages and Youtube videos have built-in numbers associated with their content. If you want a bit more granularity you could share links you may have via a url shortener like bit.ly. Many of these services will offer statistics on individual links, such as how many clicks and where the clicks are coming from (both platform and geographic location). And if you are looking for more feed back on whether social media is working, look to the quality of conversation being conducted between you and your customer.

Social Media Delicious links

Twitter –
Simple one option url to post status message:

http://twitter.com/home?

status=Great+post+on+social+bookmarking+http://electricpineapple.net/?p=96+%23electricpineapple

Basic options for most sites include the url, title and additional information (notes, description, etc). Urls may work without encoding but for best results everything should be url encoded.

Delicious -

http://delicious.com/save?

url=http://electricpineapple.net/?p=96&
title=Bookmark%20%27em%20Danno&
notes=Check out this awesome post

Digg -

http://digg.com/submit?

url=http://electricpineapple.net/?p=96&
title=Bookmark%20%27em%20Danno&
bodytext=Great post on social bookmarking&
media=news&
topic=Programming

Media options for digg are news, video or image. Topics are a fixed list of items. Digg had good documentation for this feature.

Facebook -

http://www.facebook.com/share.php?

u=http://electricpineapple.net/?p=96&
t=Bookmark%20%27em%20Danno

LinkedIn -

http://www.linkedin.com/shareArticle?

mini=true&
url=http%3A%2F%2Felectricpineapple.net%2F%3Fp%3D96&
title=Bookmark%20%27em%20Danno&
source=Electric+Pineapple&
summary=Options%20available%20for%20a%20handful%20of%20bookmarking%2F
sharing%20sites%20in%20the%20form%20of%20a%20url%20string

The mini parameter is required to be true. LinkedIn also was well documented.

Reddit -

http://www.reddit.com/submit?

url=http%3A%2F%2Felectricpineapple.net%2F%3Fp%3D3&
title=Bookmark%20%27em%20Danno

Google Bookmarks -

http://www.google.com/bookmarks/mark?

op=edit&
bkmk=http%3A%2F%2Felectricpineapple.net%2F%3Fp%3D96&
title=Bookmark%20%27em%20Danno&
labels=Social%20Bookmarking&
annotation=Options%20available%20for%20a%20handful%20of%20bookmarking%2F
sharing%20sites%20in%20the%20form%20of%20a%20url%20string

The op parameter is required for submitting. Labels and annotation are additional information fields.

Myspace -

http://www.myspace.com/Modules/PostTo/Pages/?

u=http%3A%2F%2Felectricpineapple.net%2F%3Fp%3D96&
t=Bookmark%20%27em%20Danno

Slashdot -

http://slashdot.org/bookmark.pl?

title=Bookmark%20%27em%20Danno&
url=http%3A%2F%2Felectricpineapple.net%2F%3Fp%3D96

Stumbleupon -

http://www.stumbleupon.com/submit?

url=http%3A%2F%2Felectricpineapple.net%2F%3Fp%3D96&
title=Bookmark%20%27em%20Danno

Newsvine -

http://www.newsvine.com/_tools/seed&save?

u=http%3A%2F%2Felectricpineapple.net%2F%3Fp%3D96&
h=Bookmark%20%27em%20Danno

Technorati -

http://technorati.com/faves?

add=http%3A%2F%2Felectricpineapple.net%2F%3Fp%3D96

This is by far a complete list, please add your own favorite sites in the comments.

1. You will need to create an Air project instead of a Flex project in the initial setup in Flex Builder (WindowedApplication vs. Application)
2. If you did not give the services-config.xml a full path to Drupal Services for the endpoint uri you will need to do that (http://www.mydomain.com/services/amfphp)

That’s it, 2 steps. Air is up and working.

Some assumptions:

• Drupal 6 installed
• Views module installed
• Services module installed
• AMFPHP module installed – with AMFPHP 1.9 Beta 2 – this needs to be in the amfphp module directory
  

Starting with setting up Drupal, we are going to enable the services module, amfphp module, and for this example node service, system service, user service, and views service. After enabled go to the site building/services and make sure that in settings “use keys” and “use sessid” is checked (this is done by default). Now go to the keys tab and create a key with the title of the site you are wanting the key to be used for (just a label for your purposes) and the domain which it will reside on (this will be checked by the service call). This process will produce a key we will use later in our flex application. Note this or have this page available.

Now to the Flex side. Create a new project and include in the src directory the services-config.xml that comes with amfphp (in the browser directory). Modify the uri in this line to point to your amfphp service:

<endpoint uri="http://flashservices/gateway.php" class="flex.messaging.endpoints.AMFEndpoint"/>

<endpoint uri="/services/amfphp" class="flex.messaging.endpoints.AMFEndpoint"/>

This may be different if you have your drupal install inside a directory and not at the root (i.e. /subdir/services/amfphp)

In your project properties/compiler arguments append this line to the existing arguments:

-locale en_US -services “services-config.xml”

We also want to grab the as3corlib for its crypto functions. We’ll use this to hash our api key before passing it to Druapl. Include this in your libs directory.

Now we need to add some remote objects to our Flex project and for this tutorial a couple of forms and a data grid. In this example we will create a new user, automatically log them in, create some node data and view that node data in the data grid (what you do with the information is up to you, this is simply to display different functionality).

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="vertical">
<mx:Script source="ServicesTutorial.as"/>
<mx:RemoteObject showBusyCursor="true" destination="amfphp" source="system" id="system">
<mx:method name="connect" result="onSystemConnect(event)"  fault="onFault(event)" />
</mx:RemoteObject>
<mx:RemoteObject showBusyCursor="true" destination="amfphp" source="node" id="node">
<mx:method name="save" result="onNodeSaved(event)" fault="onFault(event)"/>
</mx:RemoteObject>
<mx:RemoteObject showBusyCursor="true" destination="amfphp" source="user" id="user">
<mx:method name="save" result="onUserSaved(event)" fault="onFault(event)"/>
<mx:method name="login" result="onUserLogin(event)" fault="onFault(event)"/>
</mx:RemoteObject>
<mx:RemoteObject showBusyCursor="true" destination="amfphp" source="views" id="views">
<mx:method name="get" result="onViewsResult(event)"  fault="onFault(event)" />
</mx:RemoteObject>

<mx:Form>
<mx:FormItem label="Username">
<mx:TextInput id="usernameInput"/>
</mx:FormItem>
<mx:FormItem label="Password">
<mx:TextInput id="passwordInput" displayAsPassword="true"/>
</mx:FormItem>
<mx:FormItem label="Email">
<mx:TextInput id="emailInput"/>
</mx:FormItem>
<mx:FormItem >
<mx:Button label="Register" click="registerUser();"/>
</mx:FormItem>
</mx:Form>

<mx:Form>
<mx:FormItem label="Title">
<mx:TextInput id="titleInput" text="{selectedStories.selectedItem.node_title}"/>
</mx:FormItem>
<mx:FormItem label="Body">
<mx:TextArea id="bodyInput" text="{selectedStories.selectedItem.node_revisions_body}"/>
</mx:FormItem>
<mx:FormItem>
<mx:Button label="Submit" click="saveStory();"/>
</mx:FormItem>
</mx:Form>

<mx:DataGrid id="selectedStories" dataProvider="{theStories}">
<mx:columns>
<mx:DataGridColumn headerText="NID" dataField="nid"/>
<mx:DataGridColumn headerText="Title" dataField="node_title"/>
<mx:DataGridColumn headerText="Body" dataField="node_revisions_body"/>
</mx:columns>
</mx:DataGrid>

</mx:Application>

For the remote object the source property is the type of service (node, system, user, views), the destination matches the the destination ID in the services-config.xml, the method name is the service method we want to call (get, save, etc).

These remote objects need their corresponding result handlers so we will create another ActionScript file to hold our script. In this we will create all of the result handlers for our service calls. This including the single fault handler which would ideally would have different messages/actions for each type of service call.

import com.adobe.crypto.HMAC;
import com.adobe.crypto.SHA256;

import mx.controls.Alert;
import mx.rpc.events.FaultEvent;
import mx.rpc.events.ResultEvent;
import mx.utils.ArrayUtil;

private var apiKey:String = "your api key from drupal services";
private var apiDomain:String = "your api domain from drupal services";

public function onSystemConnect(event:ResultEvent):void{
}

public function onUserSaved(event:ResultEvent):void{
}

public function onUserLogin(event:ResultEvent):void{
}

public function onNodeSaved(event:ResultEvent):void{
}

public function onViewsResult(event:ResultEvent):void{
}

public function onFault(event:FaultEvent):void{
Alert.show("there was a problem");
}

public function hashKey(serviceMethod:String):Array{
var captureTime:String = (Math.round((new Date().getTime())/1000)).toString();
var captureRandom:String = randomString(10);
var hashString:String = captureTime + ";";
hashString += apiDomain + ";";
hashString += captureRandom +";";
hashString += serviceMethod;
return [HMAC.hash(apiKey,hashString,SHA256),apiDomain,captureTime,captureRandom];
}

private function randomString(Stringlength:Number):String{
var allowedChar:String = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789";
var allowedArray:Array = allowedChar.split("");
var randomChars:String = "";
for (var i:Number = 0; i < Stringlength; i++){
randomChars += allowedArray[Math.floor(Math.random() * allowedArray.length)];
}
return randomChars;
}

At the end of this script is the hashing function which will return an array of the information we need and a helper random string generator. You’ll notice also that there are imports for the crypto functions in the as3corelib and we have set strings containing our api key and the api domain, fill these in with the information you generated with the services module.

Now we can start calling our services and telling flex what to do with the results. First we will connect to the system to get a session ID from drupal as well as check if there is a user currently logged in. We will call the system.connect method from an init function run in creation complete. Add the creationComplete event to the application and the corresponding function.

mxml:
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="vertical" creationComplete="init();">

as:
public function init():void{
system.connect();
}

When we get a response from system.connect it returns an object with the session id and a logged in user. We want to retreive this information and store it for subsequent calls. In the onSystemConnect method we’ll handle the results and store these in appropriate varaibles, we’ll put these at the top to keep them in scope with the rest of the functions.

private var sessionID:String;
private var currentUser:String;

public function onSystemConnect(event:ResultEvent):void{
sessionID = event.result.sessid;
currentUser = event.result.user.name;
}

Also for this call to work as an anonymous user we need to open up the permissions allowing anonymous users to connect to services (user/permissions/service module/access services). You’ll notice that I am storing the user’s name rather than uid. This is because Flex uses the uid internally as a “unique identifier.” When we save the user name with other data Drupal will complete the Drupal uid when it sees the name variable.

After we connect to services we want to do something. I have chosen to give the option to register a new user in Drupal. This next call is the user.save service. We are using the hashKey function and using the returned array as our first arguments for the user.save. We are passing the “user.save” string to the hashKey function because we are calling the system.connect method. The session id we retrieved from system.connect will be passed in and a username, password and email which are also required fields for a new user. I have created a new function to handle the registration data called registerUser. You may want some validation on these fields but I am omitting that for length.

public function registerUser():void{
var hashedArray:Array = hashKey("user.save");
var userCreate:Object = new Object();
userCreate.name = usernameInput.text;
userCreate.pass = passwordInput.text;
userCreate.mail = emailInput.text;
userCreate.status = 1;
user.save(hashedArray[0], hashedArray[1], hashedArray[2], hashedArray[3], sessionID, userCreate);
}

I have also set the status on the userCreate so that the user is immediatly set to active. If this is not done admin approval would be required. And like system.connect for anonnymous users to create a new users, permissions will need to be changed (user_service module/create new users). I am choosing to log the user in right after registration but this could be done in a seperate action. Additional requirements for login are the username and password.

public function onUserSaved(event:ResultEvent):void{
var hashedArray:Array = hashKey("user.login");
user.login(hashedArray[0], hashedArray[1], hashedArray[2], hashedArray[3], sessionID, usernameInput.text, passwordInput.text);
}

After we log in I’m going to grab all of the stories that have been created with a view that I have created. This view retrieves the body, title and nid fields from stories.

[Bindable]
private var theStories:Array;

public function onUserLogin(event:ResultEvent):void{
var hashedArray:Array = hashKey("views.get");
views.get(hashedArray[0], hashedArray[1], hashedArray[2], hashedArray[3], sessionID, "getUserStories");
}

public function onViewsResult(event:ResultEvent):void{
theStories = ArrayUtil.toArray(event.result);
}

Again make sure that you have permissions to access views in the user permissions. Make sure the array we are populating is bindable because this array will populate our datagrid. You should now be able to register a new user, auto login and see the all stories in the data grid. If you wanted to log in an existing user you would have to set up another action to do just the login. The last thing we will do with services is change the content of one of the stories or create new stories by doing a node.save. Using the views that pulled in we can modify that information and send it back to Drupal.

public function saveStory():void{
var edit:Object;
if(selectedStories.selectedItem){
edit = selectedStories.selectedItem;
}
else{
edit = new Object;
}
edit.type = "story";
edit.title = titleInput.text;
edit.body = bodyInput.text;
var hashedArray:Array = hashKey("node.save");
node.save(hashedArray[0], hashedArray[1], hashedArray[2], hashedArray[3], sessionID, edit);
}

public function onNodeSaved(event:ResultEvent):void{
var hashedArray:Array = hashKey("views.get");
views.get(hashedArray[0], hashedArray[1], hashedArray[2], hashedArray[3], sessionID, "getUserStories");
}

If there is no story selected a new one will be created and saved to the view. After the node is saved it updates the views list. Give all appropriate permissions for these actions (create and edit).

I hope that this is starting point for expanding and using api keys/sessid in service calls from Flex to Drupal. Comments are welcome and I have included a full copy of the mxml and the as files below.

Source zip