Flash Does.

I’ve developed a Flash shim, which gets access to your webcam, and copies frames to data urls which can be used in HTML DOM images.

This works well, within some limits:

• The performance of the Flash->Javascript interface is adequate for normal use, but drops off as the data being copied across increases in size. This means that you need to have a postage stamp sized (80×60) video in flash to get adequate performance in HTML (~25 fps).
• Chrome (and Safari?) leaks memory when new data URLs are created. (See bug). Apart from that, this technique works on both Chrome & Firefox, but my virtual keypad is Firefox only ATM
• I don’t really know Flash. I downloaded the Flex SDK a few days ago and started hacking, so I’m pretty sure my Flash code sucks.

Here’s a demo. Take a look at the previous post to see how you are supposed to use it. I started work on making a proper Javascript API for it, but yeah… The code is on GitHub, though: http://github.com/nlothian/FlashCamShim

I decided that we need to do that on the web, too, so I build a virtual keypad, using HTML5 video and a webcam.

In HTML5, a common (example 1, example 2) pattern is to take an image from a video, copy it to a canvas and manipulate it here using Javascript. That’s the general pattern I’ve used.

I use a colored marker (setup in a calibration phase) and track its location. In order to take account of brightness changes, etc, I do cosine similarity tests on average color vectors for each “key” on the keypad using the Sylvester Javascript vector library.

I think there’s a fair bit of potential for video based interfaces. I’d like to see a combination of this and the HTML5 drumkit, for example.

You can try it out here: http://demo.nicklothian.com/keypad/. You’ll need to be using Firefox 3.5+.

Here’s a demo video (ironically as an embedded flash video):

Unfortunately, and unlike Flash, there is no way to directly use a webcam with it. However, if you want to experiment with it some VLC+Apache magic can help.

Firstly, you need to setup VLC to do HTTP streaming. These instructions are for windows, but it works on Linux, too (Linux seems a lot less stable. I suspect that is a Firefox problem rather than VLC, though). VLC can be somewhat obtuse to use, so here’s how I got it working

Step 1: Open VLC and select streaming

Step 2: Select Capture Device

Click the “Stream” button.

Step 3: Stream Dialog

Click the “Next” button.

Step 4: Add stream destination

Select “HTTP”, check “Display Locally”, select the “Video – Thora + Vorbis (OGG)” profile and click “Add”

Step 5: Configure the stream

Note that I changed the port from the default of 8080 to 8081. This is optional, but note what port you use. Verify the profile is correct, and click the “Next” button.

Step 6: Modify the stream path

Carefully alter the “Generated stream output string” to include “/stream.ogg” after the port. You need to do this because Firefox uses the file extension to work out how to handle the file (in the absence of correct headers).

Click Stream, and VLC should display a window showing your webcam.

Step 7: Verify streaming is working

Go to http://localhost:8081/stream.ogg in Firefox. Chrome kind of works, too, sometime (I usually get the first frame, and then it stops playing).

Optional Step

This stream can be embedded in a webpage using the URL above. However, if you want to manipulate it in Javascript at all you will quickly run into the same-origin restriction. It’s possible to get around this by using Apache to proxy the stream onto the same site you serve yout pages and javascript from.

Step 8: Configure Apache to Proxy Stream

I use mod_proxy, and add the following:

ProxyPass /stream/stream.ogg http://localhost:8081/stream.ogg

That will allow you to view the stream from http://yourwebserver:port/stream/stream.ogg.

In my next post I aim to show something you can do with this stream.

From the project page:

Chrome-Reload is an extension for the Chrome web browser to allow automatic periodic reloading of a page. It allows the you to configure how often each page reloads and see a count-down until the next load.

This is useful for scenarios such as monitoring constantly changing pages (eg, search results), or for keeping sessions alive in web applications.

I noticed a few weeks ago that Jake Luciani had managed to get Lucene (which Solr uses) working on Cassandra (Facebook’s highly scalable keystore).

The next step had an obvious name: Solandra – Solr running on Cassandra.

Basically there wasn’t too much to getting it going in the limited form it is now – a few minor changes to Jake’s Lucandra code, a custom Solr FieldType (exactly why I needed this I’m unsure) and correctly configured solrconfig.xml and schema.xml files.

I haven’t tested updates, so you’ll probably need Jake’s BookmarkDemo to load data in.

My changes to the Lucandra index reader include hard coding (!) the fields returned by getFieldNames(..) to match the Solr schema and the fields added in the demo.

If anyone is interested, the code is available: solandra.zip. You’ll need to be a Java developer to use it, though.

Doh.

So I get back to work, and talk to Ben-out-local-Javascript-god, who points me at the location-fragment for communication hack (see, for example Cross Domain Frame Communication with Fragment Identifiers (for Comet?)) and away I go.

I quickly discover that all the documentation on that is for use with frames, not windows, and windows work slightly different. For example, the parent window can’t poll the child for it’s location.hash, nor can the child set the parent’s location.hash.

Doh Doh.

Anyway to cut a long story short and to document it for anyone else, you can make it work. The child window needs to contain a iframe from the parent domain, then proxy communication though that. Modern non IE browsers can use Window.postMessage to do the same thing (Actually, IE8 supports Window.postMessage, too, but I couldn’t get it to work properly).

The core of the IE specific code looks like this. The child window uses this object:

/*
  XDCommsChild - Cross Domain Communication, Child Object.

   Create this in the child window with the address of the xdomaincomms.html file on the same domain as the parent window
*/
var XDCommsChild = function(xdomaincommsAddress) {
    this.xdomaincommsAddress = xdomaincommsAddress;

    this.isIe = isInternetExplorer();

    if (isIe) {
        // need hidden frame for communication. IE8 is supposed to support postMessage, but I can't get it to work properly
        document.writeln("")
    }

    this.postBack = function(data) {
        if (isIe) {
            // this method tested against IE6,7 & 8
            window.open(this.xdomaincommsAddress + '#data=' + data, 'xdcomms'); // MUST use window.open. frame.src or frame.location both fail
        } else {
            // for everything else - tested against Firefox 3, Chrome, Safari
            window.opener.postMessage(data, '*'); // should really restrict the domain data can come from here
        }
    }

    this.postBackAndCloseWindow = function(data){
        this.postBack(data);
        setTimeout("window.close()", 200); // need to use a timeout to make sure the javascript in the frame has executed if we are using that communication model
    }
}

Then the included proxy frame (deployed on the parent domain) looks like this:

           var hash = null;

            function sendData() {
                hash = location.hash;
                if (hash != null && hash.length > 1) {
                    window.parent.opener.location.hash=hash;
                    location.hash = '#';
                }
            }

            setInterval("sendData()", 50);

The the parent window uses this object:

/*
  XDCommsParent - Cross Domain Communication, Parent Object.

   Create this in the parent window with a callback function that will recieve data.
*/
var XDCommsParent = function(dataRecievedCallback) {
    this.dataRecievedCallback = dataRecievedCallback;

    this.isIe = isInternetExplorer();

    this.receiveMessage = function(event) {
        dataRecievedCallback(event.data);
    }

    if (this.isIe) {
        window.setInterval("pollForData()", 200);
    } else {
        window.addEventListener("message", this.receiveMessage, false);
    }

}

The final piece of the solution is calling it from the child window. I do this:

<button onclick="javascript:xDCommsChild.postBackAndCloseWindow(document.getElementById('thevalue').value);">Send to Parent Window</button>

I’ve been pretty happy with it except for the infuriating habit Firefox has for “locking up” periodically. The symptoms of this include non-reponsiveness, screen freezing and even the computer being unusable for 30 seconds at a time. The only clue I had was there seemed to be large amounts of disk usage.

After a while bitching and moaning about it I got so annoyed I started looking for a fix.

The first thing I tried was the Chromium Linux nightly builds. Google says “blogging about this isn’t helpful”, so I won’t except to say that I’ve got pretty high expectations of Chrome on Linux and so far I haven’t had to re-apprise that.

All the same, I wanted to fix Firefox. The next thing I tried was moving Firefox cache to a RAM drive. That’s pretty easy – just set it to use a directory under /dev/shm/ for the cache location.

I think that improved the situation marginally, but not enough to call it a fix.

The next thing I tried was to raise a Firefox bug. Somewhat to my surprise that got linked to another bug which was marked as fixed.

The comments on that bug are quite long, but the story is this:

• Firefox uses SQLite as a database for its history and bookmarks.
• SQLite, being a database is very concerned about data integrity, and to implement this it relies on the fsync system call.
• fsync has performance issues on ext3 filesystems. See for example, http://lwn.net/Articles/328363/:
  

  The problem, in short, is this: the ext3 filesystem, when running in the default data=ordered mode, can exhibit lengthy stalls when some process calls fsync() to flush data to disk. This issue most famously manifested itself as the much-lamented Firefox system-freeze problem, but it goes beyond just Firefox.

• SQLite has a no-sync mode, which trades reliability for performance.
• Firefox can use this mode via a config setting.

So the outcome of all that is this:

Create a new config key “toolkit.storage.synchronous” and set it to the integer 0 to stop Firefox lock-ups on Linux (but be aware that there is some chance a power failure could cause loss of your history and/or bookmarks).

AFAIK, no one has posted a list of classes missing. I can’t be bothered doing that either, but I did manually take a look at package level. Here’s it looks like GAE/J is missing:

java.applet
java.awt.*
javax.activation
javax.imageio.*
javax.jws.*
javax.management.*
javax.naming.*
javax.net.*
javax.print.*
javax.rmi.*
javax.sound.*
javax.swing.*
javax.tools
javax.xml.bind.*
javax.xml.crypto.*
javax.xml.soap
javax.xml.stream.*
javax.xml.ws
org.ietf.jgss
org.omg.*

From that list, I’d like to see javax.activation, javax.management and the remaining javax.xml.* and maybe javax.tools packages supported. The rest really don’t seem at all relevant to the AppEngine environment.

Extracting MP3 tags from mp3 file hosted on server using HTTP Range queries.

So I was using Apache Tika for various metadata stuff. I wanted to get the song title for a file hosted on a server, but Tika only supports MP3 ID3v1 metadata, which exists at the end of a file. Downloading an entire MP3 just for the title is wasteful, but fortunatly HTTP Range queries can help us out.

HttpClient httpClient = new HttpClient();
httpClient.getHttpConnectionManager().getParams().setConnectionTimeout(10000);
httpClient.getHttpConnectionManager().getParams().setSoTimeout(10000);

String address = "http://address of mp3 file here";

HttpMethod method = new HeadMethod();
method.setURI(new URI(address,true));

Header contentLengthHeader = null;
Header acceptHeader = null;

httpClient.executeMethod(method);
try {
	//System.out.println(Arrays.toString(method.getResponseHeaders()));
	contentLengthHeader = method.getResponseHeader("Content-Length");
	acceptHeader = method.getResponseHeader("Accept-Ranges");
} finally {
	method.releaseConnection();
}

if ((contentLengthHeader != null) && (acceptHeader != null) && "bytes".equals(acceptHeader.getValue())) {
	long contentLength = Long.parseLong(contentLengthHeader.getValue());
	long metaDataStartRange = contentLength - 128;
	if (metaDataStartRange > 0) {
		method = new GetMethod();
		method.setURI(new URI(address,true));
		method.addRequestHeader("Range", "bytes=" + metaDataStartRange + "-" + contentLength);
		System.out.println(Arrays.toString(method.getRequestHeaders()));
		httpClient.executeMethod(method);
		try {
			Parser parser = new AutoDetectParser();

			Metadata metadata = new Metadata();
			metadata.set(Metadata.RESOURCE_NAME_KEY, address);
			InputStream stream = method.getResponseBodyAsStream();
			try {
				parser.parse(stream, new DefaultHandler(), metadata);
			} catch (Exception e) {
				e.printStackTrace();
			} finally {
				stream.close();
			}
			System.out.println(Arrays.toString(metadata.names()));
			System.out.println("Title: " + metadata.get("title"));
			System.out.println("Author: " + metadata.get("Author"));
		} finally {
			method.releaseConnection();
		}
	}
} else {
	System.err.println("Range not supported. Headers were: ");
	System.err.println(Arrays.toString(method.getResponseHeaders()));
}

The next thing I needed to do was extract song titles from a shoutcast stream. Shoutcast streams are kinda-but-not-quite http. Metadata is embedded in the stream (not as part of the MP3). That makes the code pretty ugly, but whatever… This code will open a connection, read the metadata and close, so you don’t need to keep downloading gigs of data.

URL url = new URL("http://scfire-ntc-aa01.stream.aol.com:80/stream/1074");
URLConnection con = url.openConnection();
con.setRequestProperty("Icy-MetaData", "1");

InputStream stream = con.getInputStream();
try {

	BufferedReader in = new BufferedReader(new InputStreamReader(stream));

	String metaIntervalString = null;
	// get the headers
	StringBuilder headers = new StringBuilder();
	char c;
	while ((c = (char)in.read()) != -1) {
		headers.append(c);
		if (headers.length() > 5 && (headers.substring((headers.length() - 4), headers.length()).equals("\r\n\r\n"))) {
			// end of headers
			break;
		}
	}

	//System.out.println(headers);
	// headers look like this:
	//		ICY 200 OK
	//		icy-notice1: 
This stream requires Winamp

	//		icy-notice2: Firehose Ultravox/SHOUTcast Relay Server/Linux v2.6.0

	//		icy-name: .977 The 80s Channel
	//		icy-genre: 80s Pop Rock
	//		icy-url: http://www.977music.com
	//		content-type: audio/mpeg
	//		icy-pub: 1
	//		icy-metaint: 16384
	//		icy-br: 128

	Pattern p = Pattern.compile("\\r\\n(icy-metaint):\\s*(.*)\\r\\n");
	Matcher m = p.matcher(headers.toString());
	if (m.find()) {
		metaIntervalString = m.group(2);
	}

	if (metaIntervalString != null) {
		int metaInterval = Integer.parseInt(metaIntervalString.trim());

		if (metaInterval > 0) {
			int b;
			int count = 0;
			int metaDataLength = 4080; // 4080 is the max length
			boolean inData = false;
			StringBuilder metaData = new StringBuilder();
			while ((b = stream.read()) != -1) {
				count++;

				if (count == metaInterval + 1) {
					metaDataLength = b * 16;
				}

				if (count > metaInterval + 1 && count < (metaInterval + metaDataLength)) {
					inData = true;

				} else {
					inData = false;
				}

				if (inData) {
					if (b != 0) {
						metaData.append((char)b);
					}
				}

				if (count > (metaInterval + metaDataLength)) {
					break;
				}

			}
			String metaDataString = metaData.toString();
			System.out.println(metaDataString);
		}
	}
} finally {
	stream.close();
}

As they say with open source projects – “it’s done when it’s done”. But nearly 5 years to get to version 1.0 is kind of long.