As of late 2011 I started work on StrongSteam with my co-founder of ShowMeDo (Kyran Dale). We’re building a web based API that makes it easy to find things in images – things like text, objects and people. Our first APIs will focus on text and object matching.

All going well I’ll be demonstrating some of these ideas at PyCon 2012 in Santa Clara and our first iPhone app is in production, slated for release in April. The iPhone app will read Latin plant labels at botanical gardens and give you information from WikiPedia, GeoSpecies and BBC:Wildlife along with pictures and maybe video. We want to expand this app to work at museums too.

I’ll do another post here once we’re live at StrongSteam and then this Cookbook will enter maintenance mode.

The book is a cookbook with a huge set of recipes for NLTK. It makes for a great companion for the original O’Reilly NLTK book. Rather than repost it I’ll simply direct you back to my blog – my post has links for all the major topics and tools.

The wiki guide to use cmake didn’t quite work, I went ahead with ccmake (the command-line GUI equivalent):

ccmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/home/ianozsvald -D BUILD_PYTHON_SUPPORT=ON ..

Next I moved through the menu to set:

• BUILD_EXAMPLES ON
• INSTALL_C_EXAMPLES ON
• INSTALL_PYTHON_EXAMPLES ON

Next I used ‘t’ to turn on advanced options and configued:

• PYTHON_EXECUTABLE=/usr/local/bin/python2.5
• PYTHON_INCLUDE_DIR=/usr/local/include/python2.5
• PYTHON_LIBRARY=/usr/local/lib/libpython2.5.so

To finish I used ‘c’ to create settings and ‘g’ to generate setup files. Now ‘make’ and ‘make install’ did their job (it took 10 minutes to compile). In ‘~/bin’ I saw ‘opencv_createsamples’ which shows that openCV built ok.

Finally I copied the build directory’s lib/cv.so to ‘~/lib/’ and then in python2.5 I tried ‘import cv’ and it worked. In the build directory’s ‘~/bin’ I ran ‘./cxcoretest’ and all tests passed.

A few days back Jonathan Street announced his entry in the challenge’s thread – he’d beaten my initial average error of 709 (it was in part designed to be easy to beat!) and quickly brought it down to 33.4. Jonathan becomes the winner of the A.I.Cookbook’s first challenge, the challenge now rolls on to this month and the same prize is offered.

I’ll be presenting the results at the open day for the Open Plaques project (sponsored by the Royal Society of the Arts) on 25th September and I hope to be able to demonstrate that, for a few plaques at least, we can automatically get a good transcription.

In Jonathan’s write-up he describes the main steps and includes full Python source:

• image pre-processing to find the blue regions
• restricting tesseract’s character set
• spell checking
• word clean-up (to fix things like dates)

He’s taken some of the ideas I listed in the wiki and taken them further – I’m particularly happy with the blue region detection as that felt like an obvious first step that I hadn’t attempted.

In the thread there’s also a note by Andrew Elwell – he’s using OCR to update @lhcstatus (for the Large Hadron Collider – with >1,000 followers!) by screen scraping their graphical update screen.

David Rawlinson also posted in the thread about some ideas taken from his experience with automatic number plate recognition (ANPR) so we can correct mis-recognised characters (e.g. 0O0 and 1lLiI are easily mis-recognised by OCR!).

The competition runs on, the new deadline is Thursday September 23rd so I can present our progress on the 25th at the OpenPlaques event. If nobody beats Jonathan’s result by then then he becomes the winner by default. I’ll be adding more ideas for improving the result into the main wiki page. Join the Google Group if you’d like to offer ideas and get involved.

The text is very easy for a human to read but very hard for a computer to extract. Thankfully there’s a great open-source OCR system (tesseract) that was released by HP several years back.

The goal of this project is to automatically transcribe the text from several thousand plaques (taken using different cameras and phones in varying lighting conditions at various angles and distances) so that the human sysops don’t have to do the transcription work by hand!

Previously I’ve posted about my work in progress with a manual process, now I’m building towards an automated solution, progress is outlined in the wiki as Automatic Plaque Recognition.

Currently there’s a Python demo file which retrieves three example plaques  from flickr, passes them through tesseract and then uses the Levenshtein distance as an error metric against a manually transcribed string.

Once the OpenPlaques team puts together a larger test and validation set I’ll setup a monthly challenge. The challenge will have a cash prize, the goal will be to encourage entrants to write better recognition systems each month up until we can run an automatic algorithm against the entire OpenPlaques corpus.

If you’re interested in getting involved please join the A.I. Cookbook Google Group.

Next step:

The following text will give you more detail on OCR techniques.

This is what the finished system will look like:

An earlier demo was presented on my blog as Headroid1 – A Face Tracking Robot, here’s a video demo:

Requirements:

• An afternoon with some tools and Python
• pySerial, openCV with Python wrappers
• Webcam
• 2 servos (if you want the head to move) and some brackets
• Serial Servo Controller or Arduino (if you want to control your servos)

First – let Python see faces using OpenCV:

My earlier facedetect.py post shows you how well facial detection works, it includes links to get OpenCV (which includes the Python bindings).  It’ll take 30 minutes to download and compile OpenCV. To get facial detection working just plug in a webcam and run:

cd OpenCV-2.1.0/samples/python
python facedetect.py 0 # pass in id for webcam - 0 is first webcam

Now you’ll have a red rectangle around your face as long as you’re looking roughly towards the webcam. First step complete!

Second – figure out how far the face is from the centre of the screen

Having found a face we now need to determine how far it is from the centre of the image. We edit detect_and_draw(…) in facedetect.py to add the following lines:

centre = None
...
if faces:
    for ((x, y, w, h), n) in faces:
        # the input to cv.HaarDetectObjects was resized, so scale the
        # bounding box of each face and convert it to two CvPoints
        pt1 = (int(x * image_scale), int(y * image_scale))
        pt2 = (int((x + w) * image_scale), int((y + h) * image_scale))
        cv.Rectangle(img, pt1, pt2, cv.RGB(255, 0, 0), 3, 8, 0)
        # ADD THESE LINES BELOW
        # get the xy corner co-ords, calc the centre location
        x1 = pt1[0]
        x2 = pt2[0]
        y1 = pt1[1]
        y2 = pt2[1]
        centrex = x1+((x2-x1)/2)
        centrey = y1+((y2-y1)/2)
        centre = (centrex, centrey)
...
return centre

So as long as we find a face, we know where the centre of the facial rectangle is inside the webcam image. We’ll also know how big the webcam’s image is, so we know which quadrant of the webcam the face is in – and from here we’ll know which direction to move.

Third – how far should we move the servos?

Next we want to know how far to move the x and y servo locations to bring the face closer to the centre of the webcam image. We can’t just change the position in one big jump – the whole assembly will rock due to the sudden movement and then the webcam’s image rocks creating odd oscillations. Instead we move in short, stable steps.

We’ll call the following routine twice, once for the x axis and once for the y axis. We’ll allow the x axis to move up to 4 degrees each iteration, the y axis can only move a maximum of 1 degree per iteration (mechanically one degree on the y axis is lots, 4 degrees on the x axis with my webcam isn’t much). These figures will need tuning depending on your setup.

def get_delta(loc, span, max_delta, centre_tolerance):
    """How far do we move on this axis to get the webcam
       centred on the face?
       loc is the face's centre for this axis
       span is the width or height for this axis
       max_delta is the max nbr of degrees to move on this axis
       centre_tolerance is the centre region where we don't allow movement
       """
    framecentre = span/2
    delta = framecentre - loc
    if abs(delta) < centre_tolerance: # within X pixels of the centre
        delta = 0 # so don't move - else we get weird oscillations
    else:
        # the x-axis is reversed so we must remember the sign
        is_neg = delta <= 0
        to_get_near_centre = abs(delta) - centre_tolerance
        if to_get_near_centre > 35:
            delta = 4 # big movement allowed if we're far away
        else:
            delta = 1 # small movement if we're close to the centre
        if is_neg:
            delta = delta * -1
    return delta

Fourth – move the servos to re-centre the webcam using pySerial

Finally we need to control our servos so they respond to the deltas we’ve calculated. I’m using the pySerial module and BotBuilder‘s Serial Servo Board. The servo board is based on an Arduino – if you have an Arduino then these servo links will give you an easy equivalent (I’d love to see new code if you have a working Arduino solution!).

You’ll also need some brackets to mount your servos, see the end for purchase details to get an assembly like this (a USB->Serial cable is also shown to drive the serial board):

This assembly doesn’t show the webcam – we’ll add it back shortly. To control the servo board we open a connection using:

import serial
# /dev/cu.usbserial is the serial port on a Mac, it'll be COMx on Windows
# The Serial Servo Board uses 19200 baud
ser=serial.Serial(port='/dev/cu.usbserial',baudrate=19200,timeout=0)
ser.write('r') # send reset command
ser.read(100) # receive 'ready' string back

and to move the servos we simply specify the angle for the servo, e.g.:

ser.write('20a') # send servo on connection A to 20 degrees
ser.write('40a40b') # move servos on connections A and B to 40 degrees

To get an idea of how quickly the servos move watch this 30 second video:

Now we add the webcam using a bracket to complete the hardware:

Here’s the final head assembly:

Down below you’ll find the complete source code.

Questions? Join the A.I. Cookbook’s Google Group and see more details in the Cookbook wiki.

Purchase? If you’re interested in buying a hardward kit then email: kits AT aicookbook.com. We don’t have kits yet but if there’s interest, we’ll put them together via botbuilder.co.uk.

Moving forwards

The openCV book is rather excellent, everything is in C++ but the Python API is easy to figure out and the reference text makes it all clear.

Next? I’m glad you asked – here’s Headroid2 with a smiley-inspired emoticon interface from a recent A.I. talk I gave. Instructions for adding an Arduino+LOLShield for emotional feedback will follow.

Full source:

#!/usr/bin/python
"""
This program is demonstration for face and object detection using haar-like features.
The program finds faces in a camera image or video stream and displays a red box around them,
then centres the webcam via two servos so the face is at the centre of the screen
Based on facedetect.py in the OpenCV samples directory
"""
import sys
from optparse import OptionParser
import time
import math
import datetime
import serial
import cv

# Parameters for haar detection
# From the API:
# The default parameters (scale_factor=2, min_neighbors=3, flags=0) are tuned
# for accurate yet slow object detection. For a faster operation on real video
# images the settings are:
# scale_factor=1.2, min_neighbors=2, flags=CV_HAAR_DO_CANNY_PRUNING,
# min_size=<minimum possible face size

min_size = (20, 20)
image_scale = 2
haar_scale = 1.2
min_neighbors = 2
haar_flags = 0

def detect_and_draw(img, cascade):
    gray = cv.CreateImage((img.width,img.height), 8, 1)
    small_img = cv.CreateImage((cv.Round(img.width / image_scale),
			       cv.Round (img.height / image_scale)), 8, 1)

    # convert color input image to grayscale
    cv.CvtColor(img, gray, cv.CV_BGR2GRAY)

    # scale input image for faster processing
    cv.Resize(gray, small_img, cv.CV_INTER_LINEAR)

    cv.EqualizeHist(small_img, small_img)

    centre = None

    if(cascade):
        t = cv.GetTickCount()
        # HaarDetectObjects takes 0.02s
        faces = cv.HaarDetectObjects(small_img, cascade, cv.CreateMemStorage(0),
                                     haar_scale, min_neighbors, haar_flags, min_size)
        t = cv.GetTickCount() - t
        if faces:
            for ((x, y, w, h), n) in faces:
                # the input to cv.HaarDetectObjects was resized, so scale the
                # bounding box of each face and convert it to two CvPoints
                pt1 = (int(x * image_scale), int(y * image_scale))
                pt2 = (int((x + w) * image_scale), int((y + h) * image_scale))
                cv.Rectangle(img, pt1, pt2, cv.RGB(255, 0, 0), 3, 8, 0)
                # get the xy corner co-ords, calc the centre location
                x1 = pt1[0]
                x2 = pt2[0]
                y1 = pt1[1]
                y2 = pt2[1]
                centrex = x1+((x2-x1)/2)
                centrey = y1+((y2-y1)/2)
                centre = (centrex, centrey)

    cv.ShowImage("result", img)
    return centre

def move_servos(xygo):
    position = '%da%db' % (xygo[0], xygo[1])
    ser.write(position)

def get_delta(loc, span, max_delta, centre_tolerance):
    """How far do we move on this axis to get the webcam
       centred on the face?
       loc is the face's centre for this axis
       span is the width or height for this axis
       max_delta is the max nbr of degrees to move on this axis
       centre_tolerance is the centre region where we don't allow movement
       """
    framecentre = span/2
    delta = framecentre - loc
    if abs(delta) < centre_tolerance: # within X pixels of the centre
        delta = 0 # so don't move - else we get weird oscillations
    else:
        is_neg = delta <= 0
        to_get_near_centre = abs(delta) - centre_tolerance
        if to_get_near_centre > 35:
            delta = 4
        else:
            # move slower if we're closer to centre
            if to_get_near_centre > 25:
                delta = 3
            else:
                # move real slow if we're very near centre
                delta = 1
        if is_neg:
            delta = delta * -1
    return delta

if __name__ == '__main__':
    # open a serial port
    ser=serial.Serial(port='/dev/cu.usbserial',baudrate=19200,timeout=0)
    ser.write('r')
    xygo = (90,90)
    move_servos(xygo)

    # parse cmd line options, setup Haar classifier
    parser = OptionParser(usage = "usage: %prog [options] [camera_index]")
    parser.add_option("-c", "--cascade", action="store", dest="cascade", type="str", help="Haar cascade file, default %default", default = "/Users/ian/Documents/OpenCV-2.1.0//data/haarcascades/haarcascade_frontalface_alt.xml")
    (options, args) = parser.parse_args()

    cascade = cv.Load(options.cascade)

    if len(args) != 1:
        parser.print_help()
        sys.exit(1)

    input_name = args[0]
    if input_name.isdigit():
        capture = cv.CreateCameraCapture(int(input_name))
    else:
        print "We need a camera input! Specify camera index e.g. 0"
        sys.exit(0)

    cv.NamedWindow("result", 1)

    if capture:
        frame_copy = None

        while True:
            frame = cv.QueryFrame(capture)
            if not frame:
                cv.WaitKey(0)
                break
            if not frame_copy:
                frame_copy = cv.CreateImage((frame.width,frame.height),
                                            cv.IPL_DEPTH_8U, frame.nChannels)
            if frame.origin == cv.IPL_ORIGIN_TL:
                cv.Copy(frame, frame_copy)
            else:
                cv.Flip(frame, frame_copy, 0)

            centre = detect_and_draw(frame_copy, cascade)

            if centre is not None:
                cx = centre[0]
                cy = centre[1]

                # modify the *-1 if your x or y directions are reversed!
                xdelta = get_delta(cx, frame_copy.width, 6, 15) * -1
                ydelta = get_delta(cy, frame_copy.height, 1, 25) * -1

                # on my camera I introduce a delay after movements
                # else my assembly wobbles and the webcam transmits
                # a non-centred image, so weird oscillations can occur
                total_delta = abs(xdelta)+abs(ydelta)
                if total_delta > 0:
                    xygo = (xygo[0]+xdelta,xygo[1]+ydelta)

                    sleep_for = 1/10.0*min(total_delta, 10)
                    sleep_for = min(sleep_for, 0.4)

                    move_servos(xygo)
                else:
                    sleep_for = 0

            if cv.WaitKey(10) >= 0: # 10ms delay
                break

    cv.DestroyWindow("result")

pyOpenCV is the Python binding to the open source openCV (originally created by Intel for vision research). It comes as a Windows exe, it builds on Linux just fine, I posted some Mac openCV build notes a few days back.

facedetect.py is the demo program for facial detection. The goal of the video below is to show you that:

• facedetect.py is pretty good at facial detection on a standard laptop
• light sources really degrade the performance

Books:

The openCV book is rather excellent, everything is in C++ but the Python API is easy to figure out and the reference text makes it all clear.

The immediate goal is to let the OpenPlaques folk have an automatic service which machine-reads English Heritage Plaques (blue plaques – very common at historic sites in the UK) from their flickr photos and then squirt out the English text. Currently volunteers are transcribing the text by hand.

Below you’ll see a quick demo, I’ve used the bottle.py microframework to run my webservice, it takes a URL to an image, converts it to a TIFF image, passes it into Tesseract and presents the recognised text as a text output.

This isn’t live on the web yet (it needs a bit more work) but shortly it’ll be up for public use.

Update – following this tesseract image clean-up advice (isolate text region, threshold, convert to b&w) I can extract very clean text – contrast these results with what you see in the video.

IN
THIS HOUSE
LIVED
RALPH ELLIS
1885 – 1963
ARTIST
PAINTER & DESIGNER
OF
INN SIGNS (Note – I extracted the inner circle so Sussex isn’t shown)

THIS WALKWAY
WAS DONATED BY
JEAN & BRIAN CROSSLEY
OP BROCKHAM (Note – 1 typo here with OP)
TO CELEBRATE
JEAN’S 80th BIRTHDAY
DECEMBER 27
2007

The face tracking technology (SeeingMachines faceAPI) is also very nice – it tracks faces when they turn a long way away from the camera (at 0:30). The speakers states that the speech recognition tool is Microsoft’s default for Win Vista/7.

The query template appears to be “Search X for Y please”, the Search, For and Please all need to be present for the search to trigger.

A goal of the research seems to be to sell more Intel many-core CPUs – the point is made that face tracking and speech recognition are parallelisable tasks which work well across several cores.

Sidenote – if you prefer podcasts then John was recently interviewed by Ron Stephens on Python411 (with a lovely plug for ShowMeDo – cheers John!). The interview runs for an hour and is completely about Open Allure.

The project aims to build a speech and gesture recognising interface, mostly using Python, to provide us with new ways of interacting with the computer. As John states:

Open Allure is a project aimed at developing new ways to share what we know with one another by permitting the collaborative creation and experience of interactive dialogs.

These verbal exchanges give your interaction with the computer a very different quality and permit immediate feedback to help reinforce or reorganize your thinking.

Because voice recognition is still imperfect, the interface also supports making choices by gesture: your webcam watches you as you raise your hand.

Here’s a good overview video:

And here’s a demo showing speech and gesture recognition:

Here John shows the start of a chatbot interface – you can verbally ask the computer to solve math problems and to look-up word definitions:

I particularly like the direction of that latest (0.1d7) release – John has integrated a persona (shown as photos of John in the top-right corner) to visually show the state of the machine. The persona indicates if the machine is listening or responding, this feedback is very common in human-to-human interaction:

Gesture recognition uses a ‘green screen’ approach. The system records a set of images to generate an average background, it then subtracts (using a grey scale image) each new image from the average and applies a threshold. Movement shows up as regions of activity, if these regions occupy a region of interest then an event is generated:

Currently John has the best support for Open Allure on Windows using the Microsoft Speech API (support for Dragon Naturally Speaking is just starting). On the Mac and Linux there is no speech recognition yet. pyGame is used for the vision components. This Google Sheet lists the status of the components for each platform.

John has tested the open source Julius continuous speech recogniser on Linux (shown below) but it isn’t yet integrated into Open Allure:

Do you want to try it yourself? If you have Windows then you should be all set, Linux and Mac have partial support. To see what you need visit the PyPI page.

If you want to contribute then see this list of easy improvements that need to be made and join the project’s Google Group.