myAudience promo video

Rhonda Software will participate in Customer Engagement Technology World (CET World) in San Francisco, April 27-28, 2011. We are pleased to invite you to visit our booth #235.

We’ll be glad to have this chance to introduce you our innovative system myAudience – tool for automated audience measurement for digital signage, kiosks, showcases and many others. You can click this special link to register.

Your special PRIORITY CODE will automatically appear with your registration, giving you a FREE exhibits only pass. For up-to-date information about Customer Engagement Technololgy World, please visit www.CETworld.com.

For the first example I used LAME encoder (http://lame.sourceforge.net/) . Why LAME? First of all because it open source and I can recompile it with different compilers and options. In the second place the simplicity of performance measurement. Performance will be a time required to reencode mp3 file.  In the third place it shows well determinate results what allow better understand how different compile options affect speed.

The testing has been performed on computers with different CPUs under Windows operation system.

Intel Pentium 4 3GHz
Intel Core 2 Duo 2.8 GHz
AMD Athlon2x4 (635) 2.9 GHz overclocked to 3.3 GHz
Intel Core i5 (2500) 3.3 GHz

Compilation has been done by VisualStudio9 and GCC4.5.1(using MinGW)

Encoding time has been measured 10 times and average value placed to the table.

As the base 0.00% I used safe options (-O3 -march=prescott -fomit-frame-pointer -mfpmath=sse) that will work on most modern AMD and Intel CPUs. Option -march=core2 may use ssse3 instructions and therefore code may fail to work on AMD and Intel Pentimum 4 family CPUs.

Intel Pentium 4 3GHz

1. Optimization to prescott architecture gives 2% speed increase.
2. –ffast-math gives 2% more
3. Profile guided optimization gives 5% speed increase

Intel Core 2 Duo 2.8 GHz

1. Optimization to core2 architecture gives 10% speed increase.
2. –ffast-math gives only 1% increase
3. Profile guided optimization gives 6% increase

Intel Core i5 (2500) 3.3 GHz

There is no special -march option for core i3,5,7 CPUs.  Option -march=core2 can be used for them.

1. Optimization to core2 architecture gives 9% speed increase.
2. –ffast-math gives  6% increase
3. Profile guided optimization gives 4% increase

AMD Athlon2x4 (635) 2.9 GHz overclocked to 3.3 GHz

1. Optimization to amdfam10 architecture gives 18% speed increase
2. –ffast-math gives 5 %
3. Profile guided optimization gives only 1%.

Total results

Optimization to particular architecture and profile guided optimization may give up to 20 % speed increase.

As I already said  LAME is simple example. Let see how performance options affect real video analytic application .

For the second example I used critical part of real video analytic application (myAudience). It uses boost, opencv and  ffmpeg libraries. Also it runs in several threads. In comparison with LAME encoder performance measurement for this application was not so simple. Moreover because of inaccuracy of measurements in multithreading dynamic enviroment results were not so well determinate. So I have prepared just one table which shows results in general how I understand them.

Compilation has been done by GCC4.5.2 and GCC4.1.2 on CentOS_5.5

Intel Core i5 (2500) 3.3 GHz

What can we conclude after that? Few things

1. GCC4.5.2 little bit faster than GCC4.1.2 plus it allow to use profile guided optimization, and “amdfam10”, “atom” architecture options.
2. Profile guided optimization give about 4% speed increase.
3. –ffast-math gives about 2% speed increase

As you can see, tuning compiler options allows to get real improvement in performance, not so huge sometimes, but almost free, so it should be kept in mind.

1. A set of metrics is prepared. These metrics definitions describe “ground rules”, i.e. how actual logs from the system under test is interpreted, what is considered correct and what is not correct. For example, for people counting metric the following definition may be used:PEOPLE COUNTING
   <visitors_number> total number of visitors in test frame range = Correct + Missing + Unexpected + False
   Real visitors (all found and not found visitors besides False) = Correct + Missing + Unexpected
   Correct: Log visitors associated with test visitors, even if more than one log visitor is associated with one test visitor (only if test visitor had Hard detection status between frames were previous and next log visitor were correlated).
   Missing: Not found in log
   Unexpected: Log visitor is associated with earlier associated test visitor, but there is no Hard visitor detection status between these log visitors.
   False: Log visitor is not associated with any test visitor
2. For each metric a set of KPIs (Key performance indicators) is defined. For example, for people counting metric the following KPIs may be defined:
   # of Total visitors
   # of Real visitors
   # ofFalse visitors
   # of Missing visitors
   False visitor rate (percent)
   Missing visitor rate (percent)
   Counting error rate (percent)
3. Some KPIs may have a goal. For example, we may want to have Counting error rate less than 3%.
4. A set of test videos is created. Typically, there are dozens of videos prepared for the project to cover as many different situations/conditions as possible.
5. Project specific marker tool is used to create a special “markup”, or Meta information describing the objects located at each of input video. Usually, it is an xml file having the same name as an input video file. A specially trained engineer uses this tool to open a video file, go through selected frames and mark objects on them. For instance, for demography detection software, an operator may mark all persons found on a frame, specify coordinates location of their faces and add special attributes for every face like gender, ethnicity or age category. Some objects may be marked as “hard examples”. It specifies that this object is hard to detect/recognized due to different situations, like had motion blur, partially cover by another object, etc. By other words, a hard example is an object that potentially could be detected/recognized by the software, but it is not very likely, so it is ok if the software does not detect/recognize this object. Markup files are stored together with the video test files on a dedicated server for easy access.
6. Software under test is executed in a special logging mode. This mode tells the software to log everything it detects/recognizes and put it into a special (usually xml) log file. A specially prepared set of video files is given to the software version as an input. For each input video file a corresponding xml log is created.
7. Another project-specific tool, comparer is then used to get KPI values. The tool uses prepared markup (see 5) and actual data values from log (see 6). As a result, set of KP values is generated.

Metrics, KPIs and markup are created once per project/video. Testing and metrics calculation can be made for every release to see how this release changed in terms of quality and speed. This allows quickly detecting and fixing possible quality/speed degradations introduced in the release. In addition, when tested periodically, project management can see how the system evolves over the time in terms of quality/performance.
While making video for testing and markup of this video requires people to do, most of other unit testing activities could be performed automatically which helps keeping the quality and performance under control and requires little effort to perform. All of above allows us to develop state of the art, highly competitive software.

The video above explains the process of markering a video file.

Facial landmarks library construction
A set of training images is marked by hand. Coordinates of important facial landmarks (such as lips’ corners, nose tip, etc.) are stored in a database for further processing.
Using information about landmarks position it is easy to geometrically transform and align training images.
Illumination correction is applied to transformed images in order to get rid of shadows and glares and normalize overall exposure.
Gabor jets are then extracted from normalized images in every landmark location. Extracted jets are stored in facial landmarks library for further using during novel images processing.

Building of facial descriptor
Having received a novel facial image we first try to locate approximate eyes position using our hybrid method of Viola-Jones and Bayesian classifier. The purpose why eyes coordinates are required is twofold: they are used to geometrically wrap and align input image and also to get initial estimate of coordinates of other facial landmarks.
Illumination correction and background clipping is then performed.
Starting from approximate eyes position and using samples from facial landmarks library we iteratively find precise locations of all facial landmarks on novel face.
Found landmarks positions are then used to extract Gabor jets and construct informative facial descriptor. Original input image is not required any more after this step.

Facial descriptors matching
The last and the simplest step is matching two facial descriptors which yields a similarity measure between two faces – a real number between 0 (nothing in common) and 1 (complete match). A threshold found by experiment is used to make decision whether the faces belong to the same or different persons.
Descriptors matching is several orders of magnitude faster than descriptor building which makes it possible to match new face against a database of known persons in moderate time.

Some technical details
Working face size: 45×45 px
Descriptor building time: 300ms on P4 1300 MHz (single core).
Descriptors matching: 2ms on P4 1300 MHz (single core).
Recognition rate: Feret fa/fb: 88%, Yale Faces: 86%, Faces in wild: 73%. Both false positives and false negatives considered.

Introduction

HMAX aims to model hierarchical object recognition in cortex. I won’t get into details of HMAX model, just provide some useful links for curious readers:

http://riesenhuberlab.neuro.georgetown.edu/hmax.html

http://cbcl.mit.edu/cbcl/publications/index-pubs.html

My approach in using HMAX for currency recognition was pretty simple.

1. Generate proper С1-feature dictionary
2. Using C1-feature dictionary, generate C2 vectors for images in training dataset
3. Train multi-class SVM classifier using C2 vectors.
4. Test classifier on testing dataset
5. PROFIT

C1 dictionary generation
My initial approach was to generate dictionary automatically using an “interesting point” detector and then generate C-1 patches of size corresponding to the size of detected image structure. Hessian-Laplace (see Mikolajczyk K. and Schmid, C. 2004. ) method seemed to suit well for this task, since it detects characteristic scale of interesting points. My implementation of Hessian-Laplace worked reasonable fine, however number of interesting points it detected was big, so it was pretty hard for my C1 extractor to decide which one of these points was really “interesting” for currency recognition.
As a result – none of my automatically generated dictionaries (which contained about 1000 samples) produce suitable classification outputs.

So I decided to start from smaller dictionary, which would contain hand-picked, really descriptive patches from all kind of bills. For each image in C1 dataset I took 8 patches ( see example of pacthes below). All images in C1 dataset were resized to have width 300 pixels. Size of C1 dictionary now became 88.

Classifier training
The size of my training dataset was 337 images of different dollar bills of all classes (1 to 100) + set of “background” images.

I used LibSVM for classifier training and testing, using RBF and Linear classifier kernels.

To train models with RBF kernels I used script easy.py from LibSVM – this handy script automatically scales training data and the searches for the best C/gamma parameters of the kernel.
Linear classifiers were trained using semi-automated approach – I scaled training data first, then used grid.py to find best value for C parameter of the kernel.

Something like that:
svm-scale -s newdict.range newdict.l > newdict.scale

svm-train -t 0 -c 0.353553390593 newdict.scale newdict.model

And for testing:
svm-scale -r newdict.range newdict.t > newdict.t.scale

svm-predict newdict.t.scale newdict.model newdict.model.predict

Experiment results

Best classification accuracy I’ve reached so far is 88.62% using linear classifier.

Conclusions and future work

HMAX model has proved successful in classification of currency images and seem to have potential for better results.

Possible directions toward better classification could be:
1. Brightness correction. HMAX implementation I used seems to be pretty sensitive to brightness changes. I’ll need to investigate this more deeply and if necessary – normalize trainig/testing images to get rid of brightness-related issues.

2. Quality of C1-dictionary. While my current dictionary proved to be good enough to recognize dollar bill classes, I suppose it could be improved. For example – data there could be more patches per bill class, taken for different layers (bands) of C1 “pyramid”. The size of the features may also vary to achieve better results.

3. HMAX model tuning. Perhaps model modification according to approach of Jim Mutch и David G. Lowe ( PDF) would give better recognition results.

4. Use Ada-Boost in addition to SVM for better model training.

Build platform: Ubuntu 9.10
Target platform: Android

Download and prepare OpenCV library source code.

1. Download the latest version of OpenCV (http://sourceforge.net/project/showfiles.php?group_id=22870).

2. As build platform is Linux, select linux version (for example OpenCV2.1.0.tat.bz).

3. Unpack somewhere to home dir.

Download and prepare cross-compiler

1. Download Android NDK 3 for Linux (http://developer.android.com/sdk/ndk/index.html)

2. Unpack it to ~/android_ndk_3/

3. Then run ~/android_ndk_3/build/host-setup.sh but first fix the error in line 119

Change

if [ "$result" = "Pass" ] ; then

to

if [ "$result" == "Pass" ] ; then

4. Do/install whatever needed to let host-setup.sh complete successful.

Create NDK project/Modify Makefiles

There is one big issue with NKD toolchein. It has trimmed stdc library which does not contain STL. Because of that some files (like cvkdtree.cpp in cv) can not be compiled since they use vector, list and other stuff. The solution is to compile STL from source code. In my OpenCV NDK project I used STL sources from uClibc (http://www.uclibc.org).

The simpliest way to start your OpenCV NDK project is to update hello-jni project with OpenCV source files.

The ~/android_ndk_3/apps/hello-jni/project/jni folder of hello-jni project may look like this

cv
- hdr
- src
cvaux
- hdr
- src
cxcore
- hdr
- src
stl
- hdr
- src
Android.mk
hello-jni.c

The ~/android_ndk_3/apps/hello-jni/project/jni/Android.mk may looks like this

APPS_PATH := $(call my-dir)############################
# stl
############################
include $(CLEAR_VARS)LOCAL_PATH := $(APPS_PATH)/stl/src
LOCAL_C_INCLUDES := $(APPS_PATH)/stl/hdrLOCAL_MODULE := stl
LOCAL_SRC_FILES := string.cpp algorithm.cpp char_traits.cpp iterator.cpp limits.cpp list.cpp vector.cppinclude $(BUILD_STATIC_LIBRARY)############################
# cxcore
############################
include $(CLEAR_VARS)LOCAL_PATH := $(APPS_PATH)/cxcore/src
LOCAL_C_INCLUDES := $(APPS_PATH)/cxcore/hdrLOCAL_CXXFLAGS := -DHAVE_CONFIG_HLOCAL_MODULE := cxcore
LOCAL_SRC_FILES := cxalloc.cpp cxarithm.cpp cxarray.cpp cxcmp.cpp cxconvert.cpp cxcopy.cpp cxdatastructs.cpp cxdrawing.cpp cxdxt.cpp cxerror.cpp cximage.cpp cxjacobieigens.cpp cxlogic.cpp cxlut.cpp cxmathfuncs.cpp cxmatmul.cpp cxmatrix.cpp cxmean.cpp cxmeansdv.cpp cxminmaxloc.cpp cxnorm.cpp cxouttext.cpp cxpersistence.cpp cxprecomp.cpp cxrand.cpp cxsumpixels.cpp cxsvd.cpp cxswitcher.cpp cxtables.cpp cxutils.cpp dummy.cppinclude $(BUILD_STATIC_LIBRARY)############################
# cv
############################
include $(CLEAR_VARS)

LOCAL_PATH := $(APPS_PATH)/cv/src
LOCAL_C_INCLUDES := $(APPS_PATH)/cv/hdr $(APPS_PATH)/cxcore/hdr $(APPS_PATH)/stl/hdr

LOCAL_MODULE := cv
LOCAL_SRC_FILES := cvkdtree.cpp cvaccum.cpp cvadapthresh.cpp cvapprox.cpp cvcalccontrasthistogram.cpp cvcalcimagehomography.cpp cvcalibinit.cpp cvcalibration.cpp cvcamshift.cpp cvcanny.cpp cvcolor.cpp cvcondens.cpp cvcontours.cpp cvcontourtree.cpp cvconvhull.cpp cvcorner.cpp cvcornersubpix.cpp cvderiv.cpp cvdistransform.cpp cvdominants.cpp cvemd.cpp cvfeatureselect.cpp cvfilter.cpp cvfloodfill.cpp cvfundam.cpp cvgeometry.cpp cvhaar.cpp cvhistogram.cpp cvhough.cpp cvimgwarp.cpp cvinpaint.cpp cvkalman.cpp cvlinefit.cpp cvlkpyramid.cpp cvmatchcontours.cpp cvmoments.cpp cvmorph.cpp cvmotempl.cpp cvoptflowbm.cpp cvoptflowhs.cpp cvoptflowlk.cpp cvpgh.cpp cvposit.cpp cvprecomp.cpp cvpyramids.cpp cvpyrsegmentation.cpp cvrotcalipers.cpp cvsamplers.cpp cvsegmentation.cpp cvshapedescr.cpp cvsmooth.cpp cvsnakes.cpp cvstereobm.cpp cvstereogc.cpp cvsubdivision2d.cpp cvsumpixels.cpp cvsurf.cpp cvswitcher.cpp cvtables.cpp cvtemplmatch.cpp cvthresh.cpp cvundistort.cpp cvutils.cpp dummy.cpp

LOCAL_STATIC_LIBRARIES := cxcore stl

include $(BUILD_STATIC_LIBRARY)

############################
# cvaux
############################
include $(CLEAR_VARS)

LOCAL_PATH := $(APPS_PATH)/cvaux/src
LOCAL_C_INCLUDES := $(APPS_PATH)/cvaux/hdr $(APPS_PATH)/cv/hdr $(APPS_PATH)/cv/src $(APPS_PATH)/cxcore/hdr $(APPS_PATH)/stl/hdr

LOCAL_MODULE := cvaux
LOCAL_SRC_FILES := camshift.cpp cvaux.cpp cvauxutils.cpp cvbgfg_acmmm2003.cpp cvbgfg_codebook.cpp cvbgfg_common.cpp cvbgfg_gaussmix.cpp cvcalibfilter.cpp cvclique.cpp cvcorrespond.cpp cvcorrimages.cpp cvcreatehandmask.cpp cvdpstereo.cpp cveigenobjects.cpp cvepilines.cpp cvface.cpp cvfacedetection.cpp cvfacetemplate.cpp cvfindface.cpp cvfindhandregion.cpp cvhmm.cpp cvhmm1d.cpp cvhmmobs.cpp cvlcm.cpp cvlee.cpp cvlevmar.cpp cvlevmarprojbandle.cpp cvlevmartrif.cpp cvlines.cpp cvlmeds.cpp cvmat.cpp cvmorphcontours.cpp cvmorphing.cpp cvprewarp.cpp cvscanlines.cpp cvsegment.cpp cvsubdiv2.cpp cvtexture.cpp cvtrifocal.cpp cvvecfacetracking.cpp cvvideo.cpp decomppoly.cpp dummy.cpp enmin.cpp extendededges.cpp precomp.cpp vs/bgfg_estimation.cpp vs/blobtrackanalysis.cpp vs/blobtrackanalysishist.cpp vs/blobtrackanalysisior.cpp vs/blobtrackanalysistrackdist.cpp vs/blobtrackgen1.cpp vs/blobtrackgenyml.cpp vs/blobtrackingauto.cpp vs/blobtrackingcc.cpp vs/blobtrackingccwithcr.cpp vs/blobtrackingkalman.cpp vs/blobtrackinglist.cpp vs/blobtrackingmsfg.cpp vs/blobtrackingmsfgs.cpp vs/blobtrackpostprockalman.cpp vs/blobtrackpostproclinear.cpp vs/blobtrackpostproclist.cpp vs/enteringblobdetection.cpp vs/enteringblobdetectionreal.cpp vs/testseq.cpp

# failed to compile
#cv3dtracker.cpp

LOCAL_STATIC_LIBRARIES := cv cxcore stl

include $(BUILD_STATIC_LIBRARY)

The ~/android_ndk_3/apps/hello-jni/Application.mk file needs to be updated as follows

APP_PROJECT_PATH := $(call my-dir)/project
APP_MODULES      := stl cxcore cv cvaux hello-jni

To build the project go to ~/android_ndk_3 and type

make APP=hello-jni

Of course there will be compile issues. Understand and fix them. Easiest cases are related to syntax mismatch between different compilers. In more complicated cases some code should be commented out. For example usage of libs with optimizations for Intel processor is not needed for ARM.

HighGui is also can be built but only partially. Simply remove files that causing problems from Android.mk. In my case the rest of files were enough to use cvLoadImage function for bmp file.

Running facedetect openCV example

There is no way to run native C code as separate application on Android. Instead native C functions can be called from Java apps. Because of that I made native function FaceDetect using OpenCV example application facedetect.c.

The declaration looks like this

void
Java_com_example_hellojni_HelloJni_FaceDetect( JNIEnv* env, jobject thiz, jbyteArray jyuv_buff, int w, int h, jbyteArray jbgra_buff)
{
…
}

This function takes YUV_NV21 buffer (preview from camera captured by Java app), converts it to BGRA8888, searches the faces, draws circles around the faces and returns updated RGBA8888 buffer back to Java app. Java app can draw it on the screen.

people_counter.avi

Marketing researches are area where required to analyze a lot of data. E.g. we want to understand how many people are visiting a bank. In order to count this value, we need to count each man or woman which are entering to or exiting from the bank. For resolving this task there are a lot of approaches: e.g. use special gate with laser or mechanical counter. Though there are people counting tasks where such approaches cannot work or too unuseful. E.g. barrier cannot be used where people flow is very high, and laser counters have limitations as well.

Opposite the approaches above, we found papers where top-mounted camera is used for resolving the people counting task.

 The fact is that most of organization have own IP or CCTV camera based security infrastructure. And also often there is a camera which is already top-mounted. Thus top-mounted camera counting approach is looked very perspective from reusing infrastructure point of view.

We researched a lot of approaches. There are a lot of ready methods for people counting with top-mounted camera. But such methods either are patented or don’t meet our expectation in quality or speed. Thus we developed own method (see demo video).

Method counts human each time they cross a predefined counting line (that is why it is often called as “line-crossing”). The assumption is that the line should be selected orthogonally to the main people flow.

Now we skip implementation details of the method. If shortly, our method is a real time 15 fps+, we tested it usual USB or IP cameras. We tested our method in indoor and outdoor use-case and the quality is 80-90% (dependent on environment condition).

Our method can be used in other applications, e.g. vehicle counting, you could look at the video how it works. Have a nice watching…

tools.avi

This object recognition algorithm is based on own pattern-matching algorithm. The algorithm is able to recognize pre-trained objects which are defined with special set of templates. Theoretically, the algorithm works with any “3D” objects which have good projection on 2D coordinates. However, natural 3D objects are covered by few templates for set of 2D projections. For limited number of object templates, the algorithm works in real-time on PC (Intel P4 3.0 GHz) .

The currency recognition demo application works under Windows XP, Intel P4 3GHz. Quality of recognition: 85%. The solution is cross-platform. The application was tested on Linux, ARM11 and on Linux/Windows, Intel Atom.

Interesting details can be found at: http://www.computer-vision-software.com/blog/2009/10/barcode