These days online discussion grows regarding Chinese government’s deployment of Green Dam Censorware System, a piece of software aiming at blocking adult content or political content-sensitive websites. It is mandated that every computer that will be sold in China after July 1st, 2009 needs to install this software.

The part I am interested in is its pornographic image recognition engine, developed by Jinhui Technologies in Zhengzhou. Pornographic image identification is an interest computer vision application. How does this work? 

Intuitively, the Pornographic image filter detects large area of skin area based on image color information, and then removes images with large face regions using OpenCV’s face detector. More details are disclosed in “A Technical Analysis of the ’Green Dam-Youth Escort’ Software”,

The process of image detection begins when visual data is obtaining as the image is in queue to be screened, first normalizing the image’s size, then separating areas of skin tone from those without skin tone; analysis of the relationship between areas of skin tone is followed by removal of noises, then extraction of the area’s characteristics, which are then input into a trained SVM (note: Support Vector Machine) classifier. Once the image has been deemed pornographic it is sent to a human face detector; if a human face is not the primary component, the image is then classified as pornography.

The method of detecting pornographic image based on color and face detector is efficient and effective. There are related patent and technical paper along this line. Chinese government nowadays has a more strict regulations on intellectual properties of software and patents. I believe that the Jinhui company has disclosed the potential IP issues to Chinese government, or worked out a workaround.

One voice rises because Green Dam’s use of OpenCV prior to version 3.174 did not include the required license. According to a study by University of Michigan, this problem was addressed in the 3.174 filter update by adding the required OpenCV license.

This ”green” activity will have great social benefits to Chinese youth.

SIGGRAPH 2009 accepted papers are coming out. Totally 78 papers out of 439 submissions are accepted (acceptance rate: ~18%). Congratulations to the authors. Many researchers can not stand waiting long time to post their works online. Thanks to Ke-Sen Huang, he maintains a very nice website of accepted papers.

There are several interesting observations:

1.      Computer vision and image processing related papers are a big trunk of the premium graphics conference (~1/3) as usual. This shows the trend of the convergence of computer vision and computer graphics, due to extensive applications in image/video editing, computational photography, media retargeting, virtual reality, visual search, and capture from reality.

2.      Adobe Advanced Technology Labs is the big winner (17 out of 78 accepted papers coming from Adobe researchers (~22%).) Its recent aggressive recruiting of top talents is paid off. In contrast, the winner in previous SIGGRAPH, Microsoft research (18 papers in 2008) has only 7 papers, and Mitsubishi Electric Research Lab (MERL) has only 1 paper. The recent turmoil in MERL (1, 2) may explain the reason. It caused the leave of many computer graphics researchers for MIT or Adobe. Another interesting observation is that Harry Shum has only one paper this year. This paper is on visual search. This may be due to his recent transition to a senior management position (a Corporate Vice President at Microsoft, leading the Core Search Development of Microsoft.)

3.      Massachusetts Institute of Technology (MIT, 11 papers) and Columbia University (9 papers) take their leading position in university research. This year there is increasing number of papers from Israel (Tel-Aviv University has 4, and Hebrew University of Jerusalem has 3). The statistics of other strong groups in the field:

• Princeton University, Georgia Institute of Technology (GIT), University of California at Berkeley – 6 papers
• University of Southern California, Stanford University – 4 papers
• California Institute of Technology (CIT) – 3 papers
• University of Illinois at Urbana-Champaign (UIUC), Carnegie Mellon University (CMU), University of Washington – 2 papers

These two pictures were taken on the roadside of the Strip Street in Las Vegas from my recent trip. The first picture was painted with less than 10 minutes by the artist in the second picture. Very impressive! 

Could computer vision technology be used for creating digital arts? 

Recently I came across an article discussing how a camera is used to assist creating digital paining. The following YouTube video shows an interesting demo.

Colorful objects are tracked to draw the corresponding color in the demo. Simple computer vision techniques are used, such as object tracking and color recognition. Obviously there is still a big gap to create a picture similar to the one we just showed. However it is an interesting application of computer vision technologies.

Let’s review the different stages of image search:

1. Image-search-by-text-context: The first stage of image search is purely text based. The images with the same keywords in the context are extracted. This is still the most popular approach by almost all big image search players, such as Yahoo, Microsoft and Google. The advantages and disadvantages of image search by text context were discussed in a previous article.

2. Image-search-by-image-content: The content of an image is explicitly indexed by color, shape, texture, style, objects, etc., as described in our previous posts, Visual Search Engines: The Future Search Engines and Google and Microsoft Image Search by Content. One successful application of image search by image content is like.com. 

3. Image-search-by-image-similarity: Image search by image similarity is basically image matching across different scale, illumination conditions, viewpoint changes or object occlusion. Some scale-invariant or affine-invariant features could be used to index images. Google similar image search uses text context to find overcomplete images, and then each image is associated with similar images. Tineye allows users to search similar images based on image examples. 

What is the next for image search? Will similar technologies be used for video search? Your comments are welcome.

In contrast to traditional video analytics software, such as ObjectVideo , ioimage , Lenel , BRS does not need users to draw tripwire or region of interest (ROI) and define the rules. It uses a machine learning based approach to adaptively learn what behaviors or activities are "normal" in the scene, and issues alarms when finding some anomalies. 

As described in their website,

 Unlike rules-based video-analytics software, the intelligent AISight solution identifies threats and behaviors that were not previously defined or anticipated, and it does so with fewer false positives

In this sense, BRSLabs AISight is easy to deploy. On the other hand, it requires users to be clear what events they are not very intereted in, in order to learn the "normal behaviors" ahead of time.

Other 2009 NPS winners are:

Best in Show Award: Pivot3 for Serverless Computing storage solution

Best in Judges’ Innovation Award : GE Security for Vigilant V-Series life safety system

Best in Monitoring : Broadband Discovery Systems for the Merlin LE

Best in Video Storage and Distribution : Pivot 3 for serverless computing

Best in Access Control : ECKey for the EK4 Enterprise Relay

Best in Integrated Software, Products and Systems : Vidsys for RiskShield

Best in Intrusion Detection : Designed Security for Entry Sentry tailgate detection system

Best in Fire and Life Safety : (Tie) FireLite Alarms by Honeywell for the IPDACT-2UD FireWatch Series IP Communicator; System Sensor for InnovairFlex

Best in OEM : Pentax for the 1.3 Megapixel Varifocal Plus Lens

Best in False Alarm Solutions : Nascom for the Universal Flip Switch

Best in IP Devices, Products and Software : Panasonic for the iPro WV-NP502 megapixel fixed network camera

Best in Video Devices : TimeSight Systems for Video Surveillance Appliances

Best in Video Analytics : Behavioral Recognition Systems for AISight

Face recognition in unconstrained environment is very challenging due to the variations in facial expression, face pose, light condition and occlusion (e.g. eye glasses or hair). A recent project led by Dr. Learned-Miller at University of Massachusetts, Amherst, published the test results on 13,000 images of faces collected from web. The results are described in an ROC curve. At 10% false positive rate, the best true position rate (recognition rate) is around 67%. That is, in 100 face images for one person, only 67 faces are correctly recognized as this person; in 100 faces images of other persons, 10 faces are incorrectly recognized as this person. Therefore there is still big gap between this performance and the requirements of security applications, for example, access control. But are these results useful for other practical applications?  

The answer is YES, but with user interaction. Facebook, Google’s Picasa, and Apple’s iPhoto, have integrated face recognition into their photo management systems. With no exception, auto-tagging of the photos is assisted with user interaction. User interaction can correct the mis-recognized faces, and refine the recognition algorithms towards 100% recognition rate and 0% false positive rate. 

In March 24, Face.com launches face recognition applications for Facebook.com. The technology (Face Finder) used by Face.com is said to be the descriptor based method described in the following paper: 

Lior Wolf, Tal Hassner, and Yaniv Taigman, Descriptor Based Methods in the Wild, Faces in Real-Life Images Workshop in European Conference on Computer Vision (ECCV), 2008. [pdf] [webpage]

The matlab code for Local Binary Pattern (LBP) descriptor is also available online. It has the best performance among the methods tested in UMass’ test. You may want to try it out.

Have you experienced being poked several times for your nurse to find your vein for an IV insertion?

Blood collection from children is an extremely difficult task because of their thick skin. The VeinViewer by Luminetx® can help. The VeinViewer utilizes near infrared (NIR) imaging and image processing technologies to assist health care professionals to find veins easier. 

The following video is a brief report on VeinViewer from ABC.

The website has a description of this ground-breaking technology. 

Infrared light source - The light source emits a harmless, near-infrared light reflected back to the surface from the tissue surrounding the vein, while no light is reflected back from the blood inside the vessel. (Blood in the veins contains deoxygenated hemoglobin. When exposed to near infrared (NIR) light the veins’ de-oxy hemoglobin absorbs NIR while surrounding tissue reflects it. This contrast in light yields the vein image.)

Digital video camera - The digital video camera captures the near-infrared light reflected back from the patient.

Image processing unit - The microprocessor adds contrast and projects this image back on the skin in their actual location.

Digital image projector - Using Texas Instruments Digital Light Processing™ technology, the projector displays these real-time images of the vasculature onto the surface of the skin

Contrast of the captured infrared images is enhanced before projection (e.g. using histogram equalization). The projection back to skin using TI DLP needs camera-projector calibration. 

The vein imaging technology has also been used to hand vein based biometrics by Luminetx®, called SnowFlake Technologies. The idea of using palm vein for biometrics is not new. Fujitsu developed palm vein based biometrics several years ago. Vein pattern recognition is an alternative way to popular fingerprint and face biometrics.

(pictures courtesy of Luminetx®)

How does YouTube generate thumbnails for videos?

According to YouTube’s blog, YouTube’s previous approach to video thumbnail generation is to provide three thumbnails, which are auto-generated from the 20%/50%/75% points in the video index. However, the three auto-generated thumbnails may not be representative to the video content. Obviously something smarter can be done to improve the process. 

Recently, YouTube released a smart thumbnail generation feature, which generates a set of images that are visually informative of the video content using computer vision and video analytics algorithms. This reminds me the “text thumbnails” for web content or “image thumbnails” for images. For example, Google displays your web search “thumbnails” in a similar way, and Adobe’s Content-Aware Image Resizing could be used for automatic image thumbnail generation in image browsing. 

In terms of technology for YouTube’s video thumbnail generation, a simple image/video color histogram would do a decent job. More advanced computer vision algorithms, such as human action clustering/recognition/categorization, salient motion detection, audio-visual analysis, or face recognition, could generate more robust results. However, due to their high computational complexity and the huge amount of video data (13 hours of video per minute to YouTube), I tend to believe YouTube is not using these advanced features yet. It could be an interesting project for computer vision graduate students to think about efficient and robust approaches for automatic video thumbnail generation. 

FashionLatte allows users to search dresses by color, pattern and cut using visual information. It can also match dresses to both shoes and handbags to create outfits. Their current business mode is on B2B (Business-to-Business) licensing with online apparel catalogs, and developing a browser plug-in to earn click-through revenue, according to a recent report. FashionLatte team won the first place in the annual Innovation Teams Business Plan Competition sponsored by the UI’s Academy for Entrepreneurial Leadership.

We discussed the visual search technologies and companies in previous posts, such as like.com, GazoPa, TinEye. FashionLatte is not online yet, and we look forward to experiencing such a wonderful technology soon.

According to Digitalsmiths, their technology:

Indexes video with an unprecedented sense of context, nuance and intelligence

Generates powerful metadata based on visual, audio, text and speech recognition

Gives video publishers the ability to search, retrieve, publish and monetize content 

Therefore, Digitalsmiths’ business model is to license its technology to help video publishers intelligently extract information from videos and dispatch advertisements accordingly. These publishers include celebrity site, TMZ; media site, Warner Brothers; and magazine cite, Essence. 

Here is an example of searching Essence video website,

The video clips tagged with “fashion” are returned. 

A recent interview by ReelSEO with Digitalsmiths’ CEO and Co-Founder, Ben Weinberger, reveals more details of the technologies. The computer vision technologies they employed include facial recognition, scene identification and scene classification, object analysis, and even materials analysis. The speech analysis technologies include speech-to-text analysis and speech context analysis by applying natural language processing to extract speech topics.