Today’s web pages can be described as a hodgepodge of many (and quite often unrelated) elements – articles, comments, posts, ads, banners, tables of contents, hyperlinks, etc. Printed newspapers (which still constitute a serious challenge for OCR systems) are dwarfed in complexity by the complexity of web pages. Of course, HTML DOM is of some help, but its hierarchical structure is no match to lateral links between web page elements with their dynamic rendering.

Let’s consider the following page: http://richlabonte.net/exonews/xtra/whales_dying.htm

This page is one of the top-10 Google Search results for the query whales warning of tsunami. The page consists of 9 separate articles discussing different subjects, including an article about tsunami warning systems (#3) and an article on preservation of whales (#1), but not what was asked – the rather unusual topic of whales warning of tsunamis. As can be seen from this example, the relevancy of a page to a query is not necessarily well defined.

Various means have been employed to deal with web page complexity. The most widely accepted one is to use the distance between keywords occurrences as a measure of relevancy – this is often referred to as proximity measure. This approach works extremely well in the case of a classic Information Retrieval problem of matching keywords to texts. However, on a web page, close proximity of two words does not necessarily mean that they are related. They can be very close on a page while belonging to completely different contexts — for instance, think of a newspaper layout where two unrelated topics appear on different sides of the line separating two columns. The problem is exacerbated with dynamic rendering of web pages, where ads and links to outside pages are inserted inside an article.

All this leads to reinterpretation of the concept of a relevant page. It is no longer a relevant page but rather a relevant context we are after. As soon as we can split a page into contexts, the methods that are applicable to pure texts such as NLP, semantic analysis, proximity measures, etc. become much more reliable.

Glendor Search is built on this principle. We developed a patent-pending technology to analyze arbitrary web pages and divide them into independent contexts. On average, the analysis and indexing of a page requires less than one second. This translates into the ability to crawl and index the entire surface web (~20 billion pages) once a quarter using 1,000 standard PCs.

Inexpensive Viagra
Buy Cheap Propecia
Cialis Pfizer
Buy Pfizer Viagra
Buy Cialis Online Without Prescription
Cialis Tablets Vs Viagra
Cialis No Rx
Cialis From Canada
Buy Propecia Cheap
Cialis On Women
Buy Propecia Online
Pfizer Viagra Online
Canadian Pharmacy Online
How To Get Cialis No Prescription
Buy Viagra Cialis Levitra
Cialis In India
Viagra On Line
Levitra Online Pharmacy
Online Pharmacy
Cheap Cialis
Cheap Levitra Tablets
Cheap 25mg Viagra
Cialis Once Daily
Viagra Soft Tabs 100 Mg
Buy Viagra 100mg
Cialis Canada Online Pharmacy No Prescription
Cialis Cheap Delivery
Cialis Professional 100 Mg
Cialis Online Doctor
Buying Viagra Online
Canadian Rx Viagra
Cheap Generic Viagra Online
Cialis In Canada
Brand Cialis
Cialis Daily Availability
Cialis Professional
Viagra Soft Gel
Viagra Tablets
Generic Cialis
Cialis Soft Pills
Best Quality Viagra
Canadianpharmacy
Canada Cialis Online
Cialis Generic
Buy Levitra Online No Prescription
Cialis On Sale
Generic Cialis Soft Tabs
5 Mg Propecia
Buy Viagra In Canada
Canadian Healthcare Viagra
Cialis Costs
Canadian Pharmacy Viagra Scam
Dose Cialis
Best Recognized Pharmacy In Canada For Viagria
Canadian Pharmacy Cialis 5 Mg
Levitra For Cheap Canadian Pharmacy
Canadian Pharmacy Viagra
Buy Online Prescription Propecia
Buy Viagra Mexico
Pfizer Viagra Canada
Buy Generic Cialis
Cialis Online For Canadian
Cialis100mg
Levitra For Women
Online Viagra
Best Cialis Price
Cialis Quick
Buy Levitra Us
Buy Viagra Online Australia
Discount Cialis And Viagra
Order Propica
Best Price Levitra Online
Real Cialis
Canadamedscom
Overnight Viagra Delivery
Generic Cialis Next Day Delivery
No Prescription
Buy Cheap Generic Propecia
Alternatives To Cialis
Minoxidil Propecia Nizoral
Cheap Generic Levitra
Cheap Onlin Viagra In Usa
Purchase Of Viagra Or Cialis Etc
Online Pharmacy Shop Canadian Healthcare Pharmacy
Brand Name Cialis
Cialis Levitra
Canadian Pharmacy Levitra Value Pack
Buy Viagra From China
Canada Generic Propecia
Levitra 20 Mg
Best Deal For Propecia
Soft Cialis

Curious? Register for a Private Beta at www.glendor.com. We are excited to know what you think.

3, 2, 1 LAUNCH
Cialis Next Day Delivery
Generic Viagra In Canada
Buy Generic No Online Prescription Viagra
Viagra Order
Propecia Canada Pharmacy
Cialis Delivered Overnight
Buy Real Cialis Online
Buy Levitra Online Us
Cheap Viagra 100mg
Cheapest Prices On Viagra
Canadian Generic Viagra On Line
Ordering Cialis Gel
Buy Viagra Australia
Buy Levitra Now
Levitra 20mg
Buy Propecia 5mg
Best Viagra And Popular In Uk
Herbal Viagra
How Much Is Viagra 50
Buy Cheap Generic Levitra
Buying Viagra
Canadian Viagra Scam
Canadian Pharmacy Scam
Viagra Of Pfizer
Cialis At Real Low Prices
Buy Levitra Lowest Prices
Female Viagra Pills
Best Prices For Propecia
Buy Levitra Online From Canada
Is Viagra Different From Levitra
Buy Cialis Once Daily
Canadian Viagra Sales
Best Online Levitra
Viagra Side Effect
Ordering Cialis Online
Buy Levitra Online Viagra
100mg Viagra
Canadian Pharmacy Viagra Prescription
Pharmacy Support Viagra
Fill Viagra Perscription
Viagra Professional
Buy Levitra Vardenafil
Cialis 20 Mg
Online Cialis
Buy Viagra Cheap
Viagra And Three Day Delivery
Buy Cheap Online Propecia
Official Canadian Pharmacy To Buy Levitra
Best Shop For Viagra
Mexico Viagra
Viagra
Best Place Cialis
Cheap Viagra No Prescription
Bestellen Levitra Online
Generic Propecia Mastercard
Buy Viagra Online Canada
How To Buy Cialis In Canada
Online Viagra Scams
Canad Ian Pharmacy
Get Viagra
Cialis Gel
Buy Drug Propecia
Cheap Canadian Propecia
Generic Viagra From Canada
Viagra 50 Mg
Buy Cialis Professional
Viagra 100 Mg
Viagra Online Cheap
Mail Order Propecia
Buy Cialis Without A Prescription
Buy Generic Propecia
Overnight Viagra
Brand Viagra Canada
How Do U Buy Propecia In Canada
Buy Generic Levitra
Mexico Pharmacy
Best Price For Levitra
Original Cialis
How Much Does Cialis Cost
Buy Cialis Online Canada
Canada Levitra
Canada Meds Viagra
Best Canadian Pharmacy
Cialis Low Price
Generic Viagra Canada
Best Doses For Propecia
Canadian Pharmacy Ed
Buy Cialis On Line
Cialis Daily Cost
Cialiscom
Best Prices On Viagra
Combine Cialis And Levitra

• Richard Miller: Diving Deep

I look forward to when Glenbrook or Google will help us find information from these previously unavailable sources. It will mean billions more pages of relevant information available to the world.

• KurzweilAI.net: Safecrackers open up the “deep Web”
• PodTech.net: Listen to the podcast interview of Julia, our co-founder in charge of product management, recorded at the Vertical Leap conference where she spoke on the Future of Vertical Search.

It is supposedly quite easy for a human to navigate through the Deep Web. One just needs to fill up a form by choosing one of several options like destinations and dates a on travel site, or entering a word to search for a meaning or a translation. It is much more difficult for a machine to do so automatically and generically. Because the Deep Web contains a lot of factual information, it can be seen metaphorically as an ocean with a lot of fish. That is why we call the system that navigates the Deep Web a trawler.

There are two major problems with navigating Deep Web automatically. First, the trawler needs to understand what questions to ask through aforementioned forms, and ask them exhaustively. Second, the trawler can not easily navigate from one page to another since pages do not have set URLs or might not even exist. That’s why the trawler needs to remember where it came from and return to the surface (like a whale) before “diving” again to ask the next question.

If the number of sites is relatively small, say a few thousands, each set of forms could be described manually through a templating system. Its major limitations are scalability, and non resilience to changes in page formats. 

There is a third problem that is related to the size of the Deep Web. It is so big that one needs to focus on a particular subset (vertical) to have a chance to trawl it with some level of success, especially if high precision is an important factor. Since the task of determining what questions to ask includes understanding of semantics and context, the focus on a vertical comes handy.

Glenbrook’s approach to building a trawler is based on mimicking the behavior of a (human) user. It is a useful approach since the “doors” opening the Deep Web were built with a human in mind and reflect the standards (no matter how loose) that humans use to navigate the Web.

The Trawler consists of five layers:

1. Discoverer - locates perspective target home pages in Surface Web
2. Scout - navigates Surface Web part of a web site and finds the “doors” - DHTML pages that contain forms leading to the Deep Web part of a web site
3. Locksmith - fills up the forms with various requests and collects responses
4. Assessor - analyses responses and makes a decision to use this door as candidate to query the Deep Web part of the site or move elsewhere
5. Harvester - collects all relevant pages from Surface and Deep Web parts of the web site

After all potentially relevant pages are harvested the Extractor takes over. The Extractor is a hybrid system that applies Pattern Recognition, Natural Language Processing and other AI techniques to extract facts, combine them and populate a database that is used to provide factual answers to search queries.

The Extractor will be the subject of another post.

Tag: Glendor

Cross-posted from Software Only

Komissarchik and her father, Edward Komissarchik, say they have figured out how to analyze the forms on Web pages and understand the type of information the sites are looking for. Then, Glenbrook’s Web crawlers use artificial intelligence to walk themselves through sometimes complex Web forms, answering questions, such as the location of their desired job, in the same way a human would.

Julia Komissarchik likens the process to cracking a safe.

“The way to think of it is, you case the joint,'’ she said. “The scout goes through the form and tries a few options to see what the results will be. Then you have a mastermind or safecracker who gets all this information from the scout and devises a method to open the forms.'’

Finally, she said, the “harvesters'’ spring into action to gather up all the information.

Just to clarify: the “safe” analogy does not imply that the company is breaking passwords, and accessing private information. It relates to getting a machine to access generically information stored beyond interactive forms.

We announced the launch of the Glendor showcase a couple of month ago. This features the first (and still I guess, only) mashup involving jobs listings positioned on GoogleMaps.

Longer post about the concept of “web trawling” implemented by the company on its way.

Thanks to all of you who emailed us since this morning, we are grateful for reports of issues with different browser/OS combination, sorry we are not hiring at this time, and yes we can build large scale custom search and aggregation data solutions. And we are delighted that you like this showcase

Update: Gary Price, who was also quoted by Michael, posted an analysis on Search Engine Watch, that I wanted to briefly comment on. First Glenbrook’s technology does not (and can not) extract information directly from corporate databases, it goes through the public, manual, interface that companies have setup to access that data.The innovation lies in a suite of algorithms that figure out automatically the parameters to be used to extract that data, not requiring any templating of the sites to be targeted.

On server load, queries are made in a sensible way to avoid overloading servers based on response times, etc. And data can be refreshed daily, and maybe multiple times a day if the dataset is small enough. But extracting and caching data that change too frequently would not be appropriate.

On usability and searchability of the data, this is actually where the aggregation of structured data delivers its value: being able to apply on a position, a location, across a wide range of sources (in this case, jobs listings across companies).

Delighted to show you the technology at your convenience Gary…

Tag: Glendor

Cross-posted from Software Only.

I spoke with Jobster CEO Jason Goldberg on Monday, and he described their vision of how WorkZoo will allow users to expand their search beyond their network of jobs on Jobster proper and see “every” job. WorkZoo has its cut out for them – in previous testing, they lagged significantly in their parsing ability compared to Indeed.com and Simply Hired. But this combination of Jobster and WorkZoo makes sense as a combined service – it’s also is similar to the partnership that currently exists between professional social networking service LinkedIn and SimplyHired.

The consolidation has already begun. Interesting.

“Yahoo seems to be taking a cue from Indeed and Simply Hired. Ouch. (Thanks, Richard)”

Joel Cheesman actually posted on the topic before John, and there is an interesting discussion in the comments of his post.

Let’s see what Monster.com and CarreerBuilder’s next moves are in this new segment.

Update: SiliconBeat added their take on the news

I should mentiong that leaving the “Location” field empty uses San Carlos as the reference point for searches (it is sort in the center of Silicon Valley). And here are a few Bay Area zip codes: 94301 for Palo Alto, 94111 for San Francisco and 95113 for San Jose.

Michael Bazeley then pointed to Redfin, which combines satellite maps and MLS homes data for the Seattle area.

The O’Reilly Radar also referred to the Google Maps + Yahoo Traffic mashup that was taken down, and then brought back up.

So Glendor.com is a mashup as well then!

Finally, I found Google Maps Mania in our referrer logs:  An unofficial Google Maps blog tracking the websites, ideas and tools being influenced by Google Maps.