Rich keywords

Concept Search

2011-09-21T09:15:00.000-07:00

A concept search (or conceptual search) is an automated information retrieval method that is used to search electronically stored unstructured text (for example, digital archives, email, scientific literature, etc.) for information that is conceptually similar to the information provided in a search query. In other words, the ideas expressed in the information retrieved in response to a concept search query are relevant to the ideas contained in the text of the query.

Why Concept Search?

2011-09-21T09:00:00.000-07:00

Concept search techniques were developed because of limitations imposed by classical Boolean keyword search technologies when dealing with large, unstructured digital collections of text. Keyword searches often return results that include many non-relevant items (false positives) or that exclude too many relevant items (false negatives) because of the effects of synonymy and polysemy. Synonymy means that one of two or more words in the same language have the same meaning, and polysemy means that many individual words have more than one meaning.
Polysemy is a major obstacle for all computer systems that attempt to deal with human language. In English, most frequently used terms have several common meanings. For example, the word fire can mean: a combustion activity; to terminate employment; to launch, or to excite (as in fire up). For the 200 most-polysemous terms in English, the typical verb has more than twelve common meanings, or senses. The typical noun from this set has more than eight common senses. For the 2000 most-polysemous terms in English, the typical verb has more than eight common senses and the typical noun has more than five.
In addition to the problems of polysemy and synonymy, keyword searches can exclude inadvertently misspelled words as well as the variations on the stems (or roots) of words (for example, strike vs. striking). Keyword searches are also susceptible to errors introduced by optical character recognition (OCR) scanning processes, which can introduce random errors into the text of documents (often referred to as noisy text)during the scanning process.
A concept search can overcome these challenges by employing word sense disambiguation (WSD), and other techniques, to help it derive the actual meanings of the words, and their underlying concepts, rather than by simply matching character strings like keyword search technologies.

Approaches to Concept Search

2011-09-20T10:30:00.000-07:00

In general, information retrieval research and technology can be divided into two broad categories: semantic and statistical. Information retrieval systems that fall into the semantic category will attempt to implement some degree of syntactic and semantic analysis of the natural language text that a human user would provide (also see computational linguistics). Systems that fall into the statistical category will find results based on statistical measures of how closely they match the query. It must be noted, however, that systems in the semantic category also often rely on statistical methods to help them find and retrieve information.
Efforts to provide information retrieval systems with semantic processing capabilities have basically used three different approaches:

Auxiliary structures
Local co-occurrence statistics
Transform techniques (particularly matrix decompositions)

Auxiliary Structures

A variety of techniques based on Artificial Intelligence (AI) and Natural Language Processing (NLP) have been applied to semantic processing, and most of them have relied on the use of auxiliary structures such as controlled vocabularies and ontologies. Controlled vocabularies (dictionaries and thesauri), and ontologies allow broader terms, narrower terms, and related terms to be incorporated into queries.Controlled vocabularies are one way to overcome some of the most severe constraints of Boolean keyword queries. Over the years, additional auxiliary structures of general interest, such as the large synonym sets of WordNet, have been constructed. It was shown that concept search which is based on auxiliary structures, such as WordNet, can be efficiently implemented by reusing retrieval models and data structures of classical Information Retrieval. Later approaches have implemented grammars to expand the range of semantic constructs. The creation of data models that represent sets of concepts within a specific domain (domain ontologies), and which can incorporate the relationships among terms, have also been implemented in recent years.

Handcrafted controlled vocabularies contribute to the efficiency and comprehensiveness of information retrieval and related text analysis operations, but they work best when topics are narrowly defined and the terminology is standardized. Controlled vocabularies require extensive human input and oversight to keep up with the rapid evolution of language. They also are not well-suited to the growing volumes of unstructured text covering an unlimited number of topics and containing thousands of unique terms because new terms and topics need to be constantly introduced. Controlled vocabularies are also prone to capturing a particular world view at a specific point in time, which makes them difficult to modify if concepts in a certain topic area change.

Local Co-Occurrence Statistics

Information retrieval systems incorporating this approach count the number of times that groups of terms appear together (co-occur) within a sliding window of terms or sentences (for example, ± 5 sentences or ± 50 words) within a document. It is based on the idea that words that occur together in similar contexts have similar meanings. It is local in the sense that the sliding window of terms and sentences used to determine the co-occurrence of terms is relatively small.
This approach is simple, but it captures only a small portion of the semantic information contained in a collection of text. At the most basic level, numerous experiments have shown that approximately only ¼ of the information contained in text is local in nature. In addition, to be most effective, this method requires prior knowledge about the content of the text, which can be difficult with large, unstructured document collections.

Transform Techniques

Some of the most powerful approaches to semantic processing are based on the use of mathematical transform techniques. Matrix decomposition techniques have been the most successful. Some widely used matrix decomposition techniques include the following:

Matrix decomposition techniques are data-driven, which avoids many of the drawbacks associated with auxiliary structures. They are also global in nature, which means they are capable of much more robust information extraction and representation of semantic information than techniques based on local co-occurrence statistics.
Independent component analysis is a technique that works well with data of limited variability, and the semi-discrete and non-negative matrix approaches sacrifice accuracy of representation in order to reduce computational complexity.
Singular value decomposition (SVD) was first applied to text at Bell Labs in the late 1980s. It was used as the foundation for a technique called Latent Semantic Indexing (LSI) because of its ability to find the semantic meaning that is latent in a collection of text. At first, the SVD was slow to be adopted because of the resource requirements needed to work with large datasets. However, the use of LSI has significantly expanded in recent years as earlier challenges in scalability and performance have been overcome. LSI is being used in a variety of information retrieval and text processing applications, although its primary application has been for concept searching and automated document categorization.

Uses of Concept Search

2011-09-20T10:00:00.000-07:00

eDiscovery - Concept-based search technologies are increasingly being used for Electronic Document Discovery (EDD or eDiscovery) to help enterprises prepare for litigation. In eDiscovery, the ability to cluster, categorize, and search large collections of unstructured text on a conceptual basis is much more efficient than traditional linear review techniques. Concept-based searching is becoming accepted as a reliable and efficient search method that is more likely to produce relevant results than keyword or Boolean searches.

Enterprise Search and Enterprise Content Management (ECM) - Concept search technologies are being widely used in enterprise search. As the volume of information within the enterprise grows, the ability to cluster, categorize, and search large collections of unstructured text on a conceptual basis has become essential. In 2004 the Gartner Group estimated that professionals spend 30 percent of their time searching, retrieving, and managing information.The research company IDC found that a 2,000-employee corporation can save up to $30 million per year by reducing the time employees spend trying to find information and duplicating existing documents.

Content-Based Image Retrieval (CBIR) - Content-based approaches are being used for the semantic retrieval of digitized images and video from large visual corpora. One of the earliest content-based image retrieval systems to address the semantic problem was the ImageScape search engine. In this system, the user could make direct queries for multiple visual objects such as sky, trees, water, etc. using spatially positioned icons in a WWW index containing more than ten million images and videos using keyframes. The system used information theory to determine the best features for minimizing uncertainty in the classification.The semantic gap is often mentioned in regard to CBIR. The semantic gap refers to the gap between the information that can be extracted from visual data and the interpretation that the same data have for a user in a given situation.The ACM SIGMM Workshop on Multimedia Information Retrieval is dedicated to studies of CBIR.

Multimedia and Publishing - Concept search is used by the multimedia and publishing industries to provide users with access to news, technical information, and subject matter expertise coming from a variety of unstructured sources. Content-based methods for multimedia information retrieval (MIR) have become especially important when text annotations are missing or incomplete.

Digital Libraries and Archives - Images, videos, music, and text items in digital libraries and digital archives are being made accessible to large groups of users (especially on the Web) through the use of concept search techniques. For example, the Executive Daily Brief (EDB), a business information monitoring and alerting product developed by EBSCO Publishing, uses concept search technology to provide corporate end users with access to a digital library containing a wide array of business content. In a similar manner, the Music Genome Project spawned Pandora, which employs concept searching to spontaneously create individual music libraries or virtual radio stations.

Genomic Information Retrieval (GIR) - Genomic Information Retrieval (GIR) uses concept search techniques applied to genomic literature databases to overcome the ambiguities of scientific literature.

Human Resources Staffing and Recruiting - Many human resources staffing and recruiting organizations have adopted concept search technologies to produce highly relevant resume search results that provide more accurate and relevant candidate resumes than loosely related keyword results.

Effective Concept Searching

2011-09-20T09:00:00.000-07:00

The effectiveness of a concept search can depend on a variety of elements including the dataset being searched and the search engine that is used to process queries and display results. However, most concept search engines work best for certain kinds of queries:

Effective queries are composed of enough text to adequately convey the intended concepts. Effective queries may include full sentences, paragraphs, or even an entire documents. Queries composed of just a few words are not as likely to return the most relevant results.

Effective queries do not include concepts in a query that are not the object of the search. Including too many unrelated concepts in a query can negatively affect the relevancy of the result items. For example, searching for information about boating on the Mississippi River would be more likely to return relevant results than a search for boating on the Mississippi River on a rainy day in the middle of the summer in 1967.

Effective queries are expressed in a full-text, natural language style similar in style to the documents being searched. For example, using queries composed of excerpts from an introductory science textbook would not be as effective for concept searching if the dataset being searched is made up of advanced, college-level science texts. Substantial queries that better represent the overall concepts, styles, and language of the items for which the query is being conducted are generally more effective.

As with all search strategies, experienced searchers generally refine their queries through multiple searches, starting with an initial seed query to obtain conceptually relevant results that can then be used to compose and/or refine additional queries for increasingly more relevant results. Depending on the search engine, using query concepts found in result documents can be as easy as selecting a document and performing a find similar function. Changing a query by adding terms and concepts to improve result relevance is called query expansion.The use of ontologies such as WordNet has been studied to expand queries with conceptually-related words

Relevance Feedback

2011-09-19T10:30:00.000-07:00

Relevance feedback is a feature that helps users determine if the results returned for their queries meet their information needs. In other words, relevance is assessed relative to an information need, not a query. A document is relevant if it addresses the stated information need, not because it just happens to contain all the words in the query.It is a way to involve users in the retrieval process in order to improve the final result set. Users can refine their queries based on their initial results to improve the quality of their final results.
In general, concept search relevance refers to the degree of similarity between the concepts expressed in the query and the concepts contained in the results returned for the query. The more similar the concepts in the results are to the concepts contained in the query, the more relevant the results are considered to be. Results are usually ranked and sorted by relevance so that the most relevant results are at the top of the list of results and the least relevant results are at the bottom of the list.
Relevance feedback has been shown to be very effective at improving the relevance of results.A concept search decreases the risk of missing important result items because all of the items that are related to the concepts in the query will be returned whether or not they contain the same words used in the query.
Ranking will continue to be a part of any modern information retrieval system. However, the problems of heterogeneous data, scale, and non-traditional discourse types reflected in the text, along with the fact that search engines will increasingly be integrated components of complex information management processes, not just stand-alone systems, will require new kinds of system responses to a query. For example, one of the problems with ranked lists is that they might not reveal relations that exist among some of the result items.

Guidelines for Evaluating a Concept Search Engine

2011-09-19T10:00:00.000-07:00

Result items should be relevant to the information need expressed by the concepts contained in the query statements, even if the terminology used by the result items is different from the terminology used in the query.
Result items should be sorted and ranked by relevance.
Relevant result items should be quickly located and displayed. Even complex queries should return relevant results fairly quickly.
Query length should be non-fixed, i.e., a query can be as long as deemed necessary. A sentence, a paragraph, or even an entire document can be submitted as a query.
A concept query should not require any special or complex syntax. The concepts contained in the query can be clearly and prominently expressed without using any special rules.
Combined queries using concepts, keywords, and metadata should be allowed.
Relevant portions of result items should be usable as query text simply by selecting the item and telling the search engine to find similar items.
Query-ready indexes should be created relatively quickly.
The search engine should be capable of performing Federated searches. Federated searching enables concept queries to be used for simultaneously searching multiple datasources for information, which are then merged, sorted, and displayed in the results.
A concept search should not be affected by misspelled words, typographical errors, or OCR scanning errors in either the query text or in the text of the dataset being searched.

Search Engine Conferences and Forums

2011-09-19T09:00:00.000-07:00

Formalized search engine evaluation has been ongoing for many years. For example, the Text REtrieval Conference (TREC) was started in 1992 to support research within the information retrieval community by providing the infrastructure necessary for large-scale evaluation of text retrieval methodologies. Most of today's commercial search engines include technology first developed in TREC.
In 1997, a Japanese counterpart of TREC was launched, called National Institute of Informatics Test Collection for IR Systems (NTCIR). NTCIR conducts a series of evaluation workshops for research in information retrieval, question answering, text summarization, etc. A European series of workshops called the Cross Language Evaluation Forum (CLEF) was started in 2001 to aid research in multilingual information access. In 2002, the Initiative for the Evaluation of XML Retrieval (INEX) was established for the evaluation of content-oriented XML retrieval systems.
Precision and recall have been two of the traditional performance measures for evaluating information retrieval systems. Precision is the fraction of the retrieved result documents that are relevant to the user's information need. Recall is defined as the fraction of relevant documents in the entire collection that are returned as result documents.
Although the workshops and publicly available test collections used for search engine testing and evaluation have provided substantial insights into how information is managed and retrieved, the field has only scratched the surface of the challenges people and organizations face in finding, managing, and, using information now that so much information is available.Scientific data about how people use the information tools available to them today is still incomplete because experimental research methodologies haven’t been able to keep up with the rapid pace of change. Many challenges, such as contextualized search, personal information management, information integration, and task support, still need to be addressed.

Search advertising

2011-09-17T22:25:00.000-07:00

In Internet Marketing, Search Advertising is a method of placing online advertisements on Web pages that show results from search engine queries. Through the same search-engine advertising services, ads can also be placed on Web pages with other published .
Search advertisements are targeted to match key search terms (called keywords) entered on search engines. This targeting ability has contributed to the attractiveness of search advertising for advertisers. Consumers will often use a search engine to identify and compare purchasing options immediately before making a purchasing decision. The opportunity to present consumers with advertisements tailored to their immediate buying interests encourages consumers to click on search ads instead of unpaid search results, which are often less relevant. Unpaid search results are also called organic results

Origins

Its believed that Yahoo! first introduced a stand-alone search advertising buy. In 1996, Chip Royce, head of online marketing for InterZine Productions of Boca Raton, Florida, approached Yahoo!'s sales agent (Softbank Interactive Media Sales) seeking more effective, targeted advertising within Yahoo!'s search results. Yahoo! obliged placing targeted ad banners when the keyword "Golf" was searched by Yahoo! users. Yahoo! later turned this opportunity into a formal marketing program for its entire customer base and promoted this in a July 1996 article in the now defunct magazine.

Keywords

2011-09-17T22:23:00.002-07:00

Search advertising is sold and delivered on the basis of keywords. The user of a search engine enters keywords to make queries. A keyword may consist of more than one word.
Search engines conduct running auctions to sell ads according to bids received for keywords and relative relevance of user keywords to ads in the inventory. The keyword “home mortgage refinancing" is more expensive than one that is in less demand, such as “used bicycle tires.” Profit potential of the keywords also plays into bids for ads that advertisers want displayed when the keywords are searched by the user. For example, "used book" may be a popular keyword but may have low profit potential and the advertiser bids will reflect that.
Search engines build indexes of Web pages using a web Crawler. When the publisher of a Web page arranges with a search engine firm to have ads served up on that page, the search engine applies their indexing technology to associate the content of that page with keywords. Those keywords are then fed into the same auctioning system that is used by advertisers to buy ads on both search engine results pages. Advertising based on keywords in the surrounding content or context is referred to as Contextual advertising. This is usually less profitable than search advertising which is based on user intent expressed through their keywords.
Advertisers can choose whether to buy ads on search result pages (search advertising), published content pages (contextual advertising), or both. Bids on the same keywords are usually higher in search advertising than in contextual advertising.

Search Advertising Metrics

2011-09-17T22:23:00.000-07:00

Search advertising activities can be measured in five ways:
CPM: Cost per thousand viewers was the original method used for pricing online advertisements. CPM remains the most common method for pricing banner ads.
CTR: Click-through rates measure the number of times an ad is clicked as a percentage of views of the Web page on which the ad appears. Banner ads have CTRs that are generally 0.5 percent or less. In comparison, individual search engine ads can have CTRs of 10 percent, even though they appear alongside organic search results and competing paid search advertisements.
CPA: Cost per action quantifies costs for completing specified activities such as attracting a new customer or making a sale. Affiliate networks operate on a CPA basis. CPA systems function most effectively when sales cycles are short and easily tracked. Longer sales cycles rely on exposure to multiple types of ads to create brand awareness and purchasing interest before a sale is made. Longer sales cycles and sales requiring multiple customer contacts can be difficult to track, leading to a reluctance by publishers to participate in CPA programs beyond initial lead generation.
CPC: Cost per click tracks the cost of interacting with a client or potential client. In traditional marketing, CPC is viewed as a one-way process of reaching target audiences through means such as direct mail, radio ads and television ads. Search advertising provides opportunities for two-way contacts through web-based chat, Internet-based calls, call-back requests or mailing list signups.
TM: Total minutes is a metric being used by Nielsen/NetRatings to measure total time spent on a Web page rather than the number of Web page views. On July 10, 2007, Nielsen announced that they would be relying on TM as their primary metric for measuring Web page popularity, due to changes in the way Web pages provide content through audio and video streaming and by refreshing the same page without totally reloading it. Page refreshes are one aspect of Rich Internet Applications (RIA). RIA technologies include AJAX (Asynchronous JavaScript and XML) and Microsoft Silverlight.
Methodological questions regarding the use of total minutes for search advertising include how to account for Internet users that keep several browser windows open simultaneously, or who simply leave one window open unattended for long periods of time. Another question involves tracking total minutes on HTML pages that are stateless and do therefore do not generate server-side data on the length of time that they are viewed.

Metasearch engine

2011-09-17T03:05:00.000-07:00

A metasearch engine is a search tool that sends user requests to several other search engines and/or databases and aggregates the results into a single list or displays them according to their source. Metasearch engines enable users to enter search criteria once and access several search engines simultaneously. Metasearch engines operate on the premise that the Web is too large for any one search engine to index it all and that more comprehensive search results can be obtained by combining the results from several search engines. This also may save the user from having to use multiple search engines separately.
The term "metasearch" is frequently used to classify a set of commercial search engines, see the list of search engines, but is also used to describe the paradigm of searching multiple data sources in real time. The National Information Standards Organization (NISO) uses the terms Federated Search and Metasearch interchangeably to describe this web search paradigm.

Operation

architecture of a metasearch engine

Metasearch engines create what is known as a virtual database. They do not compile a physical database or catalogue of the web. Instead, they take a user's request, pass it to several other heterogeneous databases and then compile the results in a homogeneous manner based on a specific algorithm.
No two metasearch engines are alike. Some search only the most popular search engines while others also search lesser-known engines, newsgroups, and other databases. They also differ in how the results are presented and the quantity of engines that are used. Some will list results according to search engine or database. Others return results according to relevance, often concealing which search engine returned which results. This benefits the user by eliminating duplicate hits and grouping the most relevant ones at the top of the list.
Search engines frequently have different ways they expect requests submitted. For example, some search engines allow the usage of the word "AND" while others require "+" and others require only a space to combine words. The better metasearch engines try to synthesize requests appropriately when submitting them

Web search engine

2011-09-17T02:52:00.000-07:00

"Search engine" redirects here. For other uses, see Search engine (disambiguation).

The three most widely used web search engines and their approximate share as of late 2010.

A web search engine is designed to search for information on the World Wide Web and FTP servers. The search results are generally presented in a list of results and are often called hits. The information may consist of web pages, images, information and other types of files. Some search engines also mine data available in databases or open directories. Unlike web directories, which are maintained by human editors, search engines operate algorithmically or are a mixture of algorithmic and human input.

History

Timeline (full list)
Year	Engine	Current status
1993	W3Catalog	Closed
	Aliweb	Closed
	JumpStation	Closed
1994	WebCrawler	Active
	Go.com	Active
	Lycos	Active
1995	AltaVista	Bought and operated by Yahoo!
	Daum	Active
	Magellan	Active
	Excite	Active
	SAPO	Active
	Yahoo!	Active, Launched as a directory
1996	Dogpile	Active
	Inktomi	Active
	HotBot	Active
	Ask Jeeves	Active
1997	Northern Light	Active
1997	Yandex	Active
1998	Google	Active
1998	MSN Search	Active as Bing
1999	AlltheWeb	Active
	GenieKnows	Active
	Naver	Active
	Teoma	Active
	Vivisimo	Active
2000	Baidu	Active
2000	Exalead	Active
2002	Inktomi	Acquired by Yahoo!
2003	Info.com	Active
2004	Yahoo! Search	Active, Launched own web search (see Yahoo! Directory, 1995)
	A9.com	Closed
	Sogou	Active
2005	AOL Search	Active
	Ask.com	Active
	GoodSearch	Active
	SearchMe	Active
2006	wikiseek	Active
	Quaero	Active
	Ask.com	Active
	Live Search	Active as Bing, Launched as rebranded MSN Search
	ChaCha	Active
	Guruji.com	Active
2007	wikiseek	Closed
	Sproose	Closed
	Wikia Search	Closed
	Blackle.com	Active
2008	Powerset	Acquired by Microsoft
	Picollator	Closed
	Viewzi	Closed
	Boogami	Active
	LeapFish	Active
	Forestle	Active
	VADLO	Active
	Duck Duck Go	Active
2009	Bing	Active, Launched as rebranded Live Search
	Yebol	Active
	Mugurdy	Closed due to a lack of funding
	Goby	Active
2010	Yandex	Active, Launched global (English) search
	Cuil	Closed
	Blekko	Active
	Viewzi	Closed due to a lack of funding
	Yummly	Active
2011	Exalead	Acquired by Dassault Systèmes
2011	Interred	Active

During the early development of the web, there was a list of webservers edited by Tim Berners-Lee and hosted on the CERN webserver. One historical snapshot from 1992 remains.As more webservers went online the central list could not keep up. On the NCSA site new servers were announced under the title "What's New!"
The very first tool used for searching on the Internet was Archie.The name stands for "archive" without the "v". It was created in 1990 by Alan Emtage, Bill Heelan and J. Peter Deutsch, computer science students at McGill University in Montreal. The program downloaded the directory listings of all the files located on public anonymous FTP (File Transfer Protocol) sites, creating a searchable database of file names; however, Archie did not index the contents of these sites since the amount of data was so limited it could be readily searched manually.
The rise of Gopher (created in 1991 by Mark McCahill at the University of Minnesota) led to two new search programs, Veronica and Jughead. Like Archie, they searched the file names and titles stored in Gopher index systems. Veronica (Very Easy Rodent-Oriented Net-wide Index to Computerized Archives) provided a keyword search of most Gopher menu titles in the entire Gopher listings. Jughead (Jonzy's Universal Gopher Hierarchy Excavation And Display) was a tool for obtaining menu information from specific Gopher servers. While the name of the search engine "Archie" was not a reference to the Archie comic book series, "Veronica" and "Jughead" are characters in the series, thus referencing their predecessor.
In the summer of 1993, no search engine existed yet for the web, though numerous specialized catalogues were maintained by hand. Oscar Nierstrasz at the University of Geneva wrote a series of Perl scripts that would periodically mirror these pages and rewrite them into a standard format which formed the basis for W3Catalog, the web's first primitive search engine, released on September 2, 1993.
In June 1993, Matthew Gray, then at MIT, produced what was probably the first web robot, the Perl-based World Wide Web Wanderer, and used it to generate an index called 'Wandex'. The purpose of the Wanderer was to measure the size of the World Wide Web, which it did until late 1995. The web's second search engine Aliweb appeared in November 1993. Aliweb did not use a web robot, but instead depended on being notified by website administrators of the existence at each site of an index file in a particular format.
JumpStation (released in December 1993) used a web robot to find web pages and to build its index, and used a web form as the interface to its query program. It was thus the first WWW resource-discovery tool to combine the three essential features of a web search engine (crawling, indexing, and searching) as described below. Because of the limited resources available on the platform on which it ran, its indexing and hence searching were limited to the titles and headings found in the web pages the crawler encountered.
One of the first "full text" crawler-based search engines was WebCrawler, which came out in 1994. Unlike its predecessors, it let users search for any word in any webpage, which has become the standard for all major search engines since. It was also the first one to be widely known by the public. Also in 1994, Lycos (which started at Carnegie Mellon University) was launched and became a major commercial endeavor.
Soon after, many search engines appeared and vied for popularity. These included Magellan (search engine), Excite, Infoseek, Inktomi, Northern Light, and AltaVista. Yahoo! was among the most popular ways for people to find web pages of interest, but its search function operated on its web directory, rather than full-text copies of web pages. Information seekers could also browse the directory instead of doing a keyword-based search.
In 1996, Netscape was looking to give a single search engine an exclusive deal to be the featured search engine on Netscape's web browser. There was so much interest that instead a deal was struck with Netscape by five of the major search engines, where for $5 million per year each search engine would be in rotation on the Netscape search engine page. The five engines were Yahoo!, Magellan, Lycos, Infoseek, and Excite.
Search engines were also known as some of the brightest stars in the Internet investing frenzy that occurred in the late 1990s. Several companies entered the market spectacularly, receiving record gains during their initial public offerings. Some have taken down their public search engine, and are marketing enterprise-only editions, such as Northern Light. Many search engine companies were caught up in the dot-com bubble, a speculation-driven market boom that peaked in 1999 and ended in 2001.
Around 2000, Google's search engine rose to prominence.^{[citation needed]} The company achieved better results for many searches with an innovation called PageRank. This iterative algorithm ranks web pages based on the number and PageRank of other web sites and pages that link there, on the premise that good or desirable pages are linked to more than others. Google also maintained a minimalist interface to its search engine. In contrast, many of its competitors embedded a search engine in a web portal.
By 2000, Yahoo! was providing search services based on Inktomi's search engine. Yahoo! acquired Inktomi in 2002, and Overture (which owned AlltheWeb and AltaVista) in 2003. Yahoo! switched to Google's search engine until 2004, when it launched its own search engine based on the combined technologies of its acquisitions.
Microsoft first launched MSN Search in the fall of 1998 using search results from Inktomi. In early 1999 the site began to display listings from Looksmart blended with results from Inktomi except for a short time in 1999 when results from AltaVista were used instead. In 2004, Microsoft began a transition to its own search technology, powered by its own web crawler (called msnbot).
Microsoft's rebranded search engine, Bing, was launched on June 1, 2009. On July 29, 2009, Yahoo! and Microsoft finalized a deal in which Yahoo! Search would be powered by Microsoft Bing technology.

How web search engines work

High-level architecture of a standard Web crawler

A search engine operates in the following order:

Web search engines work by storing information about many web pages, which they retrieve from the html itself. These pages are retrieved by a Web crawler (sometimes also known as a spider) — an automated Web browser which follows every link on the site. Exclusions can be made by the use of robots.txt. The contents of each page are then analyzed to determine how it should be indexed (for example, words are extracted from the titles, headings, or special fields called meta tags). Data about web pages are stored in an index database for use in later queries. A query can be a single word. The purpose of an index is to allow information to be found as quickly as possible. Some search engines, such as Google, store all or part of the source page (referred to as a cache) as well as information about the web pages, whereas others, such as AltaVista, store every word of every page they find. This cached page always holds the actual search text since it is the one that was actually indexed, so it can be very useful when the content of the current page has been updated and the search terms are no longer in it. This problem might be considered to be a mild form of linkrot, and Google's handling of it increases usability by satisfying user expectations that the search terms will be on the returned webpage. This satisfies the principle of least astonishment since the user normally expects the search terms to be on the returned pages. Increased search relevance makes these cached pages very useful, even beyond the fact that they may contain data that may no longer be available elsewhere.
When a user enters a query into a search engine (typically by using key words), the engine examines its index and provides a listing of best-matching web pages according to its criteria, usually with a short summary containing the document's title and sometimes parts of the text. The index is built from the information stored with the data and the method by which the information is indexed. Unfortunately, there are currently no known public search engines that allow documents to be searched by date. Most search engines support the use of the boolean operators AND, OR and NOT to further specify the search query. Boolean operators are for literal searches that allow the user to refine and extend the terms of the search. The engine looks for the words or phrases exactly as entered. Some search engines provide an advanced feature called proximity search which allows users to define the distance between keywords. There is also concept-based searching where the research involves using statistical analysis on pages containing the words or phrases you search for. As well, natural language queries allow the user to type a question in the same form one would ask it to a human. A site like this would be ask.com.
The usefulness of a search engine depends on the relevance of the result set it gives back. While there may be millions of web pages that include a particular word or phrase, some pages may be more relevant, popular, or authoritative than others. Most search engines employ methods to rank the results to provide the "best" results first. How a search engine decides which pages are the best matches, and what order the results should be shown in, varies widely from one engine to another. The methods also change over time as Internet usage changes and new techniques evolve. There are two main types of search engine that have evolved: one is a system of predefined and hierarchically ordered keywords that humans have programmed extensively. The other is a system that generates an "inverted index" by analyzing texts it locates. This second form relies much more heavily on the computer itself to do the bulk of the work.
Most Web search engines are commercial ventures supported by advertising revenue and, as a result, some employ the practice of allowing advertisers to pay money to have their listings ranked higher in search results. Those search engines which do not accept money for their search engine results make money by running search related ads alongside the regular search engine results. The search engines make money every time someone clicks on one of these ads.

Market share and wars

Search engine	Market share in May 2011	Market share in December 2010
Google	82.80%	84.65%
Yahoo!	6.42%	6.69%
Baidu	4.89%	3.39%
Bing	3.91%	3.29%
Ask	0.52%	0.56%
AOL	0.36%	0.42%

Google's worldwide market share peaked at 86.3% in April 2010.In the United States, Google held a 63.2% market share in May 2009, according to Nielsen NetRatings.Yahoo!, Bing and other search engines are more popular in the US than in Europe.
In the People's Republic of China, Baidu held a 61.6% market share for web search in July 2009^.
By end of July 2011, based on an Experian Hitwise report, although Google remains ahead of Bing, but by success rate resulted in a visit to a web site were different, Yahoo Search by more than 81.36 percent, Bing at 80.6 percent and Google at 67.6 percent.

Search engine bias

Although search engines are programmed to rank websites based on their popularity and relevancy, empirical studies indicate various political, economic, and social biases in the information they provide.These biases could be a direct result of economic and commercial processes (e.g., companies that advertise with a search engine can become also more popular in its organic search results), and political processes (e.g., the removal of search results in order to comply with local laws). Google Bombing is one example of an attempt to manipulate search results for political, social or commercial reasons.

Functionality

2011-09-10T09:00:00.000-07:00

Image of definition link provided for many search terms.

Google search consists of a series of localized websites. The largest of those, the google.com site, is the top most-visited website in the world. Some of its features include a definition link for most searches including dictionary words, the number of results you got on your search, links to other searches (e.g. for words that Google believes to be misspelled, it provides a link to the search results using its proposed spelling), and many more.

Search syntax

Google's search engine normally accepts queries as a simple text, and breaks up the user's text into a sequence of search terms, which will usually be words that are to occur in the results, but one can also use Boolean operators, such as: quotations marks (") for a phrase, a prefix such as "+", "-" for qualified terms, or one of several advanced operators, such as "site:". The webpages of "Google Search Basics" describe each of these additional queries and options (see below: Search options). Google's Advanced Search web form gives several additional fields which may be used to qualify searches by such criteria as date of first retrieval. All advanced queries transform to regular queries, usually with additional qualified term.

Query expansion

Google applies query expansion to the submitted search query, transforming it into the query that will actually be used to retrieve results. As with page ranking, the exact details of the algorithm Google uses are deliberately obscure, but certainly the following transformations are among those that occur:

Term reordering: in information retrieval this is a standard technique to reduce the work involved in retrieving results. This transformation is invisible to the user, since the results ordering uses the original query order to determine relevance.
Stemming is used to increase search quality by keeping small syntactic variants of search terms.
There is a limited facility to fix possible misspellings in queries.

"I'm Feeling Lucky"

Google's homepage includes a button labelled "I'm Feeling Lucky". When a user types in a search and clicks on the button the user will be taken directly to the first search result, bypassing the search engine results page. The thought is that if a user is "feeling lucky", the search engine will return the perfect match the first time without having to page through the search results. However, with the introduction of Google Instant, it is not possible to use the button properly unless the Google Instant function is switched off. According to a study by Tom Chavez of "Rapt", this feature costs Google $110 million a year as 1% of all searches use this feature and bypass all advertising.
On October 30, 2009, for some users, the "I'm Feeling Lucky" button was removed from Google's main page, along with the regular search button. Both buttons were replaced with a field that reads, "This space intentionally left blank." This text faded out when the mouse was moved on the page, and normal search functionality is achieved by filling in the search field with the desired terms and pressing enter. A Google spokesperson explains, "This is just a test, and a way for us to gauge whether our users will like an even simpler search interface." Personalized Google homepages retained both buttons and their normal functions.
On May 21, 2010, the 30th anniversary of Pac-Man, the "I'm Feeling Lucky" button was replaced with a button reading the words "Insert Coin". After pressing the button, the user would begin a Google-themed game of Pac-Man in the area where the Google logo would normally be. Pressing the button a second time would begin a two-player version of the same game that includes Ms. Pacman for player 2. This version can be accessed at www.google.com/pacman/ as a permanent link to the page.

Rich Snippets

On 12 May 2009, Google announced that they would be parsing the hCard, hReview, and hProduct microformats and using them to populate search result pages with what they called "Rich Snippets".

Special features

Besides the main search-engine feature of searching for text, Google Search has more than 22 "special features" (activated by entering any of dozens of trigger words) when searching:

weather – The weather conditions, temperature, wind, humidity, and forecast,for many cities, can be viewed by typing "weather" along with a city for larger cities or city and state, U.S. zip code, or city and country for smaller cities (such as: weather Lawrence, Kansas; weather Paris; weather Bremen, Germany).
stock quotes – The market datafor a specific company or fund can be viewed, by typing the ticker symbol (or include "stock"), such as: CSCO; MSFT; IBM stock; F stock (lists Ford Motor Co.); or AIVSX (fund). Results show inter-day changes, or 5-year graph, etc. This does not work for stock names which are one letter long, such as Citigroup (C) or Macy's (M) (Ford being an exception), or are common words, such as Diamond Offshore (DO) or Majesco (COOL).
time – The current time in many cities (worldwide),can be viewed by typing "time" and the name of the city (such as: time Cairo; time Pratt, KS).
sports scores – The scores and schedules, for sports teams,can be displayed by typing the team name or league name into the search box.
unit conversion – Measurements can be converted,by entering each phrase, such as: 10.5 cm in inches; or 90 km in miles
currency conversion – A money or currency converter can be selected,by typing the names or currency codes (listed by ISO 4217): 6789 Euro in USD; 150 GBP in USD; 5000 Yen in USD; 5000 Yuan in lira (the U.S. dollar can be USD or "US$" or "$", while Canadian is CAD, etc.).
calculator – Calculation results can be determined,as calculated live, by entering a formula in numbers or words, such as: 6*77 +pi +sqrt(e^3)/888 plus 0.45. The user is given the option to search for the formula, after calculation. The calculator also uses the unit and currency conversion functions to allow unit-aware calculations. For example, "(3 EUR/liter) / (40 miles/gallon) in USD / mile" calculates the dollar cost per mile for a 40 mpg car with gas costing 3 euros a liter. The caret "^" raises a number to an exponent power, and percentages are allowed ("40% of 300").There is also some debate as to Google's calculation of 0^0. Many mathematicians believe that 0^0 is undefined but Google's calculator shows the result as 1.
numeric ranges – A set of numbers can be matched by using a double-dot between range numbers (70..73 or 90..100) to match any positive number in the range, inclusive.Negative numbers are treated as using exclusion-dash to not match the number.
dictionary lookup – A definition for a word or phrase can be found,by entering "define" followed by a colon and the word(s) to lookup (such as, "define:philosophy")
maps – Some related maps can be displayed,by typing in the name or U.S. ZIP code of a location and the word "map" (such as: New York map; Kansas map; or Paris map).
movie showtimes – Reviews or film showtimes can be listed for any movies playing nearby,by typing "movies" or the name of any current film into the search box. If a specific location was saved on a previous search, the top search result will display showtimes for nearby theaters for that movie.
public data – Trends for population (or unemployment rates) can be found for U.S. states & counties, by typing "population" or "unemployment rate" followed by a state or county name.
real estate and housing – Home listings in a given area can be displayed,using the trigger words "housing", "home", or "real estate" followed by the name of a city or U.S. zip code.
travel data/airports – The flight status for arriving or departing U.S. flights can be displayed,by typing in the name of the airline and the flight number into the search box (such as: American airlines 18). Delays at a specific airport can also be viewed (by typing the name of the city or three-letter airport code plus word "airport").
package tracking – Package mail can be tracked by typing the tracking number of a Royal Mail, UPS, FedEx or USPS package directly into the search box. Results will include quick links to track the status of each shipment.
patent numbers – U.S. patents can be searched by entering the word "patent" followed by the patent number into the search box (such as: Patent 5123123).
area code – The geographical location (for any U.S. telephone area code) can be displayed by typing a 3-digit area code (such as: 650).
synonym search – A search can match words similar to those specified,by placing the tilde sign (~) immediately in front of a search term, such as: ~fast food.

Search options

The webpages maintained by the Google Help Center have text describing more than 15 various search options.The Google operators:

OR – Search for either one, such as "price high OR low" searches for "price" with "high" or "low".
"-" – Search while excluding a word, such as "apple -tree" searches where word "tree" is not used.
"+" – Force inclusion of a word, such as "Name +of +the Game" to require the words "of" & "the" to appear on a matching page.
"*" – Wildcard operator to match any words between other specific words.

Some of the query options are as follows:

define: – The query prefix "define:" will provide a definition of the words listed after it.
stocks: – After "stocks:" the query terms are treated as stock ticker symbols for lookup.
site: – Restrict the results to those websites in the given domain,such as, site:www.acmeacme.com. The option "site:com" will search all domain URLs named with ".com" (no space after "site:").
allintitle: – Only the page titles are searched (not the remaining text on each webpage).
intitle: – Prefix to search in a webpage title,such as "intitle:google search" will list pages with word "google" in title, and word "search" anywhere (no space after "intitle:").
allinurl: – Only the page URL address lines are searched (not the text inside each webpage).
inurl: – Prefix for each word to be found in the URL;others words are matched anywhere, such as "inurl:acme search" matches "acme" in a URL, but matches "search" anywhere (no space after "inurl:").

The page-display options (or query types) are:

cache: – Highlights the search-words within the cached document, such as "cache:www.google.com xxx" shows cached content with word "xxx" highlighted.
link: – The prefix "link:" will list webpages that have links to the specified webpage, such as "link:www.google.com" lists webpages linking to the Google homepage.
related: – The prefix "related:" will list webpages that are "similar" to a specified web page.
info: – The prefix "info:" will display some background information about one specified webpage, such as, info:www.google.com. Typically, the info is the first text (160 bytes, about 23 words) contained in the page, displayed in the style of a results entry (for just the 1 page as matching the search).
filetype: – results will only show files of the desired type (ex filetype:pdf will return pdf files)

Note that Google searches the HTML coding inside a webpage, not the screen appearance: the words displayed on a screen might not be listed in the same order in the HTML coding.

Error messages

Some searches will give a 403 Forbidden error with the text

"We're sorry... ... but your query looks similar to automated requests from a computer virus or spyware application. To protect our users, we can't process your request right now. We'll restore your access as quickly as possible, so try again soon. In the meantime, if you suspect that your computer or network has been infected, you might want to run a virus checker or spyware remover to make sure that your systems are free of viruses and other spurious software. We apologize for the inconvenience, and hope we'll see you again on Google."

sometimes followed by a CAPTCHA prompt.

Google's Server Error page

The screen was first reported in 2005, and was a response to the heavy use of Google by search engine optimization companies to check on ranks of sites they were optimizing. The message is triggered by high volumes of requests from a single IP address. Google apparently uses the Google cookie as part of its determination of refusing service.
In June 2009, after the death of pop superstar Michael Jackson, this message appeared to many internet users who were searching Google for news stories related to the singer, and was assumed by Google to be a DDoS attack, although many queries were submitted by legitimate searchers.

January 2009 malware bug

A screen-shot of the error of January 31, 2009.

Google flags search results with the message "This site may harm your computer" if the site is known to install malicious software in the background or otherwise surreptitiously. Google does this to protect users against visiting sites that could harm their computers. For approximately 40 minutes on January 31, 2009, all search results were mistakenly classified as malware and could therefore not be clicked; instead a warning message was displayed and the user was required to enter the requested URL manually. The bug was caused by human error.The URL of "/" (which expands to all URLs) was mistakenly added to the malware patterns file.

Doodle for Google

On certain occasions, the logo on Google's webpage will change to a special version, known as a "Google Doodle". Clicking on the Doodle links to a string of Google search results about the topic. The first was a reference to the Burning Man Festival in 1998,and others have been produced for the birthdays of notable people like Albert Einstein, historical events like the interlocking Lego block's 50th anniversary and holidays like Valentine's Day.

Google Caffeine

In August 2009, Google announced the rollout of a new search architecture, codenamed "Caffeine".The new architecture was designed to return results faster and to better deal with rapidly updated information from services including Facebook and Twitter.Google developers noted that most users would notice little immediate change, but invited developers to test the new search in its sandbox.Differences noted for their impact upon search engine optimization included heavier keyword weighting and the importance of the domain's age.The move was interpreted in some quarters as a response to Microsoft's recent release of an upgraded version of its own search service, renamed Bing.Google announced completion of Caffeine on 8 June 2010, claiming 50% fresher results due to continuous updating of its index.With Caffeine, Google moved its back-end indexing system away from MapReduce and onto BigTable, the company's distributed database platform.Caffeine is also based on Colossus, or GFS2,an overhaul of the GFS distributed file system.

Encrypted Search

In May 2010 Google rolled out SSL-encrypted web search.The encrypted search can be accessed at encrypted.google.com

Instant Search

Google Instant, an enhancement that displays suggested results while the user types, was introduced on September 8, 2010. One concern is that people will select one of the suggested results instead of finishing their request, and that such a practice could cause bias toward familiar businesses or other search terms. Pornographic or otherwise offensive search terms are excluded from the suggested results. The instant feature appears only on the basic Google site and not specialized iGoogle pages. Google expects Google Instant to save users 2 to 5 seconds in every search, which they say will be collectively 11 million seconds per hour.Search engine marketing pundits speculate that Google Instant will have a great impact on local and paid search.
In concert with the Google Instant launch, Google disabled the ability of users to choose to see more than 10 search results per page. Instant Search can be disabled via Google's "preferences" menu, but autocomplete-style search suggestions now cannot be disabled. A Google representative stated, "It's in keeping with our vision of a unified Google search experience to make popular, useful features part of the default experience, rather than maintain different versions of Google. As Autocomplete quality has improved, we felt it was appropriate to have it always on for all of our users."

Negative reception

Many users have reported being unable to save the Instant Search "off" setting in their Google preferences.

Censorship

The publication 2600: The Hacker Quarterly has compiled a list of words that are restricted by Google Instant.These are terms the web giant's new instant search feature will not search.Most terms are often vulgar and derogatory in nature, but some apparently irrelevant searches including "Myleak" are removed.

Features

June 2011 redesign

In late June 2011, Google introduced a new look to the Google home page in order to boost the use of the Google+ social tools.

Google +1

+1 helps people discover relevant content aimed to increase Google search result status offering a status to show people know and trust the content. When a signed-in Google user is searching, Google search result snippet will show a +1 button to recommend the page and an annotation with the names of the user's connections who've +1'd your page.

Interface features

Simple white background with time-to-time changes in the title, “Google” with a special historic day or to celebrate a certain day.
Top bar has Web, Images, Videos, Maps, News, Shopping, Gmail, and more.
Voice search, allowing faster input than typing, or if the correct spelling is not known.
When signed into your Google account, your search history will be automatically recorded.
Google Instant, which rapidly generates possible searches that contain the typed characters. For example if you typed Goo, it would display Google, Google maps, Google translate...

Media features

Share your own picture to the world. You can upload your own picture.
Image search with optional settings such as size, color, type, and sorting.
Video search that is connected to YouTube. Also with optional setting such as duration, time, quality, also other sources that is related to the topic that you are searching.

Navigation bar design

One of the major changes was replacing the classic navigation bar with a black one. Google's digital creative director Chris Wiggins explains: "We're working on a project to bring you a new and improved Google experience, and over the next few months, you'll continue to see more updates to our look and feel."The new navigation bar has been negatively received by a vocal majority.

Google logo size reduction

The new design reduced the size of the Google logo.

Move of links

Links for advertising, business partners and company information pushed to the bottom edges of the browser.

International

Google is available in many languages and has been localized completely or partly for many countries.

Languages

The interface has also been made available in some languages for humorous purpose:

Domain names

In addition to the main URL Google.com, Google Inc. owns 160 domain names for each of the countries/regions in which it has been localized.

Search products

In addition to its tool for searching webpages, Google also provides services for searching images, Usenet newsgroups, news websites, videos, searching by locality, maps, and items for sale online. In 2006, Google has indexed over 25 billion web pages,400 million queries per day,1.3 billion images, and over one billion Usenet messages. It also caches much of the content that it indexes. Google operates other tools and services including Google News, Google Suggest, Google Product Search, Google Maps, Google Co-op, Google Earth, Google Docs, Picasa, Panoramio, YouTube, Google Translate, Google Blog Search and Google Desktop Search.
There are also products available from Google that are not directly search-related. Gmail, for example, is a webmail application, but still includes search features; Google Browser Sync does not offer any search facilities, although it aims to organize your browsing time.
Also Google starts many new beta products, like Google Social Search or Google Image Swirl.

Energy consumption

Google claims that a search query requires altogether about 1 kJ or 0.0003 kW·h

Search Engine

2011-09-09T11:00:00.000-07:00

PageRank

Google's rise to success was in large part due to a patented algorithm called PageRank that helps rank web pages that match a given search string.When Google was a Stanford research project, it was nicknamed BackRub because the technology checks backlinks to determine a site's importance. Previous keyword-based methods of ranking search results, used by many search engines that were once more popular than Google, would rank pages by how often the search terms occurred in the page, or how strongly associated the search terms were within each resulting page. The PageRank algorithm instead analyzes human-generated links assuming that web pages linked from many important pages are themselves likely to be important. The algorithm computes a recursive score for pages, based on the weighted sum of the PageRanks of the pages linking to them. PageRank is thought to correlate well with human concepts of importance. In addition to PageRank, Google, over the years, has added many other secret criteria for determining the ranking of pages on result lists, reported to be over 200 different indicators.The specifics of which are kept secret to keep spammers at bay and facilitate Google maintain an edge over its competitors globally.

Search results

The exact percentage of the total of web pages that Google indexes is not known, as it is very difficult to accurately calculate. Google not only indexes and caches web pages, but also takes "snapshots" of other file types, which include PDF, Word documents, Excel spreadsheets, Flash SWF, plain text files, and so on. Except in the case of text and SWF files, the cached version is a conversion to (X)HTML, allowing those without the corresponding viewer application to read the file.
Users can customize the search engine, by setting a default language, using the "SafeSearch" filtering technology and set the number of results shown on each page. Google has been criticized for placing long-term cookies on users' machines to store these preferences, a tactic which also enables them to track a user's search terms and retain the data for more than a year. For any query, up to the first 1000 results can be shown with a maximum of 100 displayed per page. The ability to specify the number of results is available only if "Instant Search" is not enabled. If "Instant Search" is enabled, only 10 results are displayed, regardless of this setting.

Non-indexable data

Despite its immense index, there is also a considerable amount of data available in online databases which are accessible by means of queries but not by links. This so-called invisible or deep Web is minimally covered by Google and other search engines.The deep Web contains library catalogs, official legislative documents of governments, phone books, and other content which is dynamically prepared to respond to a query.

Google optimization

Main article: Search engine optimization

Since Google is the most popular search engine, many webmasters have become eager to influence their website's Google rankings. An industry of consultants has arisen to help websites increase their rankings on Google and on other search engines. This field, called search engine optimization, attempts to discern patterns in search engine listings, and then develop a methodology for improving rankings to draw more searchers to their client's sites.
Search engine optimization encompasses both "on page" factors (like body copy, title elements, H1 heading elements and image alt attribute values) and Off Page Optimization factors (like anchor text and PageRank). The general idea is to affect Google's relevance algorithm by incorporating the keywords being targeted in various places "on page", in particular the title element and the body copy (note: the higher up in the page, presumably the better its keyword prominence and thus the ranking). Too many occurrences of the keyword, however, cause the page to look suspect to Google's spam checking algorithms.
Google has published guidelines for website owners who would like to raise their rankings when using legitimate optimization consultants.
It has been hypothesized, and, allegedly, is the opinion of the owner of one business about which there has been numerous complaints, that negative publicity, for example, numerous consumer complaints, may serve as well to elevate page rank on Google Search as favorable comments.The particular problem addressed in The New York Times article, which involved DecorMyEyes, was addressed shortly thereafter by an undisclosed fix in the Google algorithm. According to Google, it was not the frequently published consumer complaints about DecorMyEyes which resulted in the high ranking but mentions on news websites of events which affected the firm such as legal actions against it.There has also been a Google toolbar for use on many internet browsers.

Google Search

2011-09-09T10:00:00.000-07:00



Google Search homepage
URL	http://www.google.com/
Commercial?	Yes
Type of site	Web search engine
Registration	Optional
Available language(s)	Multilingual (124)
Owner	Google
Created by	Sergey Brin and Larry Page
Launched	September 15, 1997
Alexa rank	1 (September 2011)
Revenue	From AdWords

Google Search or Google Web Search is a web search engine owned by Google Inc. Google Search is the most-used search engine on the World Wide Web, receiving several hundred million queries each day through its various services.
The order of search results on Google's search-results pages is based, in part, on a priority rank called a "PageRank". Google Search provides many options for customized search, using Boolean operators such as: exclusion ("-xx"), inclusion ("+xx"), alternatives ("xx OR yy"), and wildcard ("x * x").
The main purpose of Google Search is to hunt for text in Web pages, as opposed to other data, such as with Google Image Search. Google Search was originally developed by Larry Page and Sergey Brin in 1997. Google Search provides at least 22 special features beyond the original word-search capability.These include synonyms, weather forecasts, time zones, stock quotes, maps, earthquake data, movie showtimes, airports, home listings, and sports scores. There are special features for numbers, including ranges (70..73), prices, temperatures, money/unit conversions ("10.5 cm in inches"), calculations ("3*4+sqrt(6)-pi/2"), package tracking, patents, area codes,and language translation of displayed pages. In June 2011, Google introduced "Google Voice Search" and "Search by Image" features for allowing the users to search words by speaking and by giving images.
The frequency of use of many search terms have reached a volume that they may indicate broader economic, social and health trends.Data about the frequency of use of search terms on Google (available through Google Adwords, Google Trends, and Google Insights for Search) have been shown to correlated with flu outbreaks and unemployment levels and provide the information faster than traditional reporting methods and government surveys.

SEO contest

2011-09-09T09:00:00.000-07:00

An SEO contest is a prize-awarding activity which challenges search engine optimization (SEO) practitioners to achieve high ranking under major search engines such as Google, Yahoo, and MSN using certain keyword(s). This type of contest is controversial because it often leads to massive amounts of link spamming as participants try to boost the rankings of their pages by any means available.
The organizing body of an SEO competition may hold the activity without promotion of a product or service in mind; or they may organize a contest in order to market something on the Internet. Participants can showcase their skills and potentially discover and share new techniques for promoting websites.

History

The first recorded SEO Contest was Schnitzelmitkartoffelsalat by German webmasters, started on 2002 November 15 in the German-language usenet group de.comm.infosystems.www.authoring.misc. In the English-language world, the nigritude ultramarine competition by SearchGuild is widely acclaimed as the mother of all SEO contests. It was started on May 7, 2004, and was won two months later by Anil Dash. On September 1 of the same year, webmasters were challenged to rank number 1 on Google in three months' time for the search phrase seraphim proudleduck.
In the first quarter of 2005, people were competing for the term loquine glupe, spawning web sites ranging from shampoo advertising to holiday resorts. The page that won in the end looked rather boring, and used lots of questionable techniques like "keyword stuffing".
Internationally, in 2005 two major contests took place in Europe. In Germany the Hommingberger Gepardenforelle by the computer magazine spawned almost 4 million results. The goal was to find out how search engines rank sites. In Poland almost at same time the Polish SEO community organized the msnbetter thangoogle contest. It topped the 4 million but failed to reach its goal to promote SEO in Poland and to get search engines companies' attention for the Polish market. Some current and pending contests are listed below.
A competition ran from January 1, 2006, to March 1, 2006, and carried the term redscowl bluesingsky, another set of made-up words. It was sponsored by SEOLogs. Shoemoney won this contest, and since he contributed the winner's money, he donated it to the number 2 winner.
Since then, SEO contests have become a part of some academic classes. In 2008 Luis von Ahn at CMU created a contest for his students, and in 2010 Adam Wierman picked it up at Caltech.

Tactics

Some webmasters resort to spam, while others use white-hat optimization techniques, like providing good content covering the competition, or optimizing page titles.Most SEO contests expect people to optimize a single web page for a non-existent phrase of two silly words. This is to keep existing web sites from getting a head start and to make sure that regular internet searchers will not be shown contest pages when searching the web for other information.
Rules and limitations can make it harder to benefit from the ranking algorithm of the targeted search engine. The January 2006 Redscowl Bluesingsky contest issued by seologs.com was open for domains created after the start of the competition only. This meant that the contestants could not benefit from the ranking advantage old web sites have over new ones. Also, it was expected that the Redscowl Bluesingsky game would be won by a domain made up entirely of the search words, such as "redscowl-bluesingsky.com", which would attract natural links and be likely to benefit from the simplicity of the URL.
Another special rule that fits well with the 'purpose' of SEO contests today is the obligation to 'link back' to the organizing body, often a search engine optimization site. Since a web document's ranking on major search engines like Yahoo!, Google or MSN Search is mainly determined by internet hyperlinks pointing to that document, forcing webmasters to link to a web site is quite a powerful way to increase its web presence. Good example are the contest announced by V7N (using the phrase v7ndotcom elursrebmem) and its counterpart by WebGuerrilla. While the first of these originally required the contestants to link to V7N forums, the second forbids its players to do just that. Instead a special link to Google engineer Matt Cutts' blog is imperative. Because of this rivalry, both the rules and prize money on both these www.SEO-sk.com contests were updated regularly up until the official start date of January 15, 2006.

Index term

2011-09-08T11:00:00.000-07:00

An index term, subject term, subject heading, or descriptor, in information retrieval, is a term that captures the essence of the topic of a document. Index terms make up a controlled vocabulary for use in bibliographic records. They are an integral part of bibliographic control, which is the function by which libraries collect, organize and disseminate documents. They are used as keywords to retrieve documents in an information system, for instance, a catalog or a search engine. A popular form of keywords on the web are tags which are directly visible and can be assigned by non-experts also. Index terms can consist of a word, phrase, or alphanumerical term. They are created by analyzing the document either manually with subject indexing or automatically with automatic indexing or more sophisticated methods of keyword extraction. Index terms can either come from a controlled vocabulary or be freely assigned.
Keywords are stored in a search index. Common words like articles (a, an, the) and conjunctions (and, or, but) are not treated as keywords because it is inefficient to do so. Almost every English-language site on the Internet has the article "the", and so it makes no sense to search for it. The most popular search engine, Google removed stop words such as "the" and "a" from its indexes for several years, but then re-introduced them, making certain types of precise search possible again.
The term "descriptor" was coined by Calvin Mooers in 1948.
The Simple Knowledge Organisation System language (SKOS) provides a way to express index terms with Resource Description Framework for use in the context of Semantic Web.

Keyword stuffing

2011-09-08T10:00:00.000-07:00

Keyword stuffing is considered to be an unethical search engine optimization (SEO) technique. Keyword stuffing occurs when a web page is loaded with keywords in the meta tags or in content. The repetition of words in meta tags may explain why many search engines no longer use these tags.
Keyword stuffing had been used in the past to obtain maximum search engine ranking and visibility for particular phrases. This method is completely outdated and adds no value to rankings today. In particular, Google no longer gives good rankings to pages employing this technique.
Hiding text from the visitor is done in many different ways. Text colored to blend with the background, CSS "Z" positioning to place text "behind" an image — and therefore out of view of the visitor — and CSS absolute positioning to have the text positioned far from the page center are all common techniques. By 2005, many invisible text techniques were easily detected by major search engines.
"Noscript" tags are another way to place hidden content within a page. While they are a valid optimization method for displaying an alternative representation of scripted content, they may be abused, since search engines may index content that is invisible to most visitors.
Sometimes inserted text includes words that are frequently searched (such as "sex"), even if those terms bear little connection to the content of a page, in order to attract traffic to advert-driven pages.
In the past, keyword stuffing was considered to be either a white hat or a black hat tactic, depending on the context of the technique, and the opinion of the person judging it. While a great deal of keyword stuffing was employed to aid in spamdexing, which is of little benefit to the user, keyword stuffing in certain circumstances was not intended to skew results in a deceptive manner. Whether the term carries a pejorative or neutral connotation is dependent on whether the practice is used to pollute the results with pages of little relevance, or to direct traffic to a page of relevance that would have otherwise been de-emphasized due to the search engine's inability to interpret and understand related ideas. This is no longer the case. Search engines now employ themed, related keyword techniques to interpret the intent of the content on a page.
With relevance to keyword stuffing, it is quoted by the largest of search engines that they recommend Keyword Research and use (with respect to the quality content you have to offer the web), to aid their visitors in the search of your valuable material. To prevent Keyword Stuffing you should wisely use keywords in respect with SEO, Search Engine Optimization. It could be best described as keywords should be reasonable and necessary, yet it is acceptable to assist with proper placement and your targeted effort to achieve search results. Placement of such words in the provided areas of HTML are perfectly allowed and reasonable. Google discusses keyword stuffing as Randomly Repeated Keywords.

In online journalism

Headlines in online news sites are increasingly packed with just the search-friendly keywords that identify the story. Puns and plays on words have gone by the wayside. Overusing this strategy is also called keyword stuffing. Old-school reporters and editors frown on the practice, but it is effective in optimizing news stories for search.

Keyword density

2011-09-08T09:00:00.000-07:00

Keyword density is the percentage of times a keyword or phrase appears on a web page compared to the total number of words on the page. In the context of search engine optimization keyword density can be used as a factor in determining whether a web page is relevant to a specified keyword or keyword phrase.
In the late 1990s, which was the early days of search engines, keyword density was an important factor in how a page was ranked. However, as webmasters discovered this and the implementation of optimum keyword density became widespread, it became a minor factor in the rankings. Search engines began giving priority to other factors that are beyond the direct control of webmasters. Today, the overuse of keywords, a practice called keyword stuffing, will cause a web page to be penalized.
Many SEO experts consider the optimum keyword density to be 1 to 3 percent. Using a keyword more than that could be considered search spam. The formula to calculate your keyword density on a web page for SEO purposes is

(N k r / T k n) * 100

,where Nkr is how many times you repeated a specific keyword and Tkn the total words in the analyzed text. This will result in a keyword density value. When calculating keyword density, be sure to ignore html tags and other embedded tags which will not actually appear in the text of the page once it is published.
When calculating the density of a keyword phrase, the formula would be

(N k r * N w p / T k n) * 100

, where Nwp is the number of words in the phrase. So, for example, for a page about search engine optimization where that phrase is used four times and there are four hundred words on the page, the keyword phrase density is (4*3/400)*100 or 3 percent.
However, from a purely mathematical viewpoint, one cannot ignore the fact that the original concept of keyword density refers to the frequency (Nkr) of appearance of a particular keyword in a dissertation. Thus, a "keyword" consisting of multiple terms, e.g. "blue suede shoes" should be considered an entity in itself. It is the frequency of the phrase "blue suede shoes" within a dissertation that drives the key(phrase) density. Thus it is "more" mathematically correct for a "keyphrase" to be calculated just like the original calculation, but considering the word group, "blue suede shoes," as a single appearance, not three. Thus:
Density = ( Nkr / Tkn ) * 100.
Furthermore, under closer inspection, one can see that these 'keywords' (kr) that actually consist of several words, artificially inflate the total word count of the dissertation. Therefore, it could be argued that the purest mathematical representation should adjust the total word count (Tkn) lower by removing the excess key(phrase) word counts from the total. Thus:
Density = ( Nkr / ( Tkn -( Nkr * ( Nwp-1 ) ) ) ) * 100. where Nwp = the number of terms in the keyphrase.
This general formula allows that the total word count will be unaffected if the key(phrase) is indeed a single term, so it acts as the original formula.

Online keyword density checkers

Keyword Density Analyser - Checks keyword density for any url and allows customization in the form of words to ignore on the page, minimum occurrences and meta tag exclusion.
Keyword Density Checker - Examines the density of the keywords
Keyword Density Checker - Checks keyword density for any url and allows customization in the form of words to ignore on the page, minimum occurrences and meta tag exclusion.
Latest SEO News - Helps webmasters analyze the keyword density from their web pages
Search Engine Optimization - Helps webmasters and Search Engine Optimizers achieve their optimum keyword density for a set of key terms.
Seo Tools - Extracts text, removes common stop words and Analyze the density of the keywords.
Seo Tools - Analyze the density of the keywords and suggest the most relevant keywords.
On-page SEO Tools - Analyze page structure, analyze the content and find SEO errors.
WordPress On-Page SEO Analyzer - WordPress based on-page SEO analyzer.

Offline keyword density checkers

CSE HTML Validator for Windows - Checks for search engine optimization issues and provides keyword density information.

Search engine optimization(SEO)

2011-09-07T11:00:00.000-07:00

Search engine optimization (SEO) is the process of improving the visibility of a website or a web page in search engines via the "natural" or un-paid ("organic" or "algorithmic") search results. In general, the earlier (or higher on the page), and more frequently a site appears in the search results list, the more visitors it will receive from the search engine's users. SEO may target different kinds of search, including image search, local search, video search, academic search,news search and industry-specific vertical search engines.
As an Internet marketing strategy, SEO considers how search engines work, what people search for, the actual search terms typed into search engines and which search engines are preferred by their targeted audience. Optimizing a website may involve editing its content and HTML and associated coding to both increase its relevance to specific keywords and to remove barriers to the indexing activities of search engines. Promoting a site to increase the number of backlinks, or inbound links, is another SEO tactic.
The acronym "SEOs" can refer to "search engine optimizers," a term adopted by an industry of consultants who carry out optimization projects on behalf of clients, and by employees who perform SEO services in-house. Search engine optimizers may offer SEO as a stand-alone service or as a part of a broader marketing campaign. Because effective SEO may require changes to the HTML source code of a site and site content, SEO tactics may be incorporated into website development and design. The term "search engine friendly" may be used to describe website designs, menus, content management systems, images, videos, shopping carts, and other elements that have been optimized for the purpose of search engine exposure.
Another class of techniques, known as black hat SEO, search engine poisoning, or spamdexing, uses methods such as link farms, keyword stuffing and article spinning that degrade both the relevance of search results and the quality of user-experience with search engines. Search engines look for sites that employ these techniques in order to remove them from their indices.

History

Webmasters and content providers began optimizing sites for search engines in the mid-1990s, as the first search engines were cataloging the early Web. Initially, all webmasters needed to do was submit the address of a page, or URL, to the various engines which would send a "spider" to "crawl" that page, extract links to other pages from it, and return information found on the page to be indexed.The process involves a search engine spider downloading a page and storing it on the search engine's own server, where a second program, known as an indexer, extracts various information about the page, such as the words it contains and where these are located, as well as any weight for specific words, and all links the page contains, which are then placed into a scheduler for crawling at a later date.
Site owners started to recognize the value of having their sites highly ranked and visible in search engine results, creating an opportunity for both white hat and black hat SEO practitioners. According to industry analyst Danny Sullivan, the phrase "search engine optimization" probably came into use in 1997.The first documented use of the term Search Engine Optimization was John Audette and his company Multimedia Marketing Group as documented by a web page from the MMG site from August, 1997^.
Early versions of search algorithms relied on webmaster-provided information such as the keyword meta tag, or index files in engines like ALIWEB. Meta tags provide a guide to each page's content. Using meta data to index pages was found to be less than reliable, however, because the webmaster's choice of keywords in the meta tag could potentially be an inaccurate representation of the site's actual content. Inaccurate, incomplete, and inconsistent data in meta tags could and did cause pages to rank for irrelevant searches.Web content providers also manipulated a number of attributes within the HTML source of a page in an attempt to rank well in search engines.
By relying so much on factors such as keyword density which were exclusively within a webmaster's control, early search engines suffered from abuse and ranking manipulation. To provide better results to their users, search engines had to adapt to ensure their results pages showed the most relevant search results, rather than unrelated pages stuffed with numerous keywords by unscrupulous webmasters. Since the success and popularity of a search engine is determined by its ability to produce the most relevant results to any given search, allowing those results to be false would turn users to find other search sources. Search engines responded by developing more complex ranking algorithms, taking into account additional factors that were more difficult for webmasters to manipulate.
Graduate students at Stanford University, Larry Page and Sergey Brin, developed "backrub," a search engine that relied on a mathematical algorithm to rate the prominence of web pages. The number calculated by the algorithm, PageRank, is a function of the quantity and strength of inbound links.PageRank estimates the likelihood that a given page will be reached by a web user who randomly surfs the web, and follows links from one page to another. In effect, this means that some links are stronger than others, as a higher PageRank page is more likely to be reached by the random surfer.
Page and Brin founded Google in 1998. Google attracted a loyal following among the growing number of Internet users, who liked its simple design.Off-page factors (such as PageRank and hyperlink analysis) were considered as well as on-page factors (such as keyword frequency, meta tags, headings, links and site structure) to enable Google to avoid the kind of manipulation seen in search engines that only considered on-page factors for their rankings. Although PageRank was more difficult to game, webmasters had already developed link building tools and schemes to influence the Inktomi search engine, and these methods proved similarly applicable to gaming PageRank. Many sites focused on exchanging, buying, and selling links, often on a massive scale. Some of these schemes, or link farms, involved the creation of thousands of sites for the sole purpose of link spamming.
By 2004, search engines had incorporated a wide range of undisclosed factors in their ranking algorithms to reduce the impact of link manipulation. Google says it ranks sites using more than 200 different signals.The leading search engines, Google, Bing, and Yahoo, do not disclose the algorithms they use to rank pages. Notable SEO service providers, such as Rand Fishkin, Barry Schwartz, Aaron Wall and Jill Whalen, have studied different approaches to search engine optimization, and have published their opinions in online forums and blogs.SEO practitioners may also study patents held by various search engines to gain insight into the algorithms.
In 2005 Google began personalizing search results for each user. Depending on their history of previous searches, Google crafted results for logged in users.In 2008, Bruce Clay said that "ranking is dead" because of personalized search. It would become meaningless to discuss how a website ranked, because its rank would potentially be different for each user and each search.
In 2007 Google announced a campaign against paid links that transfer PageRank.On June 15, 2009, Google disclosed that they had taken measures to mitigate the effects of PageRank sculpting by use of the nofollow attribute on links. Matt Cutts, a well-known software engineer at Google, announced that Google Bot would no longer treat nofollowed links in the same way, in order to prevent SEO service providers from using nofollow for PageRank sculpting.As a result of this change the usage of nofollow leads to evaporation of pagerank. In order to avoid the above, SEO engineers developed alternative techniques that replace nofollowed tags with obfuscated Javascript and thus permit PageRank sculpting. Additionally several solutions have been suggested that include the usage of iframes, Flash and Javascript.
In December 2009 Google announced it would be using the web search history of all its users in order to populate search results.
The Real-time-search was introduced in late 2009 in an attempt to make search results more timely and relevant. Historically site administrators have spent months or even years optimizing a website to increase search rankings. With the growth in popularity of social media sites and blogs the leading engines made changes to their algorithms to allow fresh content to rank quickly within the search results.

Relationship with search engines

Yahoo and Google offices

By 1997 search engines recognized that webmasters were making efforts to rank well in their search engines, and that some webmasters were even manipulating their rankings in search results by stuffing pages with excessive or irrelevant keywords. Early search engines, such as Altavista and Infoseek, adjusted their algorithms in an effort to prevent webmasters from manipulating rankings.
Due to the high marketing value of targeted search results, there is potential for an adversarial relationship between search engines and SEO service providers. In 2005, an annual conference, AIRWeb, Adversarial Information Retrieval on the Web,was created to discuss and minimize the damaging effects of aggressive web content providers.
Companies that employ overly aggressive techniques can get their client websites banned from the search results. In 2005, the Wall Street Journal reported on a company, Traffic Power, which allegedly used high-risk techniques and failed to disclose those risks to its clients.Wired magazine reported that the same company sued blogger and SEO Aaron Wall for writing about the ban.Google's Matt Cutts later confirmed that Google did in fact ban Traffic Power and some of its clients.
Some search engines have also reached out to the SEO industry, and are frequent sponsors and guests at SEO conferences, chats, and seminars. In fact, with the advent of paid inclusion, some search engines now have a vested interest in the health of the optimization community. Major search engines provide information and guidelines to help with site optimization.Google has a Sitemaps program to help webmasters learn if Google is having any problems indexing their website and also provides data on Google traffic to the website. Google guidelines are a list of suggested practices Google has provided as guidance to webmasters. Yahoo! Site Explorer provides a way for webmasters to submit URLs, determine how many pages are in the Yahoo! index and view link information.Bing Toolbox provides a way from webmasters to submit a sitemap and web feeds, allowing users to determine the crawl rate, and how many pages have been indexed by their search engine.

Getting indexed

The leading search engines, such as Google, Bing and Yahoo!, use crawlers to find pages for their algorithmic search results. Pages that are linked from other search engine indexed pages do not need to be submitted because they are found automatically. Some search engines, notably Yahoo!, operate a paid submission service that guarantee crawling for either a set fee or cost per click.Such programs usually guarantee inclusion in the database, but do not guarantee specific ranking within the search results.Two major directories, the Yahoo Directory and the Open Directory Project both require manual submission and human editorial review.Google offers Google Webmaster Tools, for which an XML Sitemap feed can be created and submitted for free to ensure that all pages are found, especially pages that aren't discoverable by automatically following links.
Search engine crawlers may look at a number of different factors when crawling a site. Not every page is indexed by the search engines. Distance of pages from the root directory of a site may also be a factor in whether or not pages get crawled.Additionally, search engines sometimes have problems with crawling sites with certain kinds of graphic content, flash files, portable document format files, and dynamic content.

Preventing crawling

To avoid undesirable content in the search indexes, webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a meta tag specific to robots. When a search engine visits a site, the robots.txt located in the root directory is the first file crawled. The robots.txt file is then parsed, and will instruct the robot as to which pages are not to be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wish crawled. Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches. In March 2007, Google warned webmasters that they should prevent indexing of internal search results because those pages are considered search spam.

Increasing prominence

A variety of methods can increase the prominence of a webpage within the search results. Cross linking between pages of the same website to provide more links to most important pages may improve its visibility.Writing content that includes frequently searched keyword phrase, so as to be relevant to a wide variety of search queries will tend to increase traffic.Updating content so as to keep search engines crawling back frequently can give additional weight to a site. Adding relevant keywords to a web page's meta data, including the title tag and meta description, will tend to improve the relevancy of a site's search listings, thus increasing traffic. URL normalization of web pages accessible via multiple urls, using the "canonical" meta tagor via 301 redirects can help make sure links to different versions of the url all count towards the page's link popularity score.

SEO techniques are classified by some into two broad categories: techniques that search engines recommend as part of good design, and those techniques that search engines do not approve of and attempt to minimize the effect of, referred to as spamdexing. Some industry commentators classify these methods, and the practitioners who employ them, as either white hat SEO, or black hat SEO.White hats tend to produce results that last a long time, whereas black hats anticipate that their sites will eventually be banned once the search engines discover what they are doing.
An SEO tactic, technique or method is considered white hat if it conforms to the search engines' guidelines and involves no deception. As the search engine guidelines are not written as a series of rules or commandments, this is an important distinction to note. White hat SEO is not just about following guidelines, but is about ensuring that the content a search engine indexes and subsequently ranks is the same content a user will see.
White hat advice is generally summed up as creating content for users, not for search engines, and then making that content easily accessible to the spiders, rather than attempting to game the algorithm. White hat SEO is in many ways similar to web development that promotes accessibility,although the two are not identical.
White Hat SEO is merely effective marketing, making efforts to deliver quality content to an audience that has requested the quality content. Traditional marketing means have allowed this through transparency and exposure. A search engine's algorithm takes this into account, such as Google's PageRank.
Black hat SEO attempts to improve rankings in ways that are disapproved of by the search engines, or involve deception. One black hat technique uses text that is hidden, either as text colored similar to the background, in an invisible div, or positioned off screen. Another method gives a different page depending on whether the page is being requested by a human visitor or a search engine, a technique known as cloaking.
Search engines may penalize sites they discover using black hat methods, either by reducing their rankings or eliminating their listings from their databases altogether. Such penalties can be applied either automatically by the search engines' algorithms, or by a manual site review. One infamous example was the February 2006 Google removal of both BMW Germany and Ricoh Germany for use of deceptive practices.Both companies, however, quickly apologized, fixed the offending pages, and were restored to Google's list.
Additionally, many professionals in the SEO industry refer to "gray hat" tactics that may skirt the lines of black and white hat tactics. Numerous references to gray hat techniques have been published, and these usually constitute practices that are not strictly disapproved by search engines, but may go against the spirit of the regulations that search engines have laid out.

As a marketing strategy

SEO is not an appropriate strategy for every website, and other Internet marketing strategies can be more effective, depending on the site operator's goals.A successful Internet marketing campaign may also depend upon building high quality web pages to engage and persuade, setting up analytics programs to enable site owners to measure results, and improving a site's conversion rate.
SEO may generate an adequate return on investment. However, search engines are not paid for organic search traffic, their algorithms change, and there are no guarantees of continued referrals. Due to this lack of guarantees and certainty, a business that relies heavily on search engine traffic can suffer major losses if the search engines stop sending visitors.It is considered wise business practice for website operators to liberate themselves from dependence on search engine traffic.Seomoz.org has suggested that "search marketers, in a twist of irony, receive a very small share of their traffic from search engines." Instead, their main sources of traffic are links from other websites.

International markets

Optimization techniques are highly tuned to the dominant search engines in the target market. The search engines' market shares vary from market to market, as does competition. In 2003, Danny Sullivan stated that Google represented about 75% of all searches.In markets outside the United States, Google's share is often larger, and Google remains the dominant search engine worldwide as of 2007.As of 2006, Google had an 85-90% market share in Germany.While there were hundreds of SEO firms in the US at that time, there were only about five in Germany.As of June 2008, the marketshare of Google in the UK was close to 90% according to Hitwise.That market share is achieved in a number of countries.
As of 2009, there are only a few large markets where Google is not the leading search engine. In most cases, when Google is not leading in a given market, it is lagging behind a local player. The most notable markets where this is the case are China, Japan, South Korea, Russia and the Czech Republic where respectively Baidu, Yahoo! Japan, Naver, Yandex and Seznam are market leaders.
Successful search optimization for international markets may require professional translation of web pages, registration of a domain name with a top level domain in the target market, and web hosting that provides a local IP address. Otherwise, the fundamental elements of search optimization are essentially the same, regardless of language.

Legal precedents

On October 17, 2002, SearchKing filed suit in the United States District Court, Western District of Oklahoma, against the search engine Google. SearchKing's claim was that Google's tactics to prevent spamdexing constituted a tortious interference with contractual relations. On May 27, 2003, the court granted Google's motion to dismiss the complaint because SearchKing "failed to state a claim upon which relief may be granted."
In March 2006, KinderStart filed a lawsuit against Google over search engine rankings. Kinderstart's website was removed from Google's index prior to the lawsuit and the amount of traffic to the site dropped by 70%. On March 16, 2007 the United States District Court for the Northern District of California (San Jose Division) dismissed KinderStart's complaint without leave to amend, and partially granted Google's motion for Rule 11 sanctions against KinderStart's attorney, requiring him to pay part of Google's legal expenses.

Search engine optimization

Exclusion standards	Robots exclusion standard • Meta tags • nofollow

Related marketing topics	Internet marketing • E-mail marketing • Display advertising • Web analytics

Search marketing related topics	Search engine marketing • Social media optimization • Online identity management • Paid inclusion • Pay per click (PPC) • Google bomb

Search engine spam	Spamdexing • Web scraping • Scraper site • Link farm • Free for all linking

Linking	Methods of website linking • Link exchange • Backlink

Other	Geotargeting • Human search engine • Stop words • Poison words • Content farm

Web traffic

2011-09-07T10:00:00.000-07:00

Web traffic is the amount of data sent and received by visitors to a web site. It is a large portion of Internet traffic. This is determined by the number of visitors and the number of pages they visit. Sites monitor the incoming and outgoing traffic to see which parts or pages of their site are popular and if there are any apparent trends, such as one specific page being viewed mostly by people in a particular country. There are many ways to monitor this traffic and the gathered data is used to help structure sites, highlight security problems or indicate a potential lack of bandwidth — not all web traffic is welcome.
Some companies offer advertising schemes that, in return for increased web traffic (visitors), pay for screen space on the site. Sites also often aim to increase their web traffic through inclusion on search engines and through Search engine optimization.

Analyzing Web Traffics

Web analytics is the measurement of the behavior of visitors to a website. In a commercial context, it especially refers to the measurement of which aspects of the website work towards the business objectives of Internet marketing initiatives; for example, which landing pages encourage people to make a purchase. Notable vendors of web analytics software and services include: Webtrends, Coremetrics, Omniture, and Google Analytics.

Measuring Web Traffic

2011-09-07T09:00:00.000-07:00

in December 2004

Web traffic is measured to see the popularity of web sites and individual pages or sections within a site.
Web traffic can be analyzed by viewing the traffic statistics found in the web server log file, an automatically generated list of all the pages served. A hit is generated when any file is served. The page itself is considered a file, but images are also files, thus a page with 5 images could generate 6 hits (the 5 images and the page itself). A page view is generated when a visitor requests any page within the web site – a visitor will always generate at least one page view (the main page) but could generate many more. Tracking applications external to the web site can record traffic by inserting a small piece of HTML code in every page of the web site.
Web traffic is also sometimes measured by packet sniffing and thus gaining random samples of traffic data from which to extrapolate information about web traffic as a whole across total Internet usage.
The following types of information are often collated when monitoring web traffic:

The number of visitors.
The average number of page views per visitor – a high number would indicate that the average visitors go deep inside the site, possibly because they like it or find it useful.
Average visit duration – the total length of a user's visit. As a rule the more time they spend the more they're interested in your company and are more prone to contact.
Average page duration – how long a page is viewed for. The more pages viewed, the better it is for your company.
Domain classes – all levels of the IP Addressing information required to deliver Webpages and content.
Busy times – the most popular viewing time of the site would show when would be the best time to do promotional campaigns and when would be the most ideal to perform maintenance
Most requested pages – the most popular pages
Most requested entry pages – the entry page is the first page viewed by a visitor and shows which are the pages most attracting visitors
Most requested exit pages – the most requested exit pages could help find bad pages, broken links or the exit pages may have a popular external link
Top paths – a path is the sequence of pages viewed by visitors from entry to exit, with the top paths identifying the way most customers go through the site
Referrers; The host can track the (apparent) source of the links and determine which sites are generating the most traffic for a particular page.

Web sites like Alexa Internet produce traffic rankings and statistics based on those people who access the sites while using the Alexa toolbar. The difficulty with this is that it's not looking at the complete traffic picture for a site. Large sites usually hire the services of companies like Nielsen NetRatings, but their reports are available only by subscription.

Controlling web traffic

2011-09-06T10:20:00.000-07:00

The amount of traffic seen by a web site is a measure of its popularity. By analysing the statistics of visitors it is possible to see shortcomings of the site and look to improve those areas. It is also possible to increase (or, in some cases decrease) the popularity of a site and the number of people that visit it.

Limiting access

It is sometimes important to protect some parts of a site by password, allowing only authorized people to visit particular sections or pages.
Some site administrators have chosen to block their page to specific traffic, such as by geographic location. The re-election campaign site for U.S. President George W. Bush (GeorgeWBush.com) was blocked to all internet users outside of the U.S. on 25 October 2004 after a reported attack on the site.^[1]
It is also possible to limit access to a web server both based on the number of connections and by the bandwidth expended by each connection. On Apache HTTP servers, this is accomplished by the limitipconn module and others.

Increase web site traffic

Web traffic can be increased by placement of a site in search engines and purchase of advertising, including bulk e-mail, pop-up ads, and in-page advertisements. Web traffic can also be increased by purchasing non-internet based advertising.
If a web page is not listed in the first pages of any search, the odds of someone finding it diminishes greatly (especially if there is other competition on the first page). Very few people go past the first page, and the percentage that go to subsequent pages is substantially lower. Consequently, getting proper placement on search engines is as important as the web site itself.

Organic traffic

Web traffic which comes from unpaid listing at search engines or directories is commonly known as "organic" traffic. Organic traffic can be generated or increased by including the web site in directories, search engines, guides (such as yellow pages and restaurant guides), and award sites.
In most cases the best way to increase web traffic is to register it with the major search engines. Just registering does not guarantee traffic, as search engines work by "crawling" registered web sites. These crawling programs (crawlers) are also known as "spiders" or "robots". Crawlers start at the registered home page, and usually follow the hyperlinks it finds, to get to pages inside the web site (internal links). Crawlers start gathering information about those pages and storing it and indexing it in the search engine database. In every case, they index the page URL and the page title. In most cases they also index the web page header (meta tag) and a certain amount of the text of the page. Then, when a search engine user looks for a particular word or phrase, the search engine looks into the database and produces the results, usually sorted by relevance according to the search engine algorithms.
Usually, the top organic result gets most of the clicks from web users. According to some studies, the top result gets between 5% and 10% of the clicks. Each subsequent result gets between 30% and 60% of the clicks of the previous one. This indicates that it is important to appear in the top results. There are some companies which specialize in search engine marketing. However, it is becoming common for webmasters to get approached by "boiler-room" companies with no real knowledge of how to get results. As opposed to pay-per-click, search engine marketing is usually paid monthly or annually, and most search engine companies cannot promise specific results for what is paid to them.
Because of the huge amount of information available on the web, crawlers might take days, weeks or months to complete review and index all the pages they find. Google, for example, as of the end of 2004 had indexed over eight billion pages. Even having hundreds or thousands of servers working on the spidering of pages, a complete reindexing takes its time. That is why some pages recently updated in certain web sites are not immediately found when doing searches on search engines.

Traffic overload

2011-09-06T10:00:00.000-07:00

Too much web traffic can dramatically slow down or even prevent all access to a web site. This is caused by more file requests going to the server than it can handle and may be an intentional attack on the site or simply caused by over-popularity. Large scale web sites with numerous servers can often cope with the traffic required and it is more likely that smaller services are affected by traffic overload. Sudden traffic load may also hang your server or may result in shutdown of your services.

Denial of service attacks

Denial-of-service attacks (DoS attacks) have forced web sites to close after a malicious attack, flooding the site with more requests than it could cope with. Viruses have also been used to co-ordinate large scale distributed denial-of-service attacks.

Sudden popularity

A sudden burst of publicity may accidentally cause a web traffic overload. A news item in the media, a quickly propagating email, or a link from a popular site may cause such a boost in visitors (sometimes called Slashdot effect)