Webmaster Blog

Robots speaking many languages

Webmaster Center team — Thu, 05 Nov 2009 23:52:00 GMT

We've already covered in past blog articles some of the basics about how webmasters can use a file called robots.txt to control how search engine crawlers (aka bots) crawl their websites. But there is so much more to talk about with bots. So let's take a bit of a deeper dive into the subject.

Topic 1: Using the proper text file encoding

The robots.txt file is used by webmasters to either specifically define which files and directories that compliant search engine bots may or may not crawl. Robots.txt files are basically text files. However, even something as seemingly straightforward as a text file is not as simple as it might seem. Which type of file encoding scheme is used to save the file makes a big difference. For example, when you use the quintessential text file editor, the Notepad utility in Windows, you can save your text files in your choice of the following encoding types:

ANSI (aka Windows-1252)
Unicode
Unicode big endian
UTF-8 (usually defined as "Unicode Transformation Format," but I've also seen alternatives starting with either "Universal " or even "UCS," which itself stands for "universal character set")

If you choose to save your robots.txt file as either Unicode or Unicode big endian, the resulting file will not be compatible with most search engine bots.

Robots.txt file requirements

To ensure that the search engine bots can read the directives for blocking or allowing content access in robots.txt file (not just with Bing, but all of them), save the file using one of the following compatible encoding formats:

American Standard Code for Information Interchange (ASCII) (a 7-bit, 128 character set)
ISO-8859-1 (an 8-bit, 256 character set backward compatible with US ASCII)
UTF-8 (a variable-length character encoding version of Unicode that is backwards compatible with US ASCII)
Windows-1252 (aka ANSI, as used in Microsoft Windows, it is an 8-bit, 256 character set backward compatible with US ASCII)

Sticking with one of these compatible encoding formats will ensure that the bots you wish to control can read, and thus act upon, your robots.txt file. For more information, check out this article covering the history of character sets from the Microsoft Typography team.

Topic 2: Writing non-ASCII alphabetic characters in robots.txt

The limited number of compatible file encoding formats for robots.txt exposes a potential problem for some users.

The Internet Engineering Task Force (IETF) proclaims that Uniform Resource Identifiers (URIs), comprising both Uniform Resource Locators (URLs) and Uniform Resource Names (URNs), must be written using the US-ASCII character set. However, ASCII's 128 characters only covers the English alphabet, numbers, and punctuation marks. However, some of the alphabetic characters from other Latin-based languages, such as ñ in Spanish and ç in French, are left out of ASCII. More significantly, most characters in non-Latin-based alphabets, such as pi (π) in Greek, ya (я) in Cyrillic, and entire alphabets from many other world languages, can't be accurately written in the limited, English-oriented ASCII.

This limitation with regard to robots.txt can come into play for webmasters when bots visit web servers using languages whose characters fall outside of the ASCII character set. If a robots.txt file is present on that server and it includes directives to block bot access to files and directories whose names include non-ASCII characters, the bot may not interpret the directive as the webmaster intended.

Percent encoding to the rescue

There is a way to make sure the bots can properly read the file and directory path names, regardless of whether it adheres to ASCII standards. When writing directives that include characters unavailable in ASCII, you can "escape" (aka percent-encode) them, which enables the bot to read them.

Percent-encoded characters, discussed in the IETF's RFC 3986, are used as character substitutes. A percent-encoded character is a sequence of one or more three-character codes (aka octets), starting with the "%" sign and followed by two hexadecimal numbers. Percent encoding converts the character's hexadecimal UTF-8 value into a sequence of one or more ASCII-based octets that a URI-compliant bot can read.

To demonstrate what percent-encoded text looks like, type www.%62%69%6e%67.com in your browser's address bar. It will be automatically decoded into www.bing.com. The octet codes %62, %69, %6e, and %67 are decoded by the browser into letters b, i, n, and g, respectively. Note though, that that the recommended use for percent encoding is really for those non-ASCII characters in a URL path to minimize the potential for decoding translation errors.

Real world example

Let's look at a real-world example. Suppose you were the webmaster for a website that contained the URL http://www.domain.com/папка/ (the folder name in the sample URL is written in Cyrillic and literally means "folder"). To block a bot from accessing that folder on your website using percent encoding in your robots.txt file, you would need to write the directive as follows:

Disallow: /%D0%BF%D0%B0%D0%BF%D0%BA%D0%B0/

If instead you simply wrote

Disallow: /папка/

the bot may not be able to read the directive and thus fail to perform as desired.

Performing percent encoding

So how do you translate your non-ASCII characters into escape-encoded octets? Well, it's a bit of a chore, frankly. If you search for them, there are a few websites and/or tools that offer to perform percent encoding for you, but rather than endorse a site I know nothing about, I'll instead tell you how to manually calculate the conversion. If you want to use an automated tool, go for it. But knowing how the process works will allow you to verify that a tool encoded your characters correctly.

Warning! I'm going to get pretty tech geeky here. If working with hexadecimal and binary numbers is not your thing, I apologize up front!

OK, thus warned, let's get to it. You first need to know the UTF-8 hexadecimal value for each character you want to encode. They are usually presented as U+HHHH. The four "H" hex digits are what you need.

As defined in IETF RFC 3987, the escape-encoded characters can be between one and four octets in length. The first octet of the sequence defines how many octets you need to represent the specific UTF-8 character. The higher the hex number, the more octets you need to express it. Remember these rules:

Characters with hex values between 0000 and 007F only require only one octet. The high-order (left most) bit of the binary octet will always be 0 and the remaining seven bits are used to define the character.
Characters with hex values between 0080 and 07FF require two octets. The right most octet (last of the sequence) will always have the first two highest order bits set to 10. The remaining six bit positions of that octet are the first six low-order bits of the hex number's converted binary value (I set the Calculator utility in Windows to Scientific view to do that conversion). The next octet (the first in the sequence, positioned to the left of the last octet) always starts with the first three highest order bits set to 110 (the number of leading 1 bits indicates the number of octets needed to represent the character - in this case, two). The remaining higher bits of the binary-converted hex number will fill in the last five lower order bit positions (add one or more 0 at the high end if there aren't enough remaining bits to complete the 8-bit octet).
Characters with hex values between 0800 and FFFF require three octets. Use the same right-to-left octet encoding process as the two-octet character, but start the first (highest) octet with 1110.
Characters with hex values higher than FFFF require four octets. Use the same right-to-left octet encoding process as the two-octet character, but start the first (highest) octet with 11110.

Below is a table to help illustrate these concepts. The letter n in the table represents the open bit positions in each octet for encoding the character's binary number.

Hexadecimal value	Octet sequence (in binary)
0000 0000-0000 007F	0nnnnnnn
0000 0080-0000 07FF	110nnnnn 10nnnnnn
0000 0800-0000 FFFF	1110nnnn 10nnnnnn 10nnnnnn
0001 0000-0010 FFFF	11110nnn 10nnnnnn 10nnnnnn 10nnnnnn

Let's demo this using the first letter of the Cyrillic example given above, п. To manually percent encode this UTF-8 character, do the following:

Look up the character's hex value. The hex value for the lower case version of this character is 043F.
Use the table above to determine the number of octets needed. 043F requires two.
Convert the hex value to binary. Windows Calculator converted it to 10000111111.
Build the lowest order octet based on the rules stated earlier. We get 10111111.
Build the next, higher order octet. We get 11010000.
This results in a binary octet sequence of 11010000 10111111.
Reconvert each octet in the sequence into hex. We get a converted sequence of D0 BF.
Write each octet with a preceding percent symbol (and no spaces in-between, please!) to finish the encoding: %D0%BF

You can confirm your percent encoded path works as expected by typing it into your browser as part of a URL. If it resolves correctly, you're golden.

There always more to talk about with robots (and so many other webmaster-related topics). If you have any questions, comments, or suggestions, feel free to post them in our SEM forum. Until next time...

-- Rick DeJarnette, Bing Webmaster Center

MSNBot 1.1 is retired

Webmaster Center team — Wed, 04 Nov 2009 22:50:00 GMT

The Bing team has been talking about its new crawler (aka bot), MSNBot 2.0b, in this blog for quite some time now. We have made numerous improvements in its performance, addressed some webmaster concerns, and published detailed information on how to control the bot with a robots.txt file. Today we are announcing that the new bot is fully operational. This development will enable Bing to do a better job at gathering the information we need from the myriad of websites we index worldwide.

As MSNBot 2.0b enters full-scale production, the time has come to retire our previous generation bot, MSNBot 1.1. By the end of the first week in November, you will no longer see the following user agent in your referrer logs:

msnbot/1.1 (+http://search.msn.com/msnbot.htm)

The only Bing user agent you will see in your logs from this point forward will be this, our new bot:

msnbot/2.0b (+http://search.msn.com/msnbot.htm)

This event is a major milestone for the Bing engineering team, and we look forward to the positive developments that this bot will bring to our search engine index and thus to Bing customers. We want to specifically thank all those webmasters who provided us with valuable feedback as we ramped up the production of the new bot. Your assistance and cooperation was essential to making this milestone happen.

Stay tuned for more information about this and other developments from the Bing engineering team. We'll have a lot more to talk about in the coming weeks and months.

If you have any questions, comments, or suggestions, feel free to post them in our Crawling/Indexing Discussion forum. Be back at you soon...

-- Rick DeJarnette, Bing Webmaster Center

Fixing 404 File Not Found frustrations (SEM 101)

Webmaster Center team — Wed, 04 Nov 2009 20:43:00 GMT

You've seen it. So have I. Nearly every person who has actively browsed the Web for more than 15 minutes has seen it. I'm talking about the dreaded 404 File Not Found error. When it occurs, users simply abandon their search on that site and go elsewhere. That's a potential lost sale, subscription, or download opportunity (aka conversion) for the affected site! It has been estimated that up to 10% of traffic to large websites on the Web is looking for pages that don't exist, so this is a big problem.

Back in the early days of the Web, when you entered an incorrect URL, you got a nearly blank, stark white screen containing nothing but the simple words, "404 File not found." Yeah, thanks. That's really helpful. But back in the day, when people actually wrote webpages in Notepad, that behavior was de rigueur. That was the way things were. Back then. But that was then. This is now. We can do better.

Unusable URLs

There are a great many ways for URLs to be rendered unusable. They can be mistyped or misspelled by the user, the page can be moved, renamed, or deleted by the webmaster, and the URL can be incorrectly written by external webmasters who create the outbound link from their sites to your, which is the most frustrating situation for the webmaster of the intended destination.

So when a potential customer of yours, interested in learning more about what you do or what it is your site has to offer, uses an erroneous URL for a page within your website today, what do they get? Do they get the Web's equivalent of the blue screen of death, a useless page that stops them dead in their tracks and forces them to move on to a competitor's site? Or do they get a page with helpful guidance that keeps them in your site, offering to assist them with finding the information they seek?

As webmaster, you control how you rename and remove pages from your site (see information on using redirects to point to moved and renamed pages in an earlier SEM 101 article). But you can't control the content of your inbound links, nor can you nudge that user who can't spell or remember your page-naming scheme (yet another good reminder to make the file names of your pages logical and easy to remember). So instead of cursing the darkness of silly users over whom you have no control, instead light a metaphorical candle by creating a custom 404 error page.

Custom 404 error messages

Creating a custom 404 error page is not that hard to do, but so many webmasters overlook doing this small but crucial step. Your custom 404 error page will appear in place of a generic 404 error message when the URL to your site is broken. By anticipating what users will likely want to know when they come to your site, you can proactively give them enough information in the custom error message to keep them in your site, and then provide them with easy means to find the information they want.

OK, so this is a great idea, but how do you do it? Well, I've got you covered there. What you do depends on which web server platform you are using to host your site. Let's take a look at what you need to do to establish this safety net for your users. After all, adding this one little feature to your site might make the difference between a bounce and a conversion!

Dynamic or static?

Before we discuss anything else, you first must decide how you want to approach this situation. You can create a single, static custom 404 error page for your entire site. That will be pretty simple to implement and is likely a good fit for smaller sites, but larger sites might feel constrained by a single, static page. There are technologies available that allow you to create dynamic error pages based on script, and that might suit a larger site's needs better, but the caveat exists that if the host scripting engine suffers a failure, then you'll have no custom 404 message at all. Given that this is not a developer blog, I'll just point all webmasters interested in creating dynamic 404 error pages out to Bing searches for information on both Internet Information Services (IIS) and Apache platforms.

If you're interested in learning more about basic, static custom 404 pages, follow on. After all, this is SEM 101, right?

Create the custom page content

You need to decide what information and features you want to include in your custom 404 page. I suggest the following:

Use the page template for your website to maintain consistency with your site's look and feel on this new page.
Include your site's navigation scheme in this page. If you have created an HTML sitemap page or a dedicated site search box, include access to those features as well.
In the page's text, first acknowledge that the URL for the page the user was intending to see does not exist. Then offer a quick description of your site's subject-matter theme and list the products/services/opportunities it offers. Follow that with a suggestion that the reader use the on-page site navigation (menus, sitemap, search box, etc.) to look for the information they are interested in.

Don't add huge, hard-to-read fonts, auto-play music, videos, or animations, or include anything else that may be off-putting to the first-time reader. Leave off the advertisements as well - this will distract the user from the critical mission at hand - to get the user back onto a real page within your site. Remember, the whole point of a custom 404 error page is to prevent a bounce (aka a single page visit session in which the visitor abandons the site without visiting any other pages). Keep the page clean and easy to read.

Save this custom page file to the root directory of your site (for this discussion, I'll call the new file 404.htm). Now let's cover how to employ it on the various web server platforms.

Apache

Apache users can configure a special text file found in the root directory of their site to implement a custom 404 error page. The file, named .htaccess (the dot precedes the file name and contains no typical file name extension), can be edited in Notepad to include the following line (using our sample error page file):

ErrorDocument 404 /404.htm

Of course, you must name and store the custom 404 error page file identified in the location as specified or even it will return a 404 File Not Found error! Use Bing search results to find more information on what to do in Apache for custom 404 messages.

IIS

If you are using IIS, the implementation of a custom 404 page is simple. Here's how:

First, open IIS and select the website you want to customize.

In IIS versions 5 and 6:

Right-click the website and select Properties.
Click the Custom Errors tab.
Double-click the listing for status code 404 to edit that setting.
In the Edit Custom Error Properties dialog box, set the Message type drop down list to URL.
In the URL text box, per our example earlier, type /404.htm.
Click OK to save your work.

In IIS 7 and higher:

In the right pane, double-click Error Pages.
Double-click the listing for status code 404 to edit that setting.
In the Response Action group, select the option Insert content few static file into the error response.
Click Set.
In the Root directory path text box, either type the physical path of the root directory (up to the portion of the path that branches into a specific locale directory) of the custom error page file or click Browse and navigate to the error file root directory and then click OK.
In the Relative file path text box, type the relative path of the localized error file, which if our example earlier was for US English only, could be written as \EN-US\404.htm.
If you are implementing localized versions of a custom 404 error page, ensure the Try to return the error file in the client language check box is selected, and then click OK twice to save your work.

IIS 7.0 and higher users can alternatively edit their web.config file to include a snippet of code to accomplish the same task. Using the sample file 404.htm referenced earlier, here is an example code snippet:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <system.webServer>
    <httpErrors errorMode="DetailedLocalOnly">
      <remove statusCode="404" />
      <error statusCode="404" path="404.htm" responseMode="File" />
    </httpErrors>
  </system.webServer>
</configuration>

Check out Bing search results for more information on creating custom 404 error pages in IIS.

Bing Web Page Error Toolkit

Aside from creating your own home-grown, dynamic custom 404 page solution for IIS, there is another option for IIS users that kicks it up a notch. The Bing Developer Center team has augmented the IIS experience in creating even more useful customized 404 error pages. Straight from the Bing Developer Center blog, check out the post from this past summer, Customize your 404 error pages with the Bing API Web Page Error Toolkit. The post reveals that the toolkit, built on the Bing API, replaces the default IIS 404 error page with a dynamically created Bing search page containing customized search results derived from keywords extracted from either the source Uniform Resource Identifier (URI) or the HTTP request. This creates a search list of relevant, alternative pages on your site, helping the user more easily and quickly find the information they originally wanted without abandoning their visit to your domain.

The Bing Web Page Error Toolkit is available as a free download. If you're using IIS, give it a try. Your users will show their gratitude with more conversions and fewer bounces! Note that you will need Microsoft Visual Studio to use this tool.

Test, test, test

Once you have implemented your custom 404 page, test it out. In a browser, type your domain name followed by a short, random string of characters you know does not match any existing file or directory name. If everything was implemented correctly, you should get the custom 404 page you created in response to that bad URL. From that point forward, your potential customers will also see that page, and if the new error page's content is well done, they will more likely stay on your site. And that's the goal after all, right?

If you have any questions, comments, or suggestions, feel free to post them in our SEM forum. Later...

-- Rick DeJarnette, Bing Webmaster Center

Translator widget: Delivering your site to the world

Webmaster Center team — Wed, 14 Oct 2009 17:04:00 GMT

About eight months back, the Microsoft Research Translator team delivered an entirely unique way of delivering your website's pages to visitors who speak a different language with no development effort on your part. Unlike any other translation widget/gadget available at that time, the Translator widget was unique in that it kept your audience on your site, rather than redirecting them to a proxy translation service. Since then, thousands of sites have adopted the Translator widget and have been able to attract a much broader audience from around the world.

Powered by the same machine translation technology that is used by Bing, Internet Explorer, and Office, the Translator widget provides a free option to deliver a "gisting" experience to a non-native audience. While machine translation cannot replace a professional or human localization, it aims to provide a rough understanding (the gist) of the content on the page to those that cannot read the original language. The translation engine is worked on continuously to deliver better quality and more languages. You can learn more about the pioneering work being done by our researchers in this space over at the Translator group's site at Microsoft Research. With the widget, given the on-demand nature of the translations, there is no load on your site and the freshest translations are delivered to the visitor.

Adopting the widget is as simple as copying and pasting a small snippet of JavaScript code into your site. You can customize and generate a snippet for your site at the widget adoption portal. Once this code snippet is pasted into an appropriate area of your page, the Translator widget appears on your site to your users in the language their browser is set to. This localization of the widget user interface ensures that your site's audience always sees "Translate this page" in their language and thereby are able to kick-off the translation (as shown in the images below). The translator team is also planning to add an "automatic" translation functionality, where you can set the widget to auto-translate the page into the visitor's browser language upon arrival.

Once translation has been kicked off, the page is translated using "progressive rendering" - a technique that ensures that the visitor can immediately get the benefit of translation without waiting for the whole page to be translated. As they navigate from page to page on your site, the pages get automatically translated, resulting in a seamless experience for your visitors. A progress bar and several other controls are displayed as well, to the visitor, floating at the top of the screen. Upon translation, hovering over the translated sentences displays tool tips that show the original source sentence, as shown in the image below. This can be useful in situations where the visitor has some familiarity with the source language.

Another interesting feature of the widget is the ability to share a link to the translated page. A visitor to a site who has translated the page to a particular language can share a link to the translated version of the page. When the recipient clicks on the link, they are taken to the page and translation to their language is kicked off automatically. For example, this page (http://viks.org/2009/06/11/instant-translations-in-bing/) can be auto-translated to Spanish by appending the code #mstto=es to the end of the URL (http://viks.org/2009/06/11/instant-translations-in-bing/#mstto=es).

So, what are you waiting for? Go get the widget and start making the Web more "worldly"!

Stay tuned to the Webmaster Center blog and the Translator team's blog for more information on additions to the widget functionality. You can also participate in the Translator user community.

-- Vikram Dendi, Senior Product Manager, Microsoft Research

Webmaster Center blog Q&A

Webmaster Center team — Fri, 09 Oct 2009 20:33:00 GMT

We've been really busy here at the Bing Webmaster Center blog team, pumping out new content on a regular basis to create a nice library of content on issues that matter to webmasters and online publishers. I thought I'd take a moment to catch my breath, pause on creating a new thematic article (or yet another multi-part series!) for SEM 101, and address some commonly asked questions in the blog comments.

Q: Why wasn't my question in the blog comments answered?

A: Well, the Webmaster Center blog is really not the right place for such back and forth exchanges. In fact, at the end of each blog post, there is a reminder to post your questions and comments in the SEM forum. The collection of Webmaster Center forums are specifically designed and staffed to address your questions, so if you have a question for the Bing Webmaster Center team, please post it in the forums where you will get a reply. Now if you have a question for other webmasters, you can certainly post that to the blog comments, but even then, you may get better results from posting it in the forums.

Q: Why did my blog comment disappear?

A: As we truly value all the input we receive from the webmaster community, we are reluctant to delete any comments, but on occasion we have to. If a blog post comment is simply blank, includes profanity or obviously objectionable, business-inappropriate content, or is merely an off-topic advertisement for an external website (the basic definition of web spam), we delete those comments. In cases where the same comment is repeated multiple times in the same post by the same sender, we delete the redundancies but leave the original.

We love to get your feedback, and whether you like or hate our content, have a suggestion for clarifying a point or care to elaborate your own, related story, we're really happy when you contribute to the community. Please continue to do so! But if your comment was deleted, there was a compelling reason for doing so. On rare occasions, we get someone who decides to post the same web spam comment across dozens of our posts simultaneously, sometimes even spanning beyond Webmaster and going into other Bing community blogs, such as Maps, Developer, Travel, or the main Search blog. I've seen a couple of instances where someone comment-bombed our blogs in a huge, redundant web spam blast. Those comments are all quickly deleted and those spammers are banned from posting again to the blogs. Seriously, who wants to read that junk?

Q: I added in my site's URL in my blog comment. That's good link building for my site (it's coming from an authoritative site, after all), right?

A: Well, the basic intent of the idea is good. You do want to get as many high-quality, authoritative, inbound links as you can. That is one of the keys to improving your page rank of your site. But in this case, as so many blog commenters do this on a regular basis, the links entered in Bing blog comments are automatically created with the rel="nofollow" attribute included in the anchor tag. This means that when search engines hit the blog page, the link using that attribute will not earn any inbound link credit for the referenced page. So sorry, folks, this one won't count.

Make no mistake, earning high-quality inbound links is hard work. You need to get webmasters from authoritative sites to link to you (this is why link exchanges don't help build page rank value). You usually do that by providing high quality content on your site that those webmasters value. But simply adding a URL to a blog comment is far from hard. And many websites make it a policy to add the rel="nofollow" attribute to all visitor-generated content links because that content can be so hard to police. Who wants to allow a visitor to link out to web spam or malware? And who has time to police the quality of every user-generated link?

You had a good idea with good intentions. But in this case, it's a waste of time to include your site's URL if your goal is to get an authoritative inbound link.

Q: Why isn't my site indexed yet?

A: This is exactly the sort of question you should post in the Webmaster Center's Crawling/Indexing Discussion forum. The number of variables here that can affect the answer specific to your question is enormous, including:

the quality of your site's content
the quantity and authoritative quality of your site's inbound links
the ability of the search engine bot to discover and crawl your site's content
the validity of the HTML code used
the age of your site
the freshness of the site's content over time
whether or not malware was detected on your site
whether or not the content is judged to be web spam or duplicate content copied from other sites
whether your site violates the Bing search guidelines
and so much more

If you post this question in the Crawling/Indexing Discussion forum, our staff can look up your website's index information and help determine what can be done to improve your situation. Take advantage of their expertise and resources for this and other similar questions!

Q: Do the search engine optimization (SEO) recommendations you give for Bing affect my SEO performance with other search engines?

A: Yes. They improve it! And of course, if you are actively performing legitimate, white-hat SEO activities for other search engines, it'll also help you with Bing as well. The basic takeaway here is that SEO is still SEO, and Bing doesn't change that. If you perform solid, reputable SEO on your website, which entails a good deal of hard work, creating unique and valuable content, earning authoritative inbound links, and the like (see our library of SEM 101 content in this blog for details), you'll see benefits in all top-tier search engines.

But remember: SEO efforts ultimately only optimize the rank position that your site's design, linking, and content deserves. It removes the technical obstacles that can impede it from getting the best rank it should. However, it won't get you anything more than that. The most important part of SEO is doing the hard work of building value necessary to make your site stand out from the competing crowd of other websites for searchers.

One other thing to remember: there is a long tail in search. After the few obvious keywords used in a particular field, there are many, many more keywords used to a lesser degree that still drive a lot of traffic to various niches of that field. Instead of always trying to be number 1 in a highly competitive field for the obvious keywords and faltering, consider doing the work of finding a less competitive keyword niche in that same field and then do the hard work necessary to earn solid ranking there.

For more information on SEO and Bing, see our recent blog post, Search Engine Optimization for Bing.

Q: How do I submit a Sitemap to Bing?

A: There are a couple of ways to do this. If you have registered your site with Bing Webmaster Center tools, log into the site, select the site to use from the Site List page (webmasters can register multiple sites for one account), and then click the Sitemaps tab. From there, you can perform a direct Sitemap submission by typing the web address of your Sitemap file (such as www.example.com/sitemap.xml -- be sure to omit the "HTTP://" protocol designation as it's not needed here). If you have not yet registered your site with Webmaster Center (why not?) and you just want to submit your Sitemap file through your web browser using our Sitemap ping service, use the following URL:

http://www.bing.com/webmaster/ping.aspx?sitemap=add your Sitemap web address here

Again, leave off the "HTTP://" as it isn't needed. For more information on using Sitemaps with Bing, see our Webmaster Center blog article, Uncovering web-based treasure with Sitemaps (SEM 101).

Q: Does Bing support sitemap index files?

A: Definitely. Bing supports Sitemaps with up to 50,000 entries, be they site URLs or, in the case of Sitemap index files, references to child Sitemap files. With a Sitemap index file containing 50,000 references to child Sitemaps, each of which containing 50,000 site URLs, your Sitemap strategy can reference up to 2.5 billion URLs. Let us know if you need more. :-)

For more information on Sitemap index file support, see the Webmaster Center blog post, Bing enhances support for large Sitemaps.

Q: I can't find you anymore. Where did your blog recently move to? How do I get to it now?

A: The blog didn't actually move (well, not since June, when Bing was introduced). We're still at http://www.bing.com/community/blogs/webmaster/. But the extras menu in the Bing user interface was recently removed and all references to Webmaster Center, the Bing Community, and the Bing Webmaster Center blogs and forums, were migrated under the More link, found on both the Bing home page and the top left menu on other Bing pages. Some folks might have missed that. Please be sure to save the Webmaster Center blog to your browser favorites or, better yet, subscribe to our blog's RSS feed.

Q: Why are your blog columns so long?

A: No reason.

Q: Thank you!

A: You're welcome! We get this comment most of all, and I wanted to make sure we acknowledged our appreciation for your kind words and support.

As always, if you have any questions, comments, or suggestions, feel free to post them in our SEM forum (or any of the other Webmaster Center forums as appropriate). Until next time...

-- Rick DeJarnette, Bing Webmaster Center

The merciless malignancy of malware Part 4 (SEM 101)

Webmaster Center team — Thu, 01 Oct 2009 17:14:53 GMT

OK, so I totally geeked out with my recommendations on how to better secure your webmaster computing environment. As a result, I had too much material for one post and thus had to split it up into two pieces. Let’s wrap up this long series of posts on malware by finishing up with the last of the security recommendations.

In Part 1 of this series on malware, we discussed how to detect a malware infection on your website using tools like Bing’s Webmaster Center. The Part 2 post covered the resources and strategies for identifying the types and locations of malware code that typically affect websites with advice on how to remove it. The Part 3 post began the run-down through 10 recommendations (well, the first 5, anyway!) on how to better secure your workstation and web server computers to prevent the malware from coming back. Today’s post, Part 4, finishes the list, and then includes information on what steps you can take to get that pesky malware warning message removed from your recently cleaned site in the Bing index.

Recommendations continued

Getting rid of malware is only part of the battle. Hardening your security practices to keep it away is just as important. Let’s continue the list of recommended security strategies started in the previous post.

6. Run Microsoft Update

I am presuming with this recommendation that you are running a modern Microsoft Windows operating system. Regularly run Microsoft Update on every Windows-based computer you use to touch your website. When you do so, I recommend that you click Custom to see the total list of available updates for your computer rather than seeing only the High Priority updates. Always keep current with the latest High Priority updates and strongly consider applying others updates as well.

Note that the second Tuesday of every month is commonly referred to as “Patch Tuesday” for Microsoft Update, and time should be set aside on those dates to make sure all Windows-based systems in your web server infrastructure get the necessary security updates. Occasionally Microsoft, when necessary, also provides high-priority security updates ahead of this schedule, so it pays to stay on top of these releases as they occur. Signing up to receive Microsoft Technical Security Notifications can help!

7. Update non-Microsoft applications, too

Applications that touch the Internet are at least as vulnerable to security holes as are web browsers and operating systems. Some major software manufacturers are beginning to build into their applications an online update system analogous to Microsoft Update. But not all have this feature yet, and not all that do perform the update automatically. It’s a really good idea to scan for and plug the often nasty security holes in the applications on your workstation through a software updating tool. I like the Secunia Software Inspector tool (check the licensing requirements for commercial use, but it’s free for many users), but there are many other choices out there. Be sure that the web applications you use are checked in that process. The bottom line is you need to regularly check for and install any software updates on all of the computers associated with your website.

Keep in mind that software manufacturers regularly release updates for their products when they discover faulty features and security holes. The hacker community makes a point of studying those patches to learn what exploits the updates fix. If you don’t stay current with software updates, your computer may become vulnerable to reverse-engineered exploits.

8. Improve your wireless security

Many computers these days, especially laptops, are connected to the Internet only by wireless connections. If you work in a big organization with a security-conscious IT shop, you’re probably fine (while you’re at work, anyway). But many small shops and even more home users install their new Wi-Fi routers using default settings across the board. Hackers have developed such efficient wireless security cracking tools over the past decade that paranoia is no longer considered irrational or delusional behavior among IT security folks. (But if tin foil hats come out, all bets are off.)

There are several things you can do to improve the security of your wireless network router. Dig up the user’s manual for that old router and learn how to do all of the following:

Update the device firmware. Go to the router manufacturer’s website and browse to the Downloads page for your model (typically within the site’s Support section) to see if you have the latest firmware release. If not, download it and install it. You may get new functional features and/or have known security holes resolved. Either way, look into it. The router’s manufacturer put it out there for a reason!
Change the administrator password from the manufacturer’s default (using the tips in Create strong passwords). Hackers typically know the default administrator password for various routers. Leaving yours with the default is honestly no better than disabling the admin password altogether.
Change the network’s Service Set Identifier (SSID) friendly name from its default to a name of your own choosing. Then once done, then disable the SSID broadcast so that the wireless network is hidden. It’s harder to crack a wireless network if you don’t see it, especially when you don’t know its name!
Enable media access control (MAC) address filtering so that only computers and devices whose MAC addresses you specify can access the network. All others are denied access.
Exclusively use Wi-Fi Protected Access version 2 (WPA2) security with Advanced Encryption Standard (AES) encryption for the most secure connections. Forget relying upon Wired Equivalent Privacy (WEP) or WPA using Temporal Key Integrity Protocol (TKIP) encryption for security. Modern wireless security cracking tools can break these encryption schemes in minutes, even with the longest keys.
Enable advanced routing features such as SPI (as discussed in tip #3 in the previous post). If your wireless router doesn’t support SPI, it’s probably using old technology and it may be time to shop for a new, more secure Wi-Fi router.

Note that none of these changes by themselves will sufficiently upgrade your wireless security, but the aggregate value of implementing them all will make your wireless network much more difficult to crack. And unless you are dealing with extremely determined hackers with an abundance of both technical resources and time to focus on cracking your specific, secured network, they will almost always move on to another of the ubiquitous, softer targets in the wifisphere.

9. Protect your website’s configuration files

Ensure that the sensitive configuration files of your web server and your web applications aren't accessible to unauthorized, external users. Place them in directories that are not served to the public and then disable directory browsing on your web server. Refer to your web server documentation for specific instructions on how to do this. I also recommend researching additional methods of securing your web server, such as IIS or Apache, from attack.

10. Perform data validation on user input

If your website accepts user input, ensure it is validated before processing or displaying it back to the user. For example, if you have a login form that accepts user names and passwords that are checked against a database, ensure that the input is scrubbed of any unexpected or invalid characters that might offer malicious manipulation of the database. Also, if user input is accepted and displayed (such as on forums), ensure users aren't able to modify the source code of the webpage, such as adding script for running <iframe> HTML code.

Also be sure that input from backend systems is validated. This protects the users of your website, even if attackers "only" managed to break into a backend system, like your database. For more information on similar, related website attacks, look into the topic of cross-site scripting (XSS).

Bonus tip: Backup your clean web content

Once you’ve ensured your site’s content and source code is clean, back it up! Disaster recovery is not just about fires, floods, and earthquakes. A sudden, major malware infection ranks right up there in terms of potential business outages, so protect your work, your site, your business, and your customers who depend on you with proper, functional backups of clean code.

Even more information on securing servers

To be ultra secure, you might simply consider flattening and rebuilding the server from scratch. But don’t simply rebuild it to the way it was – remember, it was hacked in that state! Put in place all of the hardening steps mentioned earlier, as well as triple-checking all of your permissions settings, before putting the server back in service online. For more information on dealing with hacked servers, check out What to Do If Your Website Has Been Hacked by Phishers.

Request removal of the Bing malware warning

Once you’ve resolved your malware infection, closed the security vulnerabilities that allowed your computer to be successfully attacked, and uploaded your cleaned-up source code to your web server, you’ve got one more job to do. It’s time to request that Bing re-evaluate your website for malware. Here’s how:

Open the Bing support form.
In the resulting Windows Bing Support web form, type your full name and email addresses in the text boxes provided.
In the Service: Bing drop-down list, select My Site has a malware warning.
In the new drop-down list that appears below, select the option that best matches your specific situation (in this case, that’ll be The malware has been removed.
Complete the remainder of the form, adding as much detail as possible in the comments text box to help the support team resolve your request. Once completed, type the characters shown in the security image, and then click Submit.

By following this procedure, Bing will rescan your website to check that the malware has been removed. If confirmed, your content can then be reincluded in normal search results. Once done, keep monitoring your site’s malware status in the Crawl Issues tool of Bing’s Webmaster Center, just to be sure you stay on top of any new issues.

If you have any questions or comments about malware, please feel free to post them in our General Questions forum. For regular SEM and SEO questions and suggestions, please go to our SEM forum. See you again soon…

-- Rick DeJarnette, Bing Webmaster Center

The merciless malignancy of malware Part 3 (SEM 101)

Webmaster Center team — Thu, 24 Sep 2009 22:18:10 GMT

We’re going to diverge a bit from our regularly scheduled programming. Normally this column discusses search engine optimization (SEO) and related elements of search engine marketing (SEM), but we’re knee deep into our multi-part series on malware and we’re going to begin the wrap-up with a talk about improving computer security. However, I geeked out a bit here, and the column went a bit long (yeah, even longer than usual!), so I decided to break this last section up into two pieces. Who wants to read a white paper as a blog post? I mean, besides me? :-)

While beefing up your computer security practices won’t necessarily have a direct affect your site’s SEO performance, consider the repercussions of not doing so. Presenting a malware-infected website to your customers is a great way to ruin the integrity and conversion potential of your online business. Top tier search engines like Bing will either block a malware-infected page from showing up in its search engine results pages (SERPs) or will redirect the affected page’s link to a malware warning message. Bing presents the following warning message when searchers click its SERP link for a malware-infected page:

Since the vast majority of searchers will never opt to click through to override a malware warning from a SERP, assuming the link to the affected page is even shown in the first place, failure to quickly address detected malware infections is a great way to kill off pretty much all of your search referral traffic. And those customers who navigate directly to your site will not likely come back once they’ve determined your site was the source of their newly acquired malware infection.

In Part 1 of this series on malware, we discussed how to detect a malware infection on your website using tools like Bing’s Webmaster Center. The Part 2 post was a long discussion on the resources and strategies for identifying the types and locations of malware code that typically affect websites, and included high-level information on removing it from your site. Today’s post, Part 3, and the next one, Part 4, present altogether 10 solid recommendations on how to better secure your workstation and web server computers so that the infections don’t come back. After all, what good is it to invest time in shooing away a kitchen full of house flies when you haven’t bothered to close the screen door?

Recommended security strategies

Once malware is removed, steps need to be taken to secure your website to prevent malware from reappearing on your website in the future. Securing all of the computers involved with creating, managing, and serving your website are the keys to success. If you were infected with malware, that means your computer infrastructure has one or more security vulnerabilities that need to be addressed. The following preventive measures are key tasks that either you or your hosting provider (likely a combination of both) need to take.

1. Install and use an antivirus tool

If you have not done so yet, install and run a fully capable antivirus software tool on the computer workstation you use to develop and upload your website content. If your web server is not otherwise protected, also install an appropriate antivirus solution on it as well. A high-quality antivirus product will support scanning embedded scripts and other locally saved webpage controls used in your website’s source code for any known malware, so don’t skimp on quality and features here.

Once you have an antivirus solution installed, be sure to regularly update both the tool’s program code and its malware signature files used for detection. Most modern antivirus tools have update features built-in, but make sure the update feature is working as expected before setting it and forgetting it. If you need some convincing as to why keeping your antivirus solution updated is important, I can only refer you to the Microsoft Security Intelligence Report (to which Bing is a key contributor). And lastly, remember to use your antivirus tool! You need to regularly scan your Internet-connected computers for malware to ensure they remain clean.

Microsoft offers a free, web-based, anti-malware scanner called Windows Live OneCare safety scanner. It works on computers running Windows XP, Windows Vista, and Windows 7. It checks for and removes viruses, spyware, and other likely unwanted software, as well as detects vulnerabilities in your Internet connection. Heck, it can even be used to clean up your hard drive and tune up your computer’s performance!

Microsoft has also just released its Microsoft Security Essentials program, a new, no-cost, anti-malware solution that runs in the background of your computer and protects it in real-time against viruses, spyware, and other malicious software. Check it out.

2. Install and use an anti-spyware tool

If your antivirus solution doesn’t specifically include it (and many do these days), you should also install a good anti-spyware scanning and protection tool on your workstation (since you likely don’t surf the Web directly from your web server, this protection is likely not needed there). As with the antivirus tool, keep this tool updated and use it regularly to scan your computer for problems. The last thing you want to do is introduce malware into your web server environment from a compromised workstation!

Microsoft also offers a free antispyware tool called Windows Defender. It actively protects your computer in real-time against pop-ups, performance problems, and security threats by detecting and removing spyware and other unwanted software.

3. Use a firewall

At a minimum, you should use a software firewall utility to protect your workstation and server from external hackers. A software firewall blocks unauthorized and inappropriate network traffic to your computer. Hackers employ these techniques to take control of, and thus install malware on, your system. Many software firewall options exist, both for Windows users and users of other platforms. On your server, use the firewall to block all inbound traffic except for normal web server requests traffic and a secure access method for your webmaster site uploads from predefined computers.

To improve security further, consider installing a separate hardware firewall device between your computers and the Internet that offers, at a minimum, stateful packet inspection (SPI). Firewall devices use SPI to track the state of the network connections passing through them. Rogue or malformed TCP/IP network packets, sometimes implemented by hackers to get through weaker firewall solutions, are rejected by SPI-enabled firewalls. Application-level filter firewalls are better yet, as they work at the application layer of the network protocol stack, where they can more safely examine which network protocol is used on which port and determine whether its use is appropriate.

4. Use a secure protocol to access your web server

Standard FTP protocol doesn’t encrypt the data as it’s transmitted, so if your computer or its network has been compromised by hacker using network sniffer technologies, your web server’s logon credentials are at risk of being stolen. As alluded to in the section on firewall, using Secure FTP or Secure Shell (SSH) eliminates this potential vulnerability. Make sure you do this end-to-end, from the site developer to the webmaster and from the webmaster to the server.

5. Change and strengthen your passwords

Your computer security is usually only as good as the freshness and strength of the passwords you use to access your computer. If your passwords haven’t been changed since the days 'N Sync was still hot, it’s time to say "Bye Bye Bye" to that. You need to implement a regimen of regularly changing your passwords. And when you do, please make them harder to guess than “password” or something else hyper-obvious. Check out the article, Create strong passwords, for helpful tips on doing this.

Yeah, you don’t need to tell me that this is inconvenient. But if you choose to skip doing this, while you might be happier temporarily, hackers will be thrilled. Static, simple passwords are easy to crack, and once hackers figure out your logon credentials, they can do anything they want to your site, including locking you out! Imagine having a hacked site and you can’t even log in to fix the problem!

More recommendations to come

We’ll continue with another five recommendations for securing your webmaster computing environment in our next post. If you have any questions or comments about malware, please feel free to post them in our General Questions forum. For regular SEM and SEO questions and suggestions, please go to our SEM forum. I’ll be back…

-- Rick DeJarnette, Bing Webmaster Center

The merciless malignancy of malware Part 2 (SEM 101)

Webmaster Center team — Fri, 18 Sep 2009 22:34:04 GMT

Malware infections are no laughing matter. When they afflict your website, they can infect your customers, who won’t appreciate your sharing, intentional or not (and I’m guessing it’s not)! And if Bing discovers malware on your site, your listing in the Bing search engine results pages (SERPs) will either be completely omitted or the link to your site will be disabled, so when the searcher clicks on it, only a malware warning appears. All told, this is bad news for conversions, don’t you think?

This article is Part 2 of a three-part series on malware. Part 1 covered how to detect the presence of malware on your site by using the Bing Webmaster Center tools to get access to the information the bot sees when it crawls your site’s pages and the external links they contain. In this post, we’ll cover the available resources and strategies to do a malware clean-up job. It’s usually a big job, so let’s get right to it.

Cleaning up the mess

Bing’s detection of malware on your site usually indicates that your site was hacked. Comprehensive information on how to clean up each specific malware infection could fill an entire book (and this post is quite long as is). Instead of deep dives into specifics, let’s talk about strategies and resources for combating this problem.

Sources of malware code

There are three primary ways your website might be serving malware:

External source. If hackers exploit an existing security vulnerability to gain access to your source code, they often edit your HTML or script files to make calls to externally based, malicious content on servers they control. Worse yet, hackers don’t even need access to your web server or source files to inflict this attack. If you include externally based content on your pages, and if the hackers can successfully attack the source at its external site, your pages then will unintentionally serve their malware to your customers.
Local source. Sometimes hackers, once they’ve gained full access to vulnerable web servers, put malware code directly in your webpage files and/or place malicious content in the directory structure of your website. Your page’s HTML source code may still appear to be clean, but in this case, the poisoned images, documents, or other binary files they call locally can be the source of the malware attack.
Man-in-the-middle attack. Although this is a less common form of attack due to its technical sophistication, hackers can, when server and network security is severely compromised, inject malware into your webpage content over the network as it travels from your web server to the end user.

Your webpage will be considered malicious if you serve malware from any source, be it from an external server, directly from your web server, or by man-in-the-middle attacks. A user browsing to your webpage from the Bing SERPs will not be able to distinguish your clean content from the malicious content inserted there by hackers. It’s all presented as content in your webpages, so you are ultimately responsible for protecting your customers.

Attack indicators

The malicious code changes that hackers will likely employ come in one or more of these five forms. If any of these elements in your code appear to be suspicious, unexpectedly modified, or unfamiliar to you as webmaster, investigate them further.

Script code. Webmaster should check for new JavaScript script code inserted into their pages. This code will be contained within <script> tags. It is common for inserted, malicious script to be written in an encrypted hash function, accompanied by the decryption key hash value, which allows the script to execute but prevents the webmaster from being able to read and interpret the function of the code. This is known as obfuscated JavaScript code. Such code will look like many wrapped lines of continuous, random alphanumeric characters within a <script> tag. If your pages do not normally use encrypted scripts, this form of script code will definitely stand out as different. This malicious code usually runs when the page is loaded, and it typically appends an exploit or a poisoned, hidden control to the page as it is loaded.
<iframe> code. An <iframe> is simply an HTML tag that enables an unrelated HTML document to be loaded within another HTML document. The <iframe> tag enables hackers to inject poisoned HTML and script code into another webmaster’s webpage. Injected script code often employs <iframe> tags as the means of creating hidden windows that enable malware exploits to execute without the user’s knowledge.
Page redirect. If a hacker get access to edit your home page, they can add code that will automatically and immediately redirect a web browser to another web page (usually to one similarly named and identical looking one on an external server, but possibly to one created on your server) that runs malware as the page loads. This can be done by means of <meta> refresh, JavaScript, or even 301/302 redirects. Unless you find the code for the redirect when you examine your content, you typically won’t see any malware on your page because it’s not executed there. Be sure to also visually inspect your web server configuration for unauthorized redirects.
Externally sourced content. Note that while the use of small, externally-based controls, like hit counters, can be legitimately secure when you first install them, if the webmaster of that control’s host server is not security conscious, those once-benign controls can themselves become malicious vectors later on. Also, small advertising hosts can outsource their contracts to other advertising hosts, who might sub-contract that work out again several times down the line, all done in order to sell more advertising. But the farther you get away from the original trusted external host, the more vulnerable your link becomes to that original, external ad host. Only use external (third party) content from highly trusted sources whose security practices are widely known to be good.
Obfuscation efforts. Attackers often try to hide their exploitation work from quick inspections by using external, referring domain names that are spelled very similarly to known, trusted entities of the Web. Check the spelling of domain names in all external resources to be sure the URLs were not changed to addresses that are similarly named but not the actual, intended target. This includes external references to advertisers, hit counters and other such controls, external images, analytics trackers, and the like. Also, look for URLs that substitute IP addresses for domain names, another common method of obfuscation.

What can you do?

You gotta look at the code. If someone cracked your web server’s security and modified your source code, you need to find what’s changed as the first step in identifying and cleaning up the malware. You can do this by visually inspect the HTML and script code on your pages for unauthorized changes.

When you examine your source code, carefully inspect your code on your web server. Look for newly added scripts in your HTML pages that execute when the page loads, especially obfuscated script. Consider any references to third-party domains in your source code as a potential source of malware. Suspects should include any inserted external code that runs on your site when the page is loaded, including hit counters, images, media content, and other externally sourced controls. External scripts should never be implicitly trusted without a careful consideration of that host’s security practices, as this is a major security vulnerability.

As much as possible, remove unnecessary, externally sourced content to reduce your exposure to exploits beyond your control. Only embed content from trusted third parties into your webpages. If you discover some code that was added to or modified on your page without authorization or realize a once-trusted external page element now appears to be malicious, simply remove that portion of the code from your file to clean it up.

Malware might also have been embedded in your existing images, document files, animations and media content, or other binary files that are presented on your pages. All of these should be scanned again with an antivirus tool for malware.

If you are using a version control system for maintaining your site’s source code, you can easily redeploy the last known good version before the infection occurred. Just be sure that the versioned source code from your workstation is not the source of the malware.

Diagnostic tools to use

To help in your source code examination, use these tools for additional insight on cleaning up a malware mess:

Run an antivirus utility on your source code. Install a fully capable antivirus software tool (and regularly update it to be sure its program code and malware signatures are current) to run a thorough scan of the folders containing your website’s source code. It may be able to detect some forms of malware if they are locally installed on your web server or your webpage files were modified with unauthorized, malicious scripts. Also run a thorough antivirus scan of your personal workstation (the one you use to edit the your site’s source code and connect to the web server for uploads). You may unknowingly infect an otherwise clean web server with a compromised workstation infected with malware. And if you get a key logger infection on your workstation, the hacker controlling that malware might steal your web server’s FTP logon credentials, providing them with full access to attack your site with malicious content.
Run Fiddler HTTP proxy on your website. The Fiddler web debugging proxy tool is a no-cost, web debugging proxy tool used to see what HTTP calls are being made when your page is loaded. By examining the multi-threaded, network traffic generated by your webpages, you can see if your pages are making unexpected calls to unknown resources, and if so, identify where they are going. Watch the Fiddler video tutorials and reads its documentation to learn how this valuable tool is used and how it works.

Checking for man-in-the-middle attacks

You might also inspect your source code as received by your browser using the browser’s View Source command to check for “man-in-the-middle” attacks. In that case, a direct inspection of the original webpage source code files on your web server would likely reveal no malware infection. However, by revealing and examining the source code for the infected webpages from your web browser and comparing the results to the original, clean file from the web server, you might find the malicious changes. If so, inform your web-hosting provider that they might be the victims of a "man-in-the-middle" attack. If your provider takes no action as a result, consider moving your website to a more trusted provider. Luckily, as this is a much more sophisticated attack, it is less common than overt modification of the code on your webpages.

Warning! Make sure both your browser and your operating system are running the latest security updates, along with running up-to-date antivirus, anti-spyware, and software firewall products, to minimize the vulnerabilities to your computer when loading pages likely to be infected with malware.

Also, most web browsers allow you to configure specific security settings for individual sites. Add your infected site to the list. (If your browser doesn’t allow you to specify security settings for individual sites, you can temporarily implement these settings for all sites during your testing, but you may want to revert those changes later to restore full functionality.) You’ll want to disable JavaScripts for your tests. If you’re using Internet Explorer, you’ll also want to disable ActiveX controls. These changes will protect your computer from the infection methods used by malware.

Verifying your fixes

Once you have cleaned up the problem, you should verify your work to be sure the revised code is clean.

Visually inspect the page on the web server to be sure the edits are in place.
Visually inspect the changed page and its source code in your web browser.
Use Fiddler to ensure that the malware’s unexpected external network calls have been eliminated.

A stumper

Sometimes you’ll scan your site’s code and find no clear source of malware, yet malware is clearly affecting the users of your website. If this is the case, look at portions of your code where you take user input without input validation, write cookies to the user’s computer, or other such personalized activity beyond simply displaying information to a generic user. Your site may be the victim of cross-site scripting (XSS). Resolving this specific issue is beyond the scope of this article, but it is very commonly used by hackers for exploiting computer security vulnerabilities, and you should learn how to protect your site against such attacks.

Additional information resources

Microsoft offers a number of useful, anti-malware resources to help you understand what you are up against and what you need to do. Check these out for starters:

Visit the Microsoft Malware Protection Center Threat Research and Response portal for information on malware, Microsoft security products, useful guidance and advice, and more.
Visit the Microsoft TechNet Security TechCenter to access their vast library of security resources, including the articles:

The topic of malware clean up is admittedly not really an introductory level subject, despite this being the SEM 101 column. But the negative implications of detected malware infections on a website are huge. Referrals from Bing will likely dry up after the bots detect malware because of end user protection mechanisms employed on the SERPs to prevent searchers from clicking an infected page. And on top of that, the few customers who choose to circumvent those protections on the SERP or who choose to browse directly to an infected site may possibly suffer the frustrating consequences of a malware infection. Either way, the folks whom you are trying to convert, either with a purchase, a subscription, or a download, will be forced to deal with the unpleasant mess left by the malware picked up from your site. They won’t remain your customers for long. And that’s why this topic needs to be addressed in SEM 101, even though it’s not really a 101-level topic.

-- Rick DeJarnette, Bing Webmaster Center

Temporary glitch for adding sites to Webmaster tools

Webmaster Center team — Thu, 17 Sep 2009 19:15:00 GMT

As some of you may have experienced, our “Add a site” page has been unavailable in some locales for the past several days. This is not a site-wide or even a global issue. Unfortunately, this happened due to an update that uncovered a bug in the original code, which caused us to disable the page in one of our data centers. The good news is that we are testing a fix now and will be releasing it to production just as soon as we are sure it will not cause further complications. I will update this post when I receive word that the fix is in production.

On behalf of all of us here in the Bing Webmaster Center team, we would like to extend our apologies and thank you for your continued patience.

--Brett Yount, Bing Webmaster Center

The merciless malignancy of malware Part 1 (SEM 101)

Webmaster Center team — Fri, 11 Sep 2009 19:15:00 GMT

The Web is an incredible place, filled with amazing media, fascinating content, and wonderful social opportunities, and there’s more of each than anyone can possibly ever consume. But unfortunately, it’s not a benign place. There are more than a few malefactors out there who actively seek to take over your computer for a variety of nefarious purposes. These purposes usually include turning your computer into a:

Member of their computer zombie army, available on command (and to the highest bidder) to execute massive distributed denial-of-service (DDOS) attacks on other web-based computers
Recorder of keystrokes so they can steal passwords to users’ online financial accounts, along with their cash and other, personal data of value to identity thieves
Secret, hidden repository for their stolen and hacked software and pornographic content
Vector for spreading their malicious software (aka malware) to other computers

The people who do this today are usually not the one-off, script kiddies of yore. These miscreants are now often very sophisticated computer software engineers who work for organized criminal groups. And make no mistake: the motive is now profit-based, not simple mischief. These hackers attempt to do all this and more by infecting your computer with a wide variety of malware.

Malware is the name for software created specifically to stealthily install, take control, and perform harmful actions on a computer without the computer owner’s knowledge or permission. Programs such as viruses, worms, Trojan horses, root kits, key loggers, malicious scripts, drive-by downloads, and corrupted program controls are today typically Internet-borne threats, much of it coming from otherwise innocent websites whose content is often secretly hacked.

Many tech savvy users know how to basically protect their computers from these denizens of the dark, but not everyone does. That lapse in universal security consciousness has to include, sad to say, some webmasters and web server hosts. When Bing crawls the Web to gather new and revised content to index, it invariably comes across malware-infected sites. While a few appear to be clear attempts to lure in unsuspecting users like a Venus Flytrap waiting for its next insect meal, a large number of sites appear to be infected from external sources (aka hackers), and the webmasters of these affected sites are almost guaranteed to be innocent victims of sabotage.

This is Part 1 of a three-part series on malware and what webmasters need to know. We’ll cover malware detection (how to tell if your site is infected), strategies and resources for cleaning up (what to do about it), and how to secure computers against the security vulnerabilities that allowed the malware to be injected there (how to stop it from coming back). We’ll also cover what to do once malware is cleaned up so that the Bing index lists your site as being clean again. Let’s get to it!

Detection

So how do you know if your site has unwittingly become a malware vector? It’s not always obvious for webmasters to tell. You can wait for victimized users to send you reports (often in the form of furiously rude complaints!), but by then who knows how many of your site’s visitors have been infected (and how many of them will come back once they determine where the infection came from)?

The search engine crawlers (aka bots) have seen it all. They see the attempted effort to inject malware in drive-by attacks as they crawl the Web. While the bots themselves don’t get infected, they do note the source of the infection attempt in their database.

Wouldn’t you like to peer into that database to see if the bot found malware on your site? Well, I’ve got good news for you. You can! Bing’s Webmaster Center tools offer a peek at what the bot found when crawling your webpages. And unlike the webmaster tools from other search engines, Bing Webmaster Center will show you if we detected malware when we crawled your pages. To get this invaluable insider’s view of your site, you’ll need to first have an account with Webmaster Center. If you don’t yet have an account, follow the instructions at Authenticate your website to set up your account and register your site(s). Note that you’ll need access to either the root directory of your website or to the source code to your site’s default page for deploying a customized authentication code that proves you are the owner of the site. This data about your website is business confidential, after all!

Once your site is registered and can be authenticated, log in to the tools, click the registered site you want to investigate from the Site List page, and then click the Crawl Issues tool tab. In the Select Issue Type drop down list, select Malware Infected. If any infected pages were detected by MSNBot, we’ll identify those pages for you by file name. Note that getting no explicit results in the Malware Infected list is not necessarily the equivalent of a clean bill of health for your entire website. That merely means we didn’t detect malware on the pages that are in the index. To see how many of your site’s pages are in the Bing index, click on the Summary tool tab, and then look at the Indexed pages field. If not every page in your site is indexed, you might remain reasonably suspicious, even with no detected malware. But if any malware was detected, consider this to be a giant red flag hoisted up high. In this case, every page on your site needs to be examined closely, especially those not indexed. A detected malware infection means your site has likely been hacked, and if your site’s security was compromised once, every page should be suspected as dirty until individually verified by you as clean.

You should also click on the Outbound Links tool tab and select the Show only outbound links to malware check box to see if you’re linking to any indexed, malware-infected pages on other sites. If so, you can protect your site’s customers by removing the link to the infected page. It’s also good form to inform your fellow webmaster of what you’ve detected on their site so they can fix the problem and you can restore the link (wouldn’t you want to know if another webmaster found something wrong with your site?).

Implications of a positive result

OK, so unlucky you – your site has one or more pages that were detected as infected with malware. What does this mean? Do you really need to fix it? Well, let’s address these questions by describing what Bing does with malware-infected sites.

Through the use of its malware filter and the drive-by download detection features, Bing helps protect its users against a variety of malware infections whenever possible. These protections either identify and remove malware sites from our search engine results pages (SERPs) or block access to infected URLs. If your malware-infected page does show up in the Bing SERP, the blue link to your page will be disabled. When a user clicks on the disabled link, instead of going to your page, they will see a malware warning box pop up to the right of the SERP listing. The pop up warning box looks like the following example:

A recent study at Microsoft revealed that 98% of searchers who get a malware notification will heed the warning and opt to not click the visit the website link in the warning message. That means that if your site is flagged by Bing as malware-infected, your search engine referral traffic will drop off the charts! As such, it is in your best interest as webmaster to rectify the malware issue so that you can get your search engine referral business back in gear!

In the next article of this series on malware, we’ll dive into strategies and identify resources for cleaning up a malware mess. If you have any questions or comments about malware, please feel free to post them in our General Questions forum. For regular SEM and SEO questions and suggestions, please go to our SEM forum. Until next time…

-- Rick DeJarnette, Bing Webmaster Center

Search Engine Optimization for Bing

Webmaster Center team — Thu, 03 Sep 2009 22:20:21 GMT

When I attended both SMX Advanced in Seattle back in June and SES San Jose just a couple of weeks ago, I heard a lot of questions from webmasters about Bing, especially pertaining to search engine optimization issues. Typically these included:

I want to do SEO for Bing—where should I start?
How is Bing different in terms of SEO?
What do webmasters need to know and do?
Are there any insider tips for successful ranking?

I’ll tackle these questions by providing some useful, baseline information and include pointers to more detailed, pertinent docs.

As you know, Bing is an evolution in the search engine space. With its innovative, new user interface (UI) design bringing new depth and opportunities for searchers, they can now quickly find the information they seek when they search the Internet. New UI features, such as Quick Tabs, Related Searches, and Document Preview (to name just a few), surface more information and present more opportunities to discover what searchers want to know so they can make more informed decisions more quickly. As a result, we describe Bing as a decision engine. (For more information on the new UI features in Bing, see the Bing Webmaster Center blog post, Bing white paper for webmasters & publishers released.)

Under the covers of the new UI, we do a lot of engineering work on a very large scale. For example, we crawl a variety of content types found on the Web, index that content, apply appropriate algorithms, and finally send relevant content to user queries in our search engine results pages (SERPs).

Bing’s SEO principles

SEO is fundamentally about creating websites that are good for people. The most basic advice we can give for achieving optimum rank for your site in Bing is to do the following:

Develop great, original content (including well-implemented keywords) directed toward your intended audience
Use well-architected code in your webpages (including images and Sitemaps) so that users’ web browsers and search engine crawlers can read the content you want indexed)
Earn several, high-quality, authoritative inbound links

As you can see by the links, much of this material has already been discussed in-depth in the Webmaster Center team blog in our ongoing column, search engine marketing (SEM) 101.

The type of SEO work and tasks webmasters need to perform to be successful in Bing hasn’t changed—all of the legitimate, time-tested, SEO skills and knowledge that webmasters have invested in previously apply fully today with Bing. Moreover, investments in solid, reputable SEO work made for Bing will bring similar improvements in your website’s page rank in other search engines as well.

Ultimately, SEO is still SEO. Bing doesn’t change that. Bing’s new user interface design simply adds new opportunities to searchers to find what the information they want more quickly and easily, and that benefits webmasters who have taken the time to work on the quality of their content, website architecture, and have done the hard work of earning several high-quality inbound links.

Key content and tools for performing SEO with Bing

To keep up with the latest and greatest information coming from the Bing Webmaster Center team, we recommend that you follow and review the following content:

Review the Bing official guidelines for successful indexing document for various recommendations on technical and content issues as well as known problems that can affect your site’s rank
Visit the Webmaster Center blog to keep up with the latest information from the team (you can even subscribe to our blog’s RSS feed to automate this process)
Register all of your websites with Bing Webmaster Center tools, where you can use our tools to see all sorts of data to your website pertinent to webmasters
Participate in our Webmaster Center user forums to ask questions and provide us with feedback

We look forward to working with you as partners in helping our mutual customers find the information they seek on the Internet.

-- Rajesh Srivastava, Principal Group Program Manager, Bing

How Microsoft handles bots clicking on ads

Webmaster Center team — Tue, 25 Aug 2009 16:51:26 GMT

There’s been some recent discussion in the SEO blogosphere asserting that Bing clicks its own adCenter ads. This has created some misunderstanding. Let’s take a moment to clarify what is actually happening, and what this really means for webmasters and advertisers.

The Bing team is aware of an issue shared by all search engines: paid advertising links on sites are, on occasion, crawled and indexed by search engines. Standard practice in the search industry is to scan web pages for the purpose of indexing and understanding the site’s content, and to determine which ads match best the destination site. Microsoft adCenter does not charge an advertiser for clicks generated by any known search engine bots, including our own.

AdCenter uses a variety of techniques to remove bots, including the Interactive Advertising Bureau’s (IAB) Spiders and Robots protocol. The IAB provides a list of known bots, and Microsoft bots are a part of that list. As a result, any activity generated by bots will not skew AdCenter data because it will be categorized as low quality in AdCenter Reports. You can view the Standard Quality and Low Quality data by accessing the AdCenter Reports tab.

In June, 2009, Microsoft received Click Quality Accreditation from the IAB, which holds the industry’s highest standards in click measurement. The IAB and independent third-part auditors verified that adCenter meets their requirements for Click Quality Accreditation, which includes not billing for our search bot’s ad clicks. For more information, visit the adCenter Blog, or the IAB site.

This issue exists for all search engines, and we all follow the practice of not charging for bot-driven ad clicks. We maintain the integrity of our engine and our advertiser’s experience as a very high priority, and welcome your feedback. Please visit our Webmaster Crawling/Indexing Discussion forum to leave your comments and questions.

-- Rajesh Srivastava, Principal Group Program Manager, Bing

Prevent a bot from getting “lost in space” (SEM 101)

Webmaster Center team — Fri, 21 Aug 2009 00:32:25 GMT

We recently published a non-SEM 101 blog post on controlling the crawl rate of MSNBot, the Bing web crawler (aka robot, or simply just bot). That got me thinking about robots. Naturally, that led to The Robot on Lost in Space. Will Robinson, the show’s precocious youngster who was a whiz at 1960s-style, clunky electronics (even though the show was supposedly set in 1997!), was best friends with The Robot. They looked out for each other and helped each other in times of need.

In a way, search engine bots and webmasters have a similar relationship. They need one another. Webmasters need search engine bots to crawl the pages of their sites so that they can be added to their indexes. Bots need webmasters to provide them with compelling content, well-formed code, and authoritative backlinks to serve to their search customers in search engine results pages (SERPs). This form of a “mutualistic, symbiotic” relationship is beneficial to both parties. Webmasters benefit from getting high quality content into the search engine index. Search engines benefit by being able to provide searchers with useful, relevant results to their queries, no matter how arcane.

The Robot character was actually quite powerful in its analytical capabilities, and would alert the Robinson family when danger lurked (although I never understood why it did not sense the cloak-and-dagger danger in stowaway Dr. Smith). But most important was young Will Robinson’s ability to communicate with The Robot. He could direct The Robot’s powerful intelligence toward things that needed its attention (such as your average, run-of-the-mill, hostile space aliens, who coincidently always looked like expressionless, rubber masked-humans!). The Robot’s assistance helped Will survive his rough and tumble alien environment.

As a webmaster, you, too, can communicate with the robot (the search engine variety). You can block it from crawling specified directories and files, override the generic crawl block for a subset of those files, block specified pages from being indexed, block the following of links on a page, and much more. Let’s take a look at how you can converse with your friendly search engine robot and help it navigate your website with an eye toward directing its behavior for crawling and indexing your content. This effort might help your site better survive in the rough and tumble environment of the Web.

While Will Robinson conversed with The Robot in spoken English, you’ll need to communicate with the search engine bots using the Robots Exclusion Protocol (REP). You can do this in two ways (depending on what you want to do): within the HTML code of each page or in a separate file named robots.txt (which you save to the root directory of your website). While not a perfect analogy, consider the scope of the message to the robot for defining how you tell it what to do. For site- or directory-wide directives, you’ll typically use the robots.txt file. Page- and even link-specific directives are more often handled in a page’s HTML code.

The robots.txt file

The robots.txt file is commonly used to block bots (identified as user-agents within the context of the file) from accessing directories and files that contain data the webmaster doesn’t want added to the search index, such as scripts, databases, and other information that is not intended for public consumption or has no value for searchers. For example, a basic robots.txt file might include directives such as the following:

User-agent: *
Disallow: /private/

The above sample robots.txt file code applies to all user-agents (bots to you and me) and blocks bot access to all files in the directory named /private on the web server.

But you can do more than that with the robots.txt file. What if you actually had a ton of existing files in /private but actually wanted some of them made available for the crawler to see? Instead of re-architecting your site to move certain content to a new directory (and potentially breaking internal links along the way), use the Allow directive. Allow is a non-standard REP directive, but it’s supported by Bing and other major search engines. Note that to be compatible with the largest number of search engines, you should list all Allow directives before the generic Disallow directives for the same directory. Such a pair of directives might look like this:

Allow: /private/public.doc
Disallow: /private/

Note If there is some logical confusion and both Allow and Disallow directives apply to a URL, the Allow directive takes precedent.

Wildcards

The use of wildcards is supported in robots.txt. The “*” character can be used to represent characters appended to the ends of URLs, such as session IDs and extraneous parameters. Examples of each would look like this:

Disallow: */tags/
Disallow: *private.aspx
Disallow: /*?sessionid

The 1^st line in the above example blocks bot access to any URL that contains a directory named “tags,” such as “/best_Sellers/tags/computer/”, “/newYearSpecial/tags/gift/shoes/”, and “/archive/2008/sales/tags/knife/spoon/”. The 2^nd line above blocks all URLs that end with the string “private.aspx”, regardless of the directory name (note that the preceding forward slash is redundant and thus not included). The last line above blocks access to any URL with “?sessionid” anywhere in their URL string, such as “/cart.aspx?sessionid=342bca31?”.

Notes

The last directive in the sample is not intended to block file and directory names that use that string, only URL parameters, so we added the “?” to the sample string to ensure that work as expected. However, if the parameter “sessionid” will not always be the first in a URL, you can change the string to “*?*sessionid” so you’re sure you block the URLs you intend. If you only want to block parameter names and not values, use the string “*?*sessionid=”. If you delete the “?” from the example string, this directive will block URL containing file and directory names that match the string. As you can see, this can be tricky, but also quite powerful.
A trailing “*” is always redundant since that replicates the existing behavior for MSNBot. Disallowing “/private*” is the same as disallowing “/private”, so don’t bother adding wildcard directives for those cases.

You can use the “$”wildcard character to filter by file extension.

Disallow: /*.docx$

Note The directive above will disallow any URL containing the file name extension string “.docx” from being crawled, such as a URL containing the sample string, “/sample/hello.docx”. In comparison, the directive Disallow: /*.docx blocks more URLs, as it applies to more than just file name extension strings.

Sitemaps

If you have created an XML-based Sitemap file for your site (as discussed in the recent blog post Uncovering web-based treasure with Sitemaps), you can add a reference to the location of your Sitemap file at the end of your robots.txt file. The syntax for a Sitemap reference is as follows:

Sitemap: http://www.your-url.com/sitemap.xml

Other issues

You can also add a crawl-delay directive to your robots.txt file to change the default pace at which Bing crawls your site. I’ll do my part in conserving electrons and avoiding redundancy by instead referring you to that post: Crawl delay and the Bing crawler, MSNBot.

Whatever you choose to do with robots.txt, don’t play games with constant changes to the file. Note this story from our blog about a webmaster who was having problems with getting a site properly crawled, and we discovered they were swapping out differently configured robots.txt files in an automated fashion in a misguided effort to control crawling. That was not a helpful strategy!

File format

The robots.txt file must be saved in a standard text file format, such as ASCII or UTF-8, so it can be read by the bots. One easy way to verify that the proper file format is used is to edit the file in Microsoft Notepad. Save the file using the Notepad default file format type, Text Documents (*.txt) with ANSI encoding.

Validation

Once your robots.txt file is built, I suggest that you validate it before you consider it done. If you are a member of Bing Webmaster Center, log in to get access to our online Robot.txt validation tool. Otherwise, there are a number of other online robots.txt validators available for you to use. We all want to avoid having The Robot say, “Warning! Warning! That does not compute!”

Maintenance

Once you’ve got a good robots.txt file built and validated, don’t just set it and forget it. Periodically audit the settings in the file, especially after you’ve gone through a site redesign. You need to be sure your blocking directives are still valid so that nothing is unintentionally blocking the bot from valid content or leaving sensitive material exposed.

HTML code

You can also put REP directives directly in your HTML code. Recall a few weeks back we discussed the way to optimize the <head> tag and got into how to use <meta> tags? Well, that is where most of these REP directives go as well. Let’s take a look at how these work. Here’s a sample <meta> tag that addresses bots:

There are more options than the two listed. REP values for the content attribute are designed to be aggregated to combine functionality so that it performs just as you want.

Meta tags

The name=”robots” attribute of the <meta> tag is read by the bot when it accesses an HTML page. The directives listed in the content attribute tell it what (or more specifically, what not) to do. Let’s define the function of each of the content attribute values.

Value	Function
noindex	Prevents the bot from indexing the contents of the page, but links on the page can be followed. This is useful when the page’s content is not intended for searchers to see.
nofollow	Prevents the bot from following the links on the page, but the page can be indexed. This is useful if the links created on a page are not in control of the webmaster, such as with a blog or user forum. Links to spam or malware are not what careful webmasters want to serve to their customers!
nosnippet	Instructs the bot to not display a snippet for that page in the SERPs. Snippets are the text description of the page shown between the page title and the blue link to the site.
noarchive	Instructs the bot to not display a cache link for that page in the SERP.
nocache	Same as noarchive.
noodp	Instructs the bot to not use a title and snippet from the Open Directory Project (ODP) for that page in the SERP.

Note that the HTML-based, bot-blocking directives within a page will override Allow directives found applying to an HTML file in robots.txt.

Links

What if you want almost all of the links on a page followed, but for some reason, you want to block the bot from following one or a few? Well, there is a solution for that as well: the rel=”nofollow” attribute. To see it in action, look at the following sample anchor tag:

<a rel="nofollow" href="http://www.untrustedsite.com/forum/stuff.aspx?var=1”>Read these forum comments</a>

Caveats

Note that with the rel=”nofollow” attribute, a REP-compliant bot will not follow that specific link on that page. However, if any other page on the site (or a link on an external site) refers to the blocked page without any REP directives blocking it, the page may still be crawled and could make it into the index. This caveat goes for all REP link blocking directives that are not consistently applied to a specified page. And in the case of external sites linking to the page on your site that is supposed to be blocked, which local webmasters cannot control, these pages may still be crawled and indexed. In this case, the <meta> tag’s noindex solution for that page is the best option.

I discuss this attribute in some detail at the end of a previous blog article. I’ll again conserve electrons by referring you to Making links work for you. Robots appreciate that kind of thing.

For more information on REP, see our past big blog article on the subject, Robots Exclusion Protocol: joining together to provide better documentation.

Judicious use of REP directives will help shape your website’s presence on the SERPs. It’ll help direct the search engine bots to content you want them to see and block them from content you do not want indexed, be it because the content is not useful to searchers (such as shopping cart or logon pages) or because it contains potentially business confidential data (such as references to internal IT infrastructure in scripts). Your efforts to direct the traffic here will help prevent bots from getting “lost in space.” It’s only too bad that the search engine bots don’t wave their virtual arms and trumpet “Danger, Will Robinson!” whenever they can access content not intended for the index! But that’s your call to make, not theirs.

If you have any questions, comments, or suggestions, feel free to post them in either our SEM forum or our Crawling/Indexing Discussion forum. Later…

-- Rick DeJarnette, Bing Webmaster Center

Getting the IIS SEO Toolkit up and running

Webmaster Center team — Mon, 17 Aug 2009 21:26:00 GMT

We recently published a popular blog post called Get detailed site analysis to solve problems that highlights the function and capabilities of the new, beta search engine optimization (SEO) Toolkit from the Microsoft Internet Information Server (IIS) team. The tool works as an extension to the latest version of IIS, version 7.

We’ve received some webmaster feedback with specific questions on how to set up IIS 7 before installing the IIS SEO Toolkit. So to help simplify and clarify this task for webmasters (as clearly folks are very much interested in using the tool!), we developed this quick “How-to” post with detailed instructions for the set up of IIS 7. We hope this helps!

Minimum installation requirements

The IIS SEO Toolkit will run on any version of Microsoft Windows capable of running IIS 7. This includes:

Windows Vista Service Pack (SP)1 and higher (only available in the Vista versions Home Premium, Business, and Ultimate)
Windows Server 2008
Windows 7
Windows Server 2008 R2

The IIS SEO Toolkit, due to its reliance on IIS 7, is not supported on Windows XP, Windows Server 2003, or on any alternative operating system.

Note: You must be a member of either the Administrator or Web Server Administrator groups to perform this set up procedure.

Enable IIS 7

Before you can install the IIS SEO Toolkit, you first need to enable IIS 7 on your computer. In both Windows Vista and Windows 7, this will install the web server software but will not, by default, open port 80 for inbound network traffic. To set up IIS 7, follow these instructions:

1. Click Start.

2. Click Control Panel.

3. Click Programs.

4. Select Turn Windows features on or off (as shown in the figure below). If you receive a Windows Security warning, click Allow.

5. In the resulting Windows Features dialog box, select the Internet Information Services check box (as shown in the figure below).

6. Click + to expand the view to see the following nodes: World Wide Web Services, and then Application Development Features. Select the .NET Extensibility check box (as shown in the figure below).

7. Click OK. The activation of the newly installed components may take a few minutes to complete.

For more information in installing IIS 7, see the article Installing IIS 7 on Microsoft TechNet.

Install the IIS SEO Toolkit

Once IIS 7 is enabled, follow these instructions to install the IIS SEO Toolkit extension.

You’ll need to download the version of the toolkit corresponding to the type of Windows you are running (either 32-bit or 64-bit). If you are not sure what type you have installed, you can check by doing the following:

Click Start.
Right-click Computer, and then click Properties.
In the System group, System Type will indicate whether you are running a 32-bit or a 64-bit version of Windows.

To install the IIS SEO Toolkit, click the correct link below for your installed version of Windows to download the program installation file. You can choose to run the downloaded program directly or save it locally and then start it. After starting the installation of the IIS SEO Toolkit, following the instructions as prompted.

For an introduction to and information on the features of the IIS SEO Toolkit, see the Webmaster Center blog post called Get detailed site analysis to solve problems. Thanks for taking the IIS SEO Toolkit for a test spin! We think you’ll be impressed!

--Alessandro Catorcini, Lead Program Manager, Bing API & Rick DeJarnette, Bing Webmaster Center

Uncovering web-based treasure with Sitemaps (SEM 101)

Webmaster Center team — Fri, 14 Aug 2009 23:09:00 GMT

Have you ever noticed how pirate treasure maps are like Sitemaps? While your website may not contain a treasure of gold and silver (unless it’s a metals commodities trading site!), if you have good content, that is certainly treasure to someone who is looking for it. Unfortunately, it’s buried on your website and no one knows what’s there except you! But since you want to share your site’s treasure with others, you need to let them know what you have buried and where to find it. You can wait for search engine crawlers (aka bots) and random traffic to come by to browse, but that will take time and even then, they might not discover everything that you have to offer. Instead, you can help the search bots to dig up your treasured content with a Sitemap.

Now I should pause for a moment to mention that you shouldn’t confuse sitemaps with Sitemaps. You’ve got that, right? Well, just in case that’s as clear as mud, keep this in mind: when referring to sitemap files in text (such as this!), use the lower case word “sitemap” to mean HTML-based files intended for users to browse. They typically contain a list of all the pages on your site.

On the other hand, use the capitalized word “Sitemap” to mean XML-based files designed for use by search engine bots to collect data from webmasters identifying the most important pages and directories within their sites for crawling and indexing. Both types of sitemap files can (and probably should) use all lower case letters in their file name (such as sitemap.xml and sitemap.htm), but capitalize the references to the XML-based one in text to help readers distinguish which type of sitemap you are discussing. This article, coming from the perspective of a search engine, is focusing on Sitemaps, not sitemaps. You’re with me now, right? :-)

A good Sitemap will tell search engine bots about the content stored on a site. That helps the content be seen by the bot and, with any luck (assuming the content is well formed and has value), get into the index. Users who are on a content treasure quest will query search engines with keywords to locate the content they are seeking. If the search engine indexed the content found by the bot, which can be more likely when a good Sitemap is present, then that site’s content has a better chance for appearing in the search engine results pages (SERP). After all, you can’t get onto the SERP if your pages aren’t indexed!

Structure

A Sitemap file, saved to the root directory of your site, contains references to specific URL locations for pages (or to other Sitemap files on very large sites), often describing the last modified date, the typical change frequency for a page, and the priority the specified page has compared to the other content on your site.

A brief example of the contents of a Sitemap file looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
   <url>
     <loc>http://www.mysite.com/default.htm</loc>
     <lastmod>2009-03-01</lastmod>
     <changefreq>monthly</changefreq>
     <priority>0.8</priority>
   </url>
   <url>
     <loc>http://www.mysite.com/contacts.htm</loc>
     <changefreq>yearly</changefreq>
     <priority>0.4</priority>
   </url>
</urlset>

The <urlset> tag is standard and points to the current protocol to reference. The <url> and <loc> tags are the minimum required data needed for each page entry. The other tags, <lastmod>, <changefreq>, and <priority>, are optional, additional data. To see the data entry formatting and attributes used for these optional tags, sail on over to Sitemaps XML format for reference information. Note that not every page on your site need be listed in the Sitemap—only the ones containing valuable content for the user.

File formats

Bing supports Sitemap files submitted as XML and gzip files, but not as HTM or HTML files (those would be sitemaps as opposed to Sitemaps, right? Besides, the XML content of a well-formed Sitemap file wouldn’t render correctly in browsers as an HTM file, anyway). If you’ve created a browsable HTML-based sitemap for your end users, they will thank you for the effort, but you can’t recycle it as a Sitemap. You’ll still need to create a separate, XML-based Sitemap file using the tag structure as noted above for submission to Bing.

Size matters not so much anymore

A typical treasure map only has one X to mark the spot. However, your Sitemap can list multiple locations identifying the treasures on your site. It used to be considered common wisdom in the search engine optimization (SEO) community that there can be too much of a good thing. It used to be accepted that from a search engine perspective, the most effective size for a Sitemap is approximately 150 or fewer URLs. Anything more bountiful and the crawler may not take it all in. Well not so fast, matey!

Per the post Bing enhances support for large Sitemaps made in this blog just a few weeks ago, Bing now supports Sitemap files that contain up to 50,000 references (to either URLs or links to other, child Sitemap files). This development is a boon for webmasters of very large sites. They can now create multiple child Sitemap files, each dedicated to mapping specific areas of their content organization, and store those child Sitemap files in the base directories of those content areas. Then they can link to the child Sitemaps via their primary (aka index) Sitemap file (the one stored in the root directory of a site). One index Sitemap linking to 50,000 child Sitemaps, each of those referencing up to 50,000 URLs, means they can reference up to 2.5 billion URLs through the Sitemap technology, and the Bing crawler, MSNBot, will read it all. Now that’s a lot of treasure!

Sitemap submission

There are multiple ways for webmasters to submit their Sitemaps to Bing:

Ping service. Using your browser’s address bar, you can directly submit your Sitemap to Bing. Type

http://www.bing.com/webmaster/ping.aspx?sitemap=www.YourURL.com/sitemap.xml

substituting the full URL to your Sitemap file in place of the YourURL.com example.
Webmaster Center tools. You can sign in to Bing’s Webmaster Center tools and use the Sitemaps tool (if you are not already registered to use these free tools, this is a good reason to sign up and see all the other tools available to help you analyze and optimize your site). Simply copy the URL of your Sitemap into the Direct sitemap submission text box, and then click Submit.
Robot.txt file reference. If you are using a robots.txt file to instruct search engine bots which files and directories not to crawl and thus block from adding to their indexes, you can add a line to that file, most typically done at the end, that reads as follows:

Sitemap: http://www.YourURL.com/sitemap.xml

substituting the full URL to your Sitemap file in place of the YourURL.com example.

Validation

Before submitting your Sitemap to Bing, we recommend that you run the XML code you’ve written through a Sitemap validation tool. After all, what good is a treasure map if it has errors in it? Do a search for your Sitemap validator of choice and follow the instructions on the page. If errors are found, correct them before you submit the Sitemap file to Bing.

Once you submit your Sitemap to Bing, we will read its contents, which will help us with uncovering more of the content treasures buried on your site and evaluating it as potential new additions to our index. And with more of your site’s content in the index, instead of users sailing on past your site to other ports of call in their quest for content treasure, they may stop at yours and exclaim, “Shiver me timbers, matey, look at what we have here!”

If you have any questions, comments, or suggestions, feel free to post them in our SEM forum. Until next time…

-- Rick DeJarnette, Bing Webmaster Center