CarlosAg Blog

Winsxs is huge… Free up a few Gigabytes with dism

CarlosAg — Wed, 04 Jul 2012 16:58:00 GMT

I was running out of disk space in C: and was unable to install a small software that I needed, so I decided to clean up a bit. For that I like using WinDirStat http://windirstat.info/ which very quickly allows you to find where the big files/folders are. In this case I found that my c:\Windows\winsxs folder was over 12 GB of size. One way to reclaim some of that disk space is to cleanup all files that have been backed up when a Service Pack has been installed. To do that in Windows 7 you can run the following DISM command:

dism /online /cleanup-image /spsuperseded /hidesp

That freed up 4 GB in my machine and now I can move on.

Disclaimer: I only ran this in my Windows 7 machine and it worked great, have not tried it in Server SKUs so run at your own risk.

It has been a long time since last post

CarlosAg — Fri, 16 Mar 2012 17:22:57 GMT

Wow, just realized that in the last 6 months I’ve only had a chance to post 2 items and I think it is about time to start this going again.

So why this much silence? Well, About 8 months ago a couple of big changes happened at my division as described in this link. As part of that transition my responsibilities changed and I transitioned from being the Development Manager for the Web Platform (IIS, WebMatrix, WebDeploy, etc…) to take a new role and start a new team that we called Azure UX team. Our team is in charge of reimagining the Windows Azure User Experience and we stated on a mission to really make it even better. As part of that we’ve been super busy delivering a set of projects, some of which we released in December such as our brand new Windows Azure web site at http://www.windowsazure.com/ and the new Windows Azure Billing Web site at https://account.windowsazure.com/, several updates to the Windows Azure Management Portal at http://windows.azure.com/ and some that have not yet been released.

These last months have been quite a ride, I feel privileged of having been part of the IIS family and community and its multiple releases since even before Windows Vista shipped, 7, 7.5, ARR, URL Rewrite, Web Deploy, WebMatrix, WebPI, IIS Media Services, FTP, WebDAV, SEO Toolkit, WFF, FastCGI, WinCache, AdminPack, PowerShell, Database Manager, Dynamic IP Restrictions, and many many more releases my team delivered, and obviously, the awesome new version in Windows 8. I will always remain close to all of them, both the people, the products, and the community.

At the same time, I’m extremely happy and excited to also have now a chance to be part of such an amazing group of people the AUX team, a very fast pace, dedicated and professional group, working on a new mission that will influence so deeply the shape of the cloud.

Using Windows Authentication with Web Deploy and WMSVC

CarlosAg — Wed, 14 Dec 2011 02:12:09 GMT

By default in Windows Server 2008 when you are using the Web Management Service (WMSVC) and Web Deploy (also known as MSDeploy) it will use Basic authentication to perform your deployments. If you want to enable Windows Authentication you will need to set a registry key so that the Web Management Service also supports using NTLM. To do this, update the registry on the server by adding a DWORD key named "WindowsAuthenticationEnabled" under HKEY_LOCAL_MACHINE\Software\Microsoft\WebManagement\Server, and set it to 1. If the Web Management Service is already started, the setting will take effect after the service is restarted.

For more details on other configuration options see:

http://technet.microsoft.com/en-us/library/dd722796(WS.10).aspx

Razor Migration Notes 3: Use app_offline.htm to deploy the new version

CarlosAg — Fri, 19 Aug 2011 06:02:00 GMT

This is the third post on the series:

1: Moving a SitemapPath Control to ASP.NET Web Pages

2: Use URL Rewrite to maintain your Page rankings (SEO)

ASP.NET has a nice feature to help for deployment processes where you can drop an HTML file named app_offline.htm and it will unload all assemblies and code that it has loaded letting you easily delete binaries and deploy the new version while still serving back to customers the friendly message that you provide telling them that your site is under maintenance.

One caveat though, is that Internet Explorer users might still see the “friendly” error that they display and not your nice message. This happens because of a page size validation that IE performs. See Scott’s blog on how to workaround that problem: App_Offline.htm and working around the IE Friendly Errors

Note: The live site is now running in .NET 4.0 and all using Razor.

Razor Migration Notes 2: Use URL Rewrite to maintain your Page rankings (SEO)

CarlosAg — Sat, 13 Aug 2011 17:46:39 GMT

This is the second note of the series:

1: Moving a SitemapPath Control to ASP.NET Web Pages

My current Web Site was built using ASP.NET 2.0 and WebForms, that means that all of my pages have the extension .aspx. While moving each page to use ASP.NET Web Pages their extension is being changed to .cshtml, and while I’m sure I could configure it in a way to get them to keep their aspx extensions it is a good opportunity to “start clean”. Furthermore, in ASP.NET WebPages you can also access them without the extension at all, so if you have /my-page.cshtml, you can also get to it using just /my-page. Given I will go through this migration I decided to use the clean URL format (no extension) and in the process get better URLs for SEO purposes, for example, today one of the URLs look like http://www.carlosag.net/Articles/configureComPlus.aspx but this would be a good time to enforce lower-case semantics and also get rid of those ugly camel casing and get a much more standard a friendly format for Search Engines using “-“, like: http://www.carlosag.net/articles/configure-com-plus.aspx.

Use URL Rewrite to make sure to keep your Page Ranking and no broken links

The risk of course is that if you just change the URLs of your site you will end up not only with lots of 404’s (Not Found), but your page ranking will be reset and you will loose all the “juice” that external links and history have provided to it. The right way to do this is to make sure that you perform a permanent redirect (301) from the old URL to the new URL, this way Search Engines (and browsers) will know that the content has permanently moved to a new location so they should “pass all the page ranking” to the new page.

There are many ways to achieve this, but I happen to like URL Rewrite a lot, so I decided to use it. To do that I basically created one rule that uses a Rewrite Map (think of it as a Dictionary) to match the URL and if it matches it will perform a permanent redirect to the new one. So for example, if /aboutme.aspx is requested, then it will 301 to /about-me:

<?xml version="1.0"?>
<configuration>
<system.webServer>
    <rewrite>
      <rules>
        <rule name="Redirect for OldUrls" stopProcessing="true">
          <match url=".*"/>
          <conditions>
            <add input="{OldUrls:{REQUEST_URI}}" pattern="(.+)"/>
          </conditions>
          <action type="Redirect" url="{C:1}" appendQueryString="true" redirectType="Permanent" />
        </rule>
      </rules>
      <rewriteMaps>
        <rewriteMap name="OldUrls">
          <add key="/aboutme.aspx" value="/about-me"/>
          <add key="/soon.aspx?id=1" value="/coming-soon"/>
          <add key="/Articles/configureComPlus.aspx" value="/articles/configure-com-plus"/>
          <add key="/Articles/createChartHandler.aspx" value="/articles/create-aspnet-chart-handler"/>
          <add key="/Articles/createVsTemplate.aspx" value="/articles/create-vs-template"/>
      ...
        </rewriteMap>
      </rewriteMaps>
    </rewrite>
</system.webServer>
</configuration>

Note that I could have also created a simple rule that would change the extension to cshtml, however I decided that I also wanted to change the page names. The best thing is that you can do it incrementally and only rewrite them once your new page is ready or even switch back to the old one later if any problems occur.

Summary

Using URL Rewrite you can easily keep your SEO and pages without broken links. You can also achieve lots more, check out: SEO made easy with IIS URL Rewrite 2.0 SEO templates – CarlosAg

Razor Migration Notes 1: Moving a SitemapPath Control to ASP.NET Web Pages

CarlosAg — Sun, 07 Aug 2011 16:56:00 GMT

After many years I decided that it is time to rewrite my Web site using Razor. A bit of history, I started it around 2003 using ASP.NET 1.1. When .NET 2.0 came around in 2005 I migrated to it and it was great being able to leverage features like MasterPages, Themes, Sitemaps, and many other features. Honestly it is a pretty simple Web site, with mostly content, so very few controls, Sitemap, my own custom Menu control, and a couple more. Last week it was moved to use .NET 4.0 and it feels its about time to go back and update it a bit, both in look and features. So this (if time permits) will be the first of a series of migration notes that I discover as I move it to use ASP.NET Razor (aka WebPages). Do note that this is not meant to be a best practice in anyway, I would never claim I can make such a thing, these will be only my personal notes as I discover more details in ASP.NET WebPages features and as I move my own implementation to use them.

So with that, one of the first things I faced during this migration, was the use of a Sitemap control (asp:SiteMapPath) in my MasterPage (future post about moving from MasterPages coming). I knew about Sitemap API, so I just decided to write a simple Sitemap helper that I can now use anywhere in Razor. The code is pretty simple, it basically generates an unordered list of links using <ul> and <li> with <a> inside, and used CSS to layout them in a way that I liked.

SitemapPath Control in WebForms

The original code I was using in my MasterPage looked like the following:

<asp:SiteMapPath CssClass="HeaderText" runat="server" ID="siteMap" ShowToolTips="true" NodeStyle-ForeColor="White" CurrentNodeStyle-Font-Bold="true" />

And generated the following markup:

<span id="siteMap" class="HeaderText"><a href="#siteMap_SkipLink"><img alt="Skip Navigation Links" height="0" width="0" src="http://blogs.msdn.com/WebResource.axd?d=S2jbW9E-HYlS0UQoRCcsm94KUJelFI6yS-CQIkFvzT6fyMF-zCI4oIF9bSrGjIv4IvVLF9liJbz7Om3voRpNZ8yQbW3z1KfqYr4e-0YYpXE1&t=634219272564138624" style='border-width:0px;' /></a><span><a title='Home' href='/' style='color:White;'>Home</a></span><span> > </span><span><a title='Free tools for download' href='/Tools/' style='color:White;'>Tools</a></span><span> > </span><span style='color:White;font-weight:bold;'>Code Translator</span><a id='siteMap_SkipLink'></a></span>

Which looks like the following in the browser:

I used some CSS to set the color, and background and other stuff, but still to set the last item to bold required me to use a property in the Sitemap to get it to look the way I wanted.

My Sitemap Helper in Razor

Since I was familiar with the Sitemap API and my goal was to change as “little” as possible as part of this first migration, I decided to write a Sitemap helper that I can use in my Layout pages. The code in the Page is as simple as it gets, you just call @Helpers.Sitemap() and that’s it (added the Div below to get some context in the markup, but that was already there with the SitemapPath control anyway):

<div class="bannerPath">
@Helpers.Sitemap()
</div>

This new helper version generates the markup below. I don’t know about you, but I can sure make more sense of what it says, and I imagine Search Engines will as well, I decided to use more semantically correct markup using a <nav> to signal navigational section and use a list of links.

<nav>
    <ul class="siteMap">
        <li><a href="http://blogs.msdn.com/" title="Home">Home</a> > </li>
        <li><a href="http://blogs.msdn.com/Tools/" title="Free tools for download">Tools</a> > </li>
        <li><span>Code Translator</span></li>
    </ul>
</nav>

And it looks like the following in the browser (I decided to remove the underlining, and have more padding, and a new font, but all of that is CSS):

The Sitemap helper code

The code to do the Sitemap was pretty simple, just use the SiteMap API to get the current node. Since I’m picky and I wanted to generate the markup in the “right” order (note you could use CSS to float them to the right instead), I used a Stack to push the nodes while traversing them up. Finally just generate the <li>.

@helper Sitemap()
{
    SiteMapNode currentNode = SiteMap.CurrentNode;
    <nav>
    <ul class="siteMap">
    @if (currentNode != null)
    {
        // Push into a stack to reverse them
        var node = currentNode;
        var nodes = new Stack<SiteMapNode>();
        while (node.ParentNode != null)
        {
            nodes.Push(node.ParentNode);
            node = node.ParentNode;
        }

        while(nodes.Count != 0)
        {
            SiteMapNode n = nodes.Pop();
            <li><a href="@n.Url" title="@n.Description">@n.Title</a> > </li>
        }
        <li><span>@currentNode.Title</span></li>
    }
    else
    {
        <li><span>@Page.Title</span></li>
    }
    </ul>
    </nav>
}

To make it look the way I wanted I used the following CSS:

.siteMap

{ float:right; font-size:11px; color:White; display:inline; margin-top:3px; margin-bottom:3px; margin-left:0px; margin-right:10px; } .siteMap li,span { float:left; list-style-type:none; padding-left:5px; border-width:0px;} .siteMap span { font-weight:bold; } .siteMap a,a.Visited { color:White; text-decoration:none; }

Conclusion

SitemapPath control gives you a really easy way to put together a navigation control based on the Sitemap APIs (and the Web.Sitemap file in my case). Creating a simple ASP.NET Razor helper is actually pretty easy since all the functionality needed is there in the base API’s and although it required some code (20 lines of code) now I feel like I have more control over my markup, can style it in anyway I want using CSS and have cleaner markup rendered.

I’m sure there are better ways to do this, but as I said, the goal of this first pass is to push my site soon with as little changes possible while keeping the same functionality first.

Get IIS bindings at runtime without being an Administrator

CarlosAg — Sat, 22 Jan 2011 01:45:00 GMT

Today there was a question in StackOverflow asking whether it was possible to read the IIS binding information such as Port and Protocols from the ASP.NET application itself to try to handle redirects from HTTP to HTTPS in a way that was reliable without worrying about using different ports than 80/443.

Turns out this is possible in the context of the IIS worker process by using Microsoft.Web.Administration.

The following function will take care of that by reading the Worker Process isolated configuration file and find the HTTP based bindings.

    private static IEnumerable<KeyValuePair<string, string>> GetBindings(HttpContext context) {
        // Get the Site name
        string siteName = System.Web.Hosting.HostingEnvironment.SiteName;

        // Get the sites section from the AppPool.config
        Microsoft.Web.Administration.ConfigurationSection sitesSection =
            Microsoft.Web.Administration.WebConfigurationManager.GetSection(null, null, "system.applicationHost/sites");

        foreach (Microsoft.Web.Administration.ConfigurationElement site in sitesSection.GetCollection()) {
            // Find the right Site
            if (String.Equals((string)site["name"], siteName, StringComparison.OrdinalIgnoreCase)) {

                // For each binding see if they are http based and return the port and protocol
                foreach (Microsoft.Web.Administration.ConfigurationElement binding in site.GetCollection("bindings")) {
                    string protocol = (string)binding["protocol"];
                    string bindingInfo = (string)binding["bindingInformation"];

                    if (protocol.StartsWith("http", StringComparison.OrdinalIgnoreCase)) {
                        string[] parts = bindingInfo.Split(':');
                        if (parts.Length == 3) {
                            string port = parts[1];
                            yield return new KeyValuePair<string, string>(protocol, port);
                        }
                    }
                }
            }
        }
    }

If you want to try it, you could use the following page, just save it as test.aspx and add the function above, the result is a simple table that shows the protocol and port to be used:

<%@ Page Language="C#" %>
<%@ Import Namespace="System.Collections.Generic" %>
<script runat="server">
    protected void Page_Load(object sender, EventArgs e) {
        Response.Write("<table border='1'>");
        foreach (KeyValuePair<string, string> binding in GetBindings(this.Context)) {
            Response.Write("<tr><td>");
            Response.Write(binding.Key);
            Response.Write("</td><td>");
            Response.Write(binding.Value);
            Response.Write("</td></tr>");
        }
        Response.Write("</table>");
    }
</script>

Also, you will need to add Microsoft.Web.Administration to your compilation assemblies inside the web.config for it to work:

<?xml version="1.0"?>
<configuration>
  <system.web>
    <compilation debug="true">
      <assemblies>
        <add assembly="Microsoft.Web.Administration, Version=7.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35"/>
      </assemblies>
    </compilation>
  </system.web>
</configuration>

Managing ASP.NET Configuration Settings using IIS Manager

CarlosAg — Tue, 02 Nov 2010 17:08:00 GMT

Today somebody asked a question about how to manage some ASP.NET configuration settings such as changing the trust level of the application and adding a few application settings and changing compilation settings to debug. I thought it would be trivial to search the web for an article or something that would show the features we added in IIS 7.0 to manage those, but to my surprise I was not able to find anything that would clearly show it, so I decided to write this pretty quickly for anyone that is not aware.

With the release of IIS 7.0 (included in Windows Vista and Windows Server 2008), and of course included in IIS 7.5 (Windows 7 and Windows Server 2008 R2) we added a set of features for managing some of the configuration of common ASP.NET features inside the same IIS Manager. Those features include:

.NET Authorization Rules¹ – To manage the authorization rules for ASP.NET, this is particularly useful when using Classic Mode. This UI basically is to manage the system.web/authorization section in a graphical way.
.NET Compilation – This exposes the settings used by the ASP.NET compilation engine, such as list of assemblies, Debug settings, VB settings (Option Strict, Option Explicit), Temp directory, etc. This UI saves all the settings in the system.web/compilation section.
.NET Error Pages¹ – Allows you to manage the ASP.NET custom errors, exposing the system.web/customErrors.
.NET Globalization – Allows you to manage the globalization settings such as file encoding, ui culture, etc. This modifies the section system.web/globalization
.NET Providers ² – Allows you to manage the different provider configuration for the ASP.NET providers, such as Roles, Membership and Profile. (system.web/membership, system.web/profile, system.web/roleManager, etc).
.NET Users, .NET Roles and .NET Profile ² – Configure options that track settings for ASP.NET applications. All these features uses the ASP.NET runtime configuration to allow you to manage their settings, such as adding users, roles and profile settings. (What this post is about). These does not modify configuration but instead they use the Provider configured (such as SqlMembershipProvider, SqlRoleProvider, WindowsTokenRoleProvider, etc)
.NET Trust Levels, allows you to configure the security trust level policy for the application. Modifies system.web/trust section.
Application Settings – Allows you to manage the name/value pair stored in the .NET appSettings section.
Connection Strings – Configures the database connection strings that can be used by ASP.NET applications. Manages the connectionStrings section.
Machine Key – Allows you to modify the machine key and other related settings stored in system.web/machineKey section.
Pages and Controls – Allows you to modify settings from the system.web/pages section, such as Base class, Namespaces, and Controls registered.
Session State – Allows you to configure the session state settings such as connection string, cookie configuration and other configuration included in system.web/sessionState.
SMTP E-mail – Configure the SMTP settings such as Server, Delivery mode, or Pickup directory, included in system.net/mailSettings/smtp section.

¹ – These features are included in Windows 7 and Windows Server 2008 R2, but can be installed for Windows Vista and Windows Server 2008 when downloading the Administration Pack for IIS7.

² – Note, these features require hosting the ASP.NET runtime and due to technical limitations only application pools configure to run using .NET Version 2.0 will show these features. This means that if you configure your application pool to run .NET 4.0 (in IIS 7.0 and IIS 7.5) you will not see those features. As a workaround you could temporarily change the application pool to run in 2.0, make your changes and switch it back to 4.0 (of course, not recommended for production environments).

These features are not meant to expose all the settings included in ASP.NET, and they only include configuration settings up to .NET 2.0. I should also add that IIS includes a generic configuration editor that allows you to manage a lot more configuration settings from ASP.NET, IIS, and more, in the image below you can see a lot more sections like webParts, trace, siteMap, and others:

The best thing is that you can apply the changes immediately or you can also make changes and just generate the code to automate them later using code, command line or scripts them using Javascript, Managed code, or AppCmd.exe.

Using the SEO Toolkit to generate a Sitemap of a remote Web Site

CarlosAg — Tue, 12 Oct 2010 18:09:00 GMT

The SEO Toolkit includes a set of features (like Robots Editor and Sitemap Editor) that only work when you are working with a local copy of your Web Site. The reason behind it is that we have to understand where we need to save the files that we need to generate (like Robots.txt and Sitemap XML files) without having to ask for physical paths as well as to verify that the functionality is added correctly such as only allowing Robots.txt in the root of a site, etc. Unfortunately this means that if you have a remote server that you cannot have a running local copy then you cannot use those features. (Note that you can still use Site Analysis tool since that will crawl your Web Site regardless of platform or framework and will store the report locally just fine.)

The Good News

The good news is that you can technically trick the SEO Toolkit into thinking you have a working copy locally and allow you to generate the Sitemap or Robots.txt file without too much hassle (“too much” being the key).

For this sample, lets assume we want to create a Sitemap from a remote Web site, in this case I will use my own Web site (http://www.carlosag.net/ , but you can specify your own Web site, below are the steps you need to follow to enable those features for any remote Web site (even if it is running in other versions of IIS or any other Web Server).

Create a Fake Site

Open IIS Manager (Start Menu->InetMgr.exe)
Expand the Tree until you can see the “Sites” node.
Right-click the “Sites” node and select “Add Web Site…”

Specify a Name (in my case I’ll use MySite)
Click “Select” to choose the DefaultAppPool from the Application Pool list. This will avoid creating an additional AppPool that will never run.
Specify a Physical Path where you will want the Robots and Sitemap files to be saved. I recommend creating just a temporary directory that clearly states this is a fake site. So I will choose c:\FakeSite\ for that.
Important. Set the Host name so that it matches your Web Site, for example in my case www.carlosag.net.
Uncheck the “Start Web site immediately”, since we do not need this to run.
Click OK

This is how my Create site dialog looks like:

Use the Sitemap Editor

Since we have a site that SEO Toolkit thinks it is locally now you should be able to use the features as usual.

Select the new site created above in the Tree
Double-click the Search Engine Optimization icon in the Home page
Click the link “Create a new Sitemap”
Specify a name, in my case Sitemap.xml
Since this is a remote site, you will see that the physical location option shows an empty list, so change the “URL structure” to will use the “<Run new Site Analysis>..” or if you already have one you can choose that.

If creating a new one, just specify a name and click OK (I will use MySite). At this point the SEO Toolkit starts crawling the Remote site to discover links and URLs, when it is done it will present you the virtual namespace structure so you can work with.

After the crawling is done, you can now check any files you want to include in your Sitemap and leverage the Server response to define the changed date and all the features as if the content was local. and Click OK

This is the way the dialog looks when discovered my remote Web site URLs:

You will find your Sitemap.xml generated in the physical directory specified when creating the site (in my case c:\FakeSite\Sitemap.xml").

Use the Robots Editor

Just as with the Sitemap Editor, once you prepare a fake site for the remote server, you should be able to use the Robots Editor and leverage the same Site analysis output to build your Robots.txt file.

Summary

In this blog I show how you can use the Sitemap and Robots Editor included in the SEO Toolkit when working with remote Web sites that might be running in different platforms or different versions of IIS.

Free SEO Analysis using IIS SEO Toolkit

CarlosAg — Fri, 14 May 2010 00:28:00 GMT

In my spare time I’ve been thinking about new ideas for the SEO Toolkit, and it occurred to me that rather than continuing trying to figure out more reports and better diagnostics against some random fake sites, that it could be interesting to ask openly for anyone that is wanting a free SEO analysis report of your site and test drive some of it against real sites.

So what is in it for you, I will analyze your site to look for common SEO errors, I will create a digest of actions to do and other things (like generating a diagram of your site, layer information, etc), and will deliver it to you in a form of an email. If you agree I will post some of the results (hiding identification information like site, url, etc, so that it is made anonymously if needed).
and what is in it for me, well I will crawl your Web Site (once or twice at most, with a limit set to a few hundred pages) using the SEO Toolkit to test drive some ideas and reporting stuff that I’m starting to build and to continue investigating common patterns and errors.

So if you want in, just post me your URL in the comments of this blog (make sure you are reading this blog from a URL inside http://blogs.msdn.com/carlosag/ , otherwise you might be posting comments in some syndicating site.), I will only allow the first few sites (if successful I will start another batch in the future) and I will be doing one by one in the following days. Make sure to include a way to contact you whether using the MSDN user infrastructure or include an email so that I can contact you with the results.

Alternatively I will take also URLs using Twitter at http://twitter.com/CarlosAguilarM so hurry up and let me know if you want me to look at your site.

IIS SEO Toolkit and W3C Validation Service

CarlosAg — Mon, 10 May 2010 21:21:00 GMT

One thing that I’ve been asked several times about the SEO Toolkit is if it does a full standards validation on the markup and content that is processed, and if not, to add support for more comprehensive standards validation, in particular XHTML and HTML 4.01. Currently the markup validation performed by the SEO Toolkit is really simple, its main goal is to make sure that the markup is correctly organized, for example that things like <b><i>Test</b></i> are not found in the markup, the primary reason is to make sure that basic blocks of markup are generally "easy" to parse by Search Engines and that the semantics will not be terribly broken if a link, text or style is not correctly closed (since all of them would affect SEO).

So the first thing I would say is that we have heard the feedback and are looking at what we could possibly add in future versions, however why wait, right?

One thing that many people do not realize is that the SEO Toolkit can be extended to add new violations, new metadata and new rules to the analysis process and as such during a demo I gave a few weeks ago I decided to write a sample on how to consume the online W3C Markup Validation Service from the SEO Toolkit.

Download

You can download the SEOW3Validator including the source code at http://www.carlosag.net/downloads/SEOW3Validator.zip.

How to install it

To run it you just need to:

Unzip the contents in a folder.
Install the SEOW3Validator.dll assembly in the GAC:

Open a Windows Explorer window and navigate to c:\Windows\assembly
Drag and Drop the SEOW3Validator.dll to the c:\Windows\assembly explorer window.
Alternatively you can just run gacutil.exe /i SEOW3Validator.dll, usually located at C:\Program Files\Microsoft SDKs\Windows\v6.0A\bin or v7A.
If you have problems with this, you could try just copying the assembly to the GAC (copy SEOW3Validator.dll c:\Windows\assembly\GAC_MSIL\SEOW3Validator\1.0.0.0__995ee9b8fa017847\SEOW3Validator.dll)

Register the moduleProvider in Administration.config: In an elevated prompt open C:\Windows\System32\Inetsrv\config\Administration.config and add the following line right inside the <moduleProviders> right before closing the </moduleProviders>:

You should be able to now run the SEO Toolkit just as before but now you will find new violations, for example in my site I get the ones below. Notice that there are a new set of violations like W3 Validator – 68, etc, and all of them belong to the W3C category. (I would have liked to have better names, but the way the W3 API works is not really friendly for making this any better).

And when double clicking any of those results you get the details as reported by the W3 Validation Service:

The Code

The code is actually pretty simple, the main class is called SEOW3ValidatorExtension that derives from CrawlerModule and overrides the Process method to call the W3C Validation service sending the actual markup in the request, this means that it does not matter if your site is an Intranet or in the Internet, it will work; and for every warning and error that is returned by the Validator it will add a new violation to the SEO report.

The code looks like this:

    W3Validator validator = new W3Validator();
    W3ValidatorResults results = validator.Validate(context.UrlInfo.FileName,
        context.UrlInfo.ContentTypeNormalized,
        context.UrlInfo.Response);

    foreach (W3ValidatorWarning warning in results.Warnings) {
        context.UrlInfo.AddViolation(CreateWarning(warning));
    }

    foreach (W3ValidatorError error in results.Errors) {
        context.UrlInfo.AddViolation(CreateError(error));
    }

I created a helper class W3Validator that basically encapsulates the consumption of the W3C Validation Service, the code is far from what I would like it to be however there are some "interesting" decisions on the way the API is exposed, I would have probably designed the service differently and not return the results formatted in HTML when this is actually an API/WebService that can be presented somewhere else than a browser. So a lot of the code is to just re-format the results to look "decent", but to be honest I did not want to spend too much time on it so everything was put together quite quickly. Also, if you look at the names I used for violations, I did not want to hard-code specific Message IDs and since the Error Message was different for all of them even within the same Message ID, it was not easy to provide better messages. Anyway, overall it is pretty usable and should be a good way to do W3 Validation.

Note that one of the cool things you get for free is that since these are stored as violations, you can then re-run the report and use the Compare Report feature to see the progress while fixing them. Also, since they are stored as part of the report you will not need to keep running the validator over and over again but instead just open it and continue looking at them, as well as analyzing the data in the Reports and Queries, and be able to export them to Excel, etc.

Hopefully this will give you a good example on some of the interesting things you can achieve with the SEO Toolkit and its extensibility.

IIS SEO Toolkit Available in 10 Languages

CarlosAg — Fri, 09 Apr 2010 06:00:00 GMT

A couple of months ago I blogged about the release of the v1.0.1 of the IIS Search Engine Optimization Toolkit. In March we released the localized versions of the SEO Toolkit so now it is available in 10 languages: English, Japanese, French, Russian, Korean, German, Spanish, Chinese Simplified, Italian and Chinese Traditional.

Here are all the direct links to download it.

Name	Language	Download URL
IIS SEO Toolkit 32bit	english	http://download.microsoft.com/download/A/C/A/ACA8D740-A59D-4D25-A2D5-1DCFD1D9A01F/IISSEO_x86.msi
IIS SEO Toolkit 64bit	english	http://download.microsoft.com/download/A/C/A/ACA8D740-A59D-4D25-A2D5-1DCFD1D9A01F/IISSEO_amd64.msi
IIS SEO Toolkit 32bit	ja-jp	http://download.microsoft.com/download/3/6/1/36179752-3497-4C2C-B2C5-9B4FA14EAC3A/IISSEO_x86_ja-JP.msi
IIS SEO Toolkit 64bit	ja-jp	http://download.microsoft.com/download/3/6/1/36179752-3497-4C2C-B2C5-9B4FA14EAC3A/IISSEO_amd64_ja-JP.msi
IIS SEO Toolkit 32bit	fr-fr	http://download.microsoft.com/download/D/C/5/DC576407-7273-412C-9AC8-AE78E4CFE017/IISSEO_x86_fr-FR.msi
IIS SEO Toolkit 64bit	fr-fr	http://download.microsoft.com/download/D/C/5/DC576407-7273-412C-9AC8-AE78E4CFE017/IISSEO_amd64_fr-FR.msi
IIS SEO Toolkit 32bit	ru-ru	http://download.microsoft.com/download/8/6/A/86A0BCE1-419F-4550-968E-A8E5A8467B32/IISSEO_x86_ru-RU.msi
IIS SEO Toolkit 64bit	ru-ru	http://download.microsoft.com/download/8/6/A/86A0BCE1-419F-4550-968E-A8E5A8467B32/IISSEO_amd64_ru-RU.msi
IIS SEO Toolkit 32bit	ko-kr	http://download.microsoft.com/download/F/8/6/F8654213-40C6-4706-9128-536A6A4BC570/IISSEO_x86_ko-KR.msi
IIS SEO Toolkit 64bit	ko-kr	http://download.microsoft.com/download/F/8/6/F8654213-40C6-4706-9128-536A6A4BC570/IISSEO_amd64_ko-KR.msi
IIS SEO Toolkit 32bit	de-de	http://download.microsoft.com/download/1/8/1/1813318E-6358-4BDC-B148-C1826A74994D/IISSEO_x86_de-DE.msi
IIS SEO Toolkit 64bit	de-de	http://download.microsoft.com/download/1/8/1/1813318E-6358-4BDC-B148-C1826A74994D/IISSEO_amd64_de-DE.msi
IIS SEO Toolkit 32bit	es-es	http://download.microsoft.com/download/1/6/6/166C82C0-4B72-4282-9A86-47C85CE7E20C/IISSEO_x86_es-ES.msi
IIS SEO Toolkit 64bit	es-es	http://download.microsoft.com/download/1/6/6/166C82C0-4B72-4282-9A86-47C85CE7E20C/IISSEO_amd64_es-ES.msi
IIS SEO Toolkit 32bit	zh-cn	http://download.microsoft.com/download/D/6/C/D6C6DE59-2EE8-4DD3-9E30-739A5BE42F3C/IISSEO_x86_zh-CN.msi
IIS SEO Toolkit 64bit	zh-cn	http://download.microsoft.com/download/D/6/C/D6C6DE59-2EE8-4DD3-9E30-739A5BE42F3C/IISSEO_amd64_zh-CN.msi
IIS SEO Toolkit 32bit	it-it	http://download.microsoft.com/download/6/1/F/61FC149C-A950-40F4-9795-F3D4F2115721/IISSEO_x86_it-IT.msi
IIS SEO Toolkit 64bit	it-it	http://download.microsoft.com/download/6/1/F/61FC149C-A950-40F4-9795-F3D4F2115721/IISSEO_amd64_it-IT.msi
IIS SEO Toolkit 32bit	zh-tw	http://download.microsoft.com/download/6/1/F/61FC149C-A950-40F4-9795-F3D4F2115721/IISSEO_x86_zh-TW.msi
IIS SEO Toolkit 64bit	zh-tw	http://download.microsoft.com/download/6/4/0/64067386-3BF6-493E-B7DB-4423839C316B/IISSEO_amd64_zh-TW.msi

Here is a screenshot of how the SEO Toolkit running in Spanish.

If you want to read the download files in the Microsoft Download Center you can click the links below:

IIS Search Engine Optimization Toolkit - 32bit ja-jp
IIS Search Engine Optimization Toolkit - 32bit fr-fr
IIS Search Engine Optimization Toolkit - 32bit ru-ru
IIS Search Engine Optimization Toolkit - 32bit ko-kr
IIS Search Engine Optimization Toolkit - 32bit de-de
IIS Search Engine Optimization Toolkit - 32bit es-es
IIS Search Engine Optimization Toolkit - 32bit zh-cn
IIS Search Engine Optimization Toolkit - 32bit it-it
IIS Search Engine Optimization Toolkit - 32bit zh-tw

To learn more about the SEO Toolkit you can visit:

http://blogs.msdn.com/carlosag/archive/tags/SEO/default.aspx

http://www.iis.net/expand/SEOToolkit

And for any help or provide us feedback you can do that in the IIS.NET SEO Forum.

Setting up a Reverse Proxy using IIS, URL Rewrite and ARR

CarlosAg — Fri, 02 Apr 2010 02:10:00 GMT

Today there was a question in the IIS.net Forums asking how to expose two different Internet sites from another site making them look like if they were subdirectories in the main site.

So for example the goal was to have a site: www.site.com expose a www.site.com/company1 and a www.site.com/company2 and have the content from “www.company1.com” served for the first one and “www.company2.com” served in the second one. Furthermore we would like to have the responses cached in the server for performance reasons. The following image shows a simple diagram of this:

This sounds easy since its just about routing or proxying every single request to the correct servers, right? Wrong!!! If it only it was that easy. Turns out the most challenging thing is that in this case we are modifying the structure of the underlying URLs and the original layout in the servers which makes relative paths break and of course images, Stylesheets (css), javascripts and other resources are not shown correctly.

To try to clarify this, imagine that a user requests using his browser the page at http://www.site.com/company1/default.aspx, and so based on the specification above the request is proxied/routed to http://www.company1.com/default.aspx on the server-side. So far so good, however, imagine that the markup returned by this HTML turns out to have an image tag like “<img src=/some-image.png />”, well the problem is that now the browser will resolve that relative path using the base path on the original request he made which was http://www.site.com/company1/default.aspx resulting in a request for the image at http://www.site.com/some-image.png instead of the right “company1” folder that would be http://www.site.com/company1/some-image.png .

Do you see it? Basically the problem is that any relative path or for that matter absolute paths as well need to be translated to the new URL structure imposed by the original goal.

So how do we do it then?

Enter URL Rewrite 2.0 and Application Request Routing

URL Rewrite 2.0 includes the ability to rewrite the content of a response as it is getting served back to the client which will allow us to rewrite those links without having to touch the actual application.

Software Required:

IIS 7.0 or IIS 7.5 (included in Windows Vista, Windows 2008, Windows 7 or Windows 2008 R2).
Install URL Rewrite 2.0
Install Application Request Routing

Steps

The first thing you need to do is enable Proxy support in ARR.

To do that just launch IIS Manager and click the server node in the tree view.
Double click the “Application Request Routing Cache” icon
Select the “Server Proxy Settings…” task in the Actions panel
And Make sure that “Enable Proxy” checkbox is marked. What this will do is allow any request in the server that is rewritten to a server that is not the local machine will be routed to the right place automatically without any further configuration.

Configure URL Rewrite to route the right folders and their requests to the right site. But rather than bothering you with UI steps I will show you the configuration and then explain step by step what each piece is doing.
Note that for this post I will only take care of Company1, but you can imagine the same steps apply for Company2, and to test this you can just save the configuration file below as web.config and save it in your inetpub\wwwroot\ or in any other site root and you can test it.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <rule name="Route the requests for Company1" stopProcessing="true">
                    <match url="^company1/(.*)" />
                    <conditions>
                        <add input="{CACHE_URL}" pattern="^(https?)://" />
                    </conditions>
                    <action type="Rewrite" url="{C:1}://www.company1.com/{R:1}" />
                    <serverVariables>
                        <set name="HTTP_ACCEPT_ENCODING" value="" />
                    </serverVariables>
                </rule>
            </rules>
            <outboundRules>
                <rule name="ReverseProxyOutboundRule1" preCondition="ResponseIsHtml1">
                    <match filterByTags="A, Area, Base, Form, Frame, Head, IFrame, Img, Input, Link, Script" pattern="^http(s)?://www.company1.com/(.*)" />
                    <action type="Rewrite" value="/company1/{R:2}" />
                </rule>
                <rule name="RewriteRelativePaths" preCondition="ResponseIsHtml1">
                    <match filterByTags="A, Area, Base, Form, Frame, Head, IFrame, Img, Input, Link, Script" pattern="^/(.*)" negate="false" />
                    <action type="Rewrite" value="/company1/{R:1}" />
                </rule>
                <preConditions>
                    <preCondition name="ResponseIsHtml1">
                        <add input="{RESPONSE_CONTENT_TYPE}" pattern="^text/html" />
                    </preCondition>
                </preConditions>
            </outboundRules>
        </rewrite>
    </system.webServer>
</configuration>

Setup the Routing

The first rule is an inbound rewrite rule that basically captures all the requests to the root folder /company1/*, so if using Default Web Site, anything going to http://localhost/company1/* will be matched by this rule and it will rewrite it to www.company1.com respecting the HTTP vs HTTPS traffic.

One thing to highlight which is what took me a bit of time is the “serverVariables” entry in that rule that basically is overwriting the Accept-Encoding header, the reason I do this is because if you do not remove that header then the response will likely be compressed (Gzip or deflate) and Output Rewriting is not supported on that case, and you will end up with an error message like:

HTTP Error 500.52 - URL Rewrite Module Error.
Outbound rewrite rules cannot be applied when the content of the HTTP response is encoded ("gzip").

Also note that to be able to use this feature for security reasons you need to explicitly enable this by allowing the server variable. See enabling server variables here.

Outbound Rewriting to fix the Links

The last two rules just rewrite the links and scripts and other resources so that the URLs are translated to the right structure. The first one rewrites absolute paths, and the last one rewrites the relative paths. Note that if you use relative paths using “..” this will not work, but you can easily fix the rule above, I was too lazy to do that and since I never use those when I create a site it works for me :)

Setting up Caching for ARR

A huge added value of using ARR is that now we can with a couple of clicks enable disk caching so that the requests are cached locally in the www.site.com, so that not every single request ends up paying the price to go to the backend servers.

To do that just launch IIS Manager and click the server node in the tree view.
Double click the “Application Request Routing Cache” icon
Select the “Add Drive…” task in the Actions panel.
Specify a directory where you want to keep your cache. Note that this can be any subfolder in your system.
Make sure that “Enable Disk Cache” checkbox is marked in the Server Proxy Settings mentioned above.

As easy as that now you will see caching working and your site will act as a container of other servers in the internet. Pretty cool hah! :)

So in this post we saw how with literally few lines of XML, URL Rewrite and ARR we were able to enable a proxy/routing scenario with the ability to rewrite links and furthermore with caching support.

SEO made easy with IIS URL Rewrite 2.0 SEO templates

CarlosAg — Thu, 01 Apr 2010 00:20:00 GMT

A few weeks ago my team released the version 2.0 of the URL Rewrite for IIS. URL Rewrite is probably the most powerful Rewrite engine for Web Applications. It gives you many features including Inbound Rewriting (ie. Rewrite the URL, Redirect to another URL, Abort Requests, use of Maps, and more), and in Version 2.0 it also includes Outbound Rewriting so that you can rewrite URLs or any markup as the content is being sent back even if its generated using PHP, ASP.NET or any other technology.

It also includes a very powerful User Interface that allows you to test your regular expressions and even better it includes a set of templates for common types of Rules. Some of those rules are incredibly valuable for SEO (Search Engine Optimization) purposes. The SEO rules are:

Enforce Lowercase URLs. It will make sure that every URL is used with only lower case and if not it will redirect with a 301 to the lower-case version.
Enforce a Canonical Domain Name. It will help you specify what domain name you want to use for your site and it will redirect the traffic to the right host name.
Append or Remove the Trailing Slash. It will make sure your request either include or not include the trailing slash depending on your preference.

For more information on the SEO Templates look at: http://learn.iis.net/page.aspx/806/seo-rule-templates/

What is really cool is that you can use the SEO Toolkit to run it against your application and you probably will get some violations around lower-case, or canonical domains, etc. And after seeing those you can use URL Rewrite 2.0 to fix them with one click.

I have personally used it in my Web site, try the following three URLs and all of them will be redirected to the canonical form (http://www.carlosag.net/Tools/CodeTranslator/) and you will see URL Rewrite in action:

Note that at the end those templates just translate to web.config settings that become part of your application that can be XCOPY with it. This works with ASP.NET, PHP, or any other server technology including static files. Below is the output of the Canonical Host Name rule which I use on my Web site’s web.config.

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.webServer>
        <rewrite>
            <rules>
                <rule name="CanonicalHostNameRule1">
                    <match url="(.*)" />
                    <conditions>
                        <add input="{HTTP_HOST}" pattern="^www\.carlosag\.net$" negate="true" />
                    </conditions>
                    <action type="Redirect" url="http://www.carlosag.net/{R:1}" />
                </rule>
            </rules>
        </rewrite>
    </system.webServer>
</configuration>

There are many more features that I could talk, but for now this was just a quick SEO related post.

Analyze your IIS Log Files - Favorite Log Parser Queries

CarlosAg — Thu, 25 Mar 2010 22:39:00 GMT

The other day I was asked if I knew about a tool that would allow users to easily analyze the IIS Log Files, to process and look for specific data that could easily be automated. My recommendation was that if they were comfortable with using a SQL-like language that they should use Log Parser. Log Parser is a very powerful tool that provides a generic SQL-like language on top of many types of data like IIS Logs, Event Viewer entries, XML files, CSV files, File System and others; and it allows you to export the result of the queries to many output formats such as CSV (Comma-Separated Values, etc), XML, SQL Server, Charts and others; and it works well with IIS 5, 6, 7 and 7.5.

To use it you just need to install it and use the LogParser.exe that is found in its installation directory (on my x64 machine it is located at: C:\Program Files (x86)\Log Parser 2.2).

I also thought on sharing some of my favorite queries. To run them, just execute LogParser.exe and make sure to specify that the input is an IIS Log file (-i:W3C) and for ease of use in this case we will export to a CSV file that can be then opened in Excel (-o:CSV) for further analysis:

LogParser.exe -i:W3C "Query-From-The-Table-Below" -o:CSV

Purpose

Query

Sample Output

Number of Hits per Client IP, including a Reverse DNS lookup (SLOW)

SELECT c-ip As Machine,
REVERSEDNS(c-ip) As Name,
COUNT(*) As Hits
FROM c:\inetpub\logs\LogFiles\W3SVC1\*
GROUP BY Machine ORDER BY Hits DESC

Machine	Name	Hits
::1	CARLOSAGDEV	57
127.0.0.1	MACHINE1	28
127.X.X.X	MACHINE2	1

Top 25 File Types

SELECT TOP 25
EXTRACT_EXTENSION(cs-uri-stem) As Extension,
COUNT(*) As Hits
FROM c:\inetpub\logs\LogFiles\W3SVC1\*
GROUP BY Extension
ORDER BY Hits DESC

Extension	Hits
gif	52127
bmp	20377
axd	10321
txt	460
htm	362

Top 25 URLs

SELECT TOP 25
cs-uri-stem as Url,
COUNT(*) As Hits
FROM c:\inetpub\logs\LogFiles\W3SVC1\*
GROUP BY cs-uri-stem
ORDER By Hits DESC

Url	Hits
/WebResource.axd	10318
/favicon.ico	8523
/Tools/CodeTranslator/Translate.ashx	6519
/App_Themes/Silver/carlosag.css	5898
/images/arrow.gif	5720

Number of hits per hour for the month of March

SELECT
QUANTIZE(TO_LOCALTIME(TO_TIMESTAMP(date, time)), 3600) AS Hour,
COUNT(*) AS Hits
FROM c:\inetpub\logs\LogFiles\W3SVC1\*
WHERE date>'2010-03-01' and date<'2010-04-01'
Group By Hour

Hour		Hits
3/3/2010	10:00:00	33
3/3/2010	11:00:00	5
3/3/2010	12:00:00	3

Number of hits per Method (GET, POST, etc)

SELECT
cs-method As Method,
COUNT(*) As Hits
FROM c:\inetpub\logs\LogFiles\W3SVC1\*
GROUP BY Method

Method	Hits
GET	133566
POST	10901
HEAD	568
OPTIONS	11
PROPFIND	18

Number of requests made by user

SELECT TOP 25
cs-username As User,
COUNT(*) as Hits
FROM c:\inetpub\logs\LogFiles\W3SVC1\*
WHERE User Is Not Null
GROUP BY User

User	Count
Administrator	566
Guest	1

Extract Values from Query String (d and t) and use them for Aggregation

SELECT TOP 25
    EXTRACT_VALUE(cs-uri-query,'d') as Query_D,
    EXTRACT_VALUE(cs-uri-query,'t') as Query_T,
    COUNT(*) As Hits
FROM c:\inetpub\logs\LogFiles\W3SVC1\*
WHERE Query_D IS NOT NULL
GROUP BY Query_D, Query_T
ORDER By Hits DESC

Query_D	Query_T	Hits
Value in Query1	Value in T1	1556
Value in Query2	Value in T2	938
Value in Query3	Value in T3	877
Value in Query4	Value in T4	768

Find the Slowest 25 URLs (in average) in the site

SELECT TOP 25
    cs-uri-stem as URL,
    MAX(time-taken) As Max,
    MIN(time-taken) As Min,
    Avg(time-taken) As Average
FROM c:\inetpub\logs\LogFiles\W3SVC1\*
GROUP BY URL
ORDER By Average DESC

URL	Max	Min	Average
/Test/Default.aspx	23215	23215	23215
/WebSite/Default.aspx	5757	2752	4178
/Remote2008.jpg	3510	3510	3510
/wordpress/	6541	2	3271
/RemoteVista.jpg	3314	2	1658

List the count of each Status and Substatus code

SELECT TOP 25
    STRCAT(TO_STRING(sc-status),
    STRCAT('.', TO_STRING(sc-substatus))) As Status,
    COUNT(*) AS Hits
FROM c:\inetpub\logs\LogFiles\W3SVC1\*
GROUP BY Status
ORDER BY Status ASC

Status	Hits
200	144
304	38
400	9
403.14	10
404	64
404.3	2
500.19	23

List all the requests by user agent

SELECT
cs(User-Agent) As UserAgent,
COUNT(*) as Hits
FROM c:\inetpub\logs\LogFiles\W3SVC1\*
GROUP BY UserAgent
ORDER BY Hits DESC

UserAgent	Hits
iisbot/1.0+(+http://www.iis.net/iisbot.html)	104
Mozilla/4.0+(compatible;+MSIE+8.0;…	77
Microsoft-WebDAV-MiniRedir/6.1.7600	23
DavClnt	1

List all the Win32 Error codes that have been logged

SELECT
    sc-win32-status As Win32-Status,
    WIN32_ERROR_DESCRIPTION(sc-win32-status) as Description,
    COUNT(*) AS Hits
FROM c:\inetpub\logs\LogFiles\W3SVC1\*
WHERE Win32-Status<>0
GROUP BY Win32-Status
ORDER BY Win32-Status ASC

Win32-Status	Description	Hits
2	The system cannot find the file specified.	64
13	The data is invalid.	9
50	The request is not supported.	2

A final note: any time you deal with Date and Time, remember to use the TO_LOCALTIME function to convert the log times to your local time, otherwise you will find it very confusing when your entries seem to be reported incorrectly.

If you need any help you can always visit the Log Parser Forums to find more information or ask specific questions.

Any other useful queries I missed?

IIS SEO Toolkit: Find warnings of HTTP content linked by pages using HTTPS

CarlosAg — Fri, 26 Feb 2010 09:07:00 GMT

Are you an developer/owner/publisher/etc of a site that uses HTTPS (SSL) for secure access? If you are, please continue to read.

Have you ever visited a Web site that is secured using SSL (Secure Sockets Layer) just to get an ugly Security Warning message like:

Do you want to view only the webpage content that was delivered securely?

This webpage contains content that will not be delivered using a secure HTTPS connection, which could compromise the security of the entire webpage.

How frustrating is this for you? Do you think that end-users know what is the right answer to the question above? Honestly, I think it actually even feels like the Yes/No buttons and the phrasing of the question would cause me to click the wrong option.

What this warning is basically trying to tell the user is that even though he/she navigated to a page that you thought was secured by using SSL, the page is consuming resources that are coming from an unsecured location, this could be scripts, style-sheets or other types of objects that could potentially pose a security risk since they could be tampered on the way or come from different locations.

As a site owner/developer/publisher/etc should always make sure that you are not going to expose your customers to such a bad experience, leaving them with an answer that they can’t possibly choose right. For one if they ‘choose Yes’ they will get an incomplete experience being broken images, broken scripts or something worse; otherwise they can ‘choose No’ which is even worse since that means you are actually teaching them to ignore this warnings which could indeed in some cases be real signs of security issues.

Bottom-line it should be imperative that any issue like this should be treated as a bug and fixed in the application if possible.

But the big question is how do you find these issues? Well the answer is very simple yet extremely time consuming, just navigate to every single page of your site using SSL and as you do that examine every single resource in the page (styles, objects, scripts, etc) and see if the URL is pointing to a non-HTTPS location.

Enter the IIS Search Engine Optimization (SEO) Toolkit.

The good news is that using the SEO Toolkit is extremely simple to find these issues.

To do that just start a new Analysis using the IIS SEO Toolkit using the HTTPS URL of your site, for example: https://www.example.com/
Once the analysis is over just select the option “Query->Open Query” and open the following XML file:

<?xml version="1.0" encoding="utf-8"?>
<query dataSource="links">
<filter>
    <expression field="LinkingUrl" operator="Begins" value="https://" />
    <expression field="LinkedUrl" operator="Begins" value="http://" />
    <expression field="LinkType" operator="NotEquals" value="Link" />
    <expression field="LinkType" operator="NotEquals" value="Rss" />
</filter>
<displayFields>
    <field name="LinkingUrl" />
    <field name="LinkedUrl" />
    <field name="LinkedStatus" />
    <field name="LinkType" />
</displayFields>
</query>

Just by doing that it will open a Query Window that will show all the links in your site that have such a problem. Note that the query simply looks for all the resources that are being linked by a URL that begins with HTTPS and that the target resource is using HTTP and that are not normal links (since they do not have that problem).
This is how my quick repro looks like. Note that it actually tells you the type of resource it is (an image and a style in this case). Additionally if you double click the row it will show you exactly the place in the markup where the problem occurs so you can easily fix it.

Summary

Using the IIS SEO Toolkit and it powerful Query Engine you can easily detect conditions on your site that otherwise would take an incredible amount of time and that would be prohibitively expensive to do constantly.

Announcing: IIS SEO Toolkit v1.0.1

CarlosAg — Mon, 01 Feb 2010 07:42:39 GMT

Last week we released a refresh for the IIS Search Engine Optimization (SEO) Toolkit v1.0. This version is a minor update that includes fixes for all the important bugs reported in the IIS.NET SEO Forum.

Some of the fixes included in this version are:

Pages sending the XHTML content type 'application/xhtml+xml' are not parsed correctly as HTML causing their links and violations to be empty.
Report Analysis fails if the META tags include certain characters.
<style> tag is not parsed correctly if it is empty causing Invalid Markup violations to be flagged incorrectly.
Memory is not released when the "Store Copies of analyzed web pages locally" button is unchecked.
HTML with leading empty lines and doctype fails to parse correctly causing their links and violations to be empty.
Internal Link criteria option of "host: <sitename> and subdomains (*.<sitename>)" fails to work as expected under certain configurations.
System.NullReferenceException when content attribute is misisng in Meta tag
Windows authentication does not work with servers configured with NTLM or Kerberos only challenge.
External META tags are stored in the report making it cumbersome to use the important ones.
Several localization related bugs.
DTD error when navigating to the Sitemap and Sitemap Index User Interface.
And others…

This release is compatible with v1.0 RTM and it will upgrade if already installed. So go ahead and install the new version using Web Platform Installed by clicking: http://go.microsoft.com/?linkid=9695987

Learn more about it at: http://www.iis.net/expand/SEOToolkit

IIS SEO Toolkit – Crawler Module Extensibility

CarlosAg — Mon, 23 Nov 2009 22:55:00 GMT

Sample SEO Toolkit CrawlerModule Extensibility

In this blog we are going to write an example on how to extend the SEO Toolkit functionality, so for that we are going to pretend our company has a large Web site that includes several images, and now we are interested in making sure all of them comply to a certain standard, lets say all of them should be smaller than 1024x768 pixels and that the quality of the images is no less than 16 bits per pixel. Additionally we would also like to be able to make custom queries that can later allow us to further analyze the contents of the images and filter based on directories and more.

For this we will extend the SEO Toolkit crawling process to perform the additional processing for images, we will be adding the following new capabilities:

Capture additional information from the Content. In this case we will capture information about the image, in particular we will extend the report to add a "Image Width", "Image Height" and a "Image Pixel Format".
Flag additional violations. In this example we will flag three new violations:
1. Image is too large. This violation will be flagged any time the content length of the image is larger than the "Maximum Download Size per URL" configured at the start of the analysis. It will also flag this violation if the resolution is larger than 1024x768.
2. Image pixel format is too small. This violation will be flagged if the image is 8 or 4 bits per pixel.
3. Image has a small resolution. This will be flagged if the image resolution per inch is less than 72dpi.

Enter CrawlerModule

A crawler module is a class that extends the crawling process in Site Analysis to provide custom functionality while processing each URL. By deriving from this class you can easily raise your own set of violations or add your own data and links to any URL.

public abstract class CrawlerModule : IDisposable
{
    // Methods
    public virtual void BeginAnalysis();
    public virtual void EndAnalysis(bool cancelled);
    public abstract void Process(CrawlerProcessContext context);

    // Properties
    protected WebCrawler Crawler { get; }
    protected CrawlerSettings Settings { get; }
}

It includes three main methods:

BeginAnalysis. This method is invoked once at the beginning of the crawling process and allows you to perform any initialization needed. Common tasks include registering custom properties in the Report that can be accessed through the Crawler property.
Process. This method is invoked for each URL once its contents has been downloaded. The context argument includes a property URLInfo that provides all the metadata extracted for the URL. It also includes a list of Violations and Links in the URL. Common tasks include augmenting the metadata of the URL whether using its contents or external systems, flagging new custom Violations, or discovering new links in the contents.
EndAnalysis. This method is invoked once at the end of the crawling process and allows you to do any final calculations on the report once all the URLs have been processed. Common tasks in this method include performing aggregations of data across all the URLs, or identifying violations that depend on all the data being available (such as finding duplicates).

Coding the Image Crawler Module

Create a Class Library in Visual Studio and add the code shown below.

Open Visual Studio and select the option File->New Project
In the New Project dialog select the Class Library project template and specify a name and a location such as "SampleCrawlerModule"
Using the Menu "Project->Add Reference", add a reference to the IIS SEO Toolkit client library (C:\Program Files\Reference Assemblies\Microsoft\IIS\Microsoft.Web.Management.SEO.Client.dll).
Since we are going to be registering this through the IIS Manager extensibility, add a reference to the IIS Manager extensibility DLL (c:\windows\system32\inetsrv\Microsoft.Web.Management.dll) using the "Project->Add Reference" menu.
Also, since we will be using the .NET Bitmap class you need to add a reference to "System.Drawing" using the "Project->Add Reference" menu.
Delete the auto-generated Class1.cs since we will not be using it.
Using the Menu "Project->Add New Item" Add a new class named "ImageExtension".

using System;
using System.Drawing;
using System.Drawing.Imaging;
using Microsoft.Web.Management.SEO.Crawler;

namespace SampleCrawlerModule {

    /// <summary>
    /// Extension to add validation and metadata to images while crawling
    /// </summary>
    internal class ImageExtension : CrawlerModule {
        private const string ImageWidthField = "iWidth";
        private const string ImageHeightField = "iHeight";
        private const string ImagePixelFormatField = "iPixFmt";

        public override void BeginAnalysis() {
            // Register the properties we want to augment at the begining of the analysis
            Crawler.Report.RegisterProperty(ImageWidthField, "Image Width", typeof(int));
            Crawler.Report.RegisterProperty(ImageHeightField, "Image Height", typeof(int));
            Crawler.Report.RegisterProperty(ImagePixelFormatField, "Image Pixel Format", typeof(string));
        }

        public override void Process(CrawlerProcessContext context) {
            // Make sure only process the Content Types we need to
            switch (context.UrlInfo.ContentTypeNormalized) {
                case "image/jpeg":
                case "image/png":
                case "image/gif":
                case "image/bmp":
                    // Process only known content types
                    break;
                default:
                    // Ignore any other
                    return;
            }

            //--------------------------------------------
            // If the content length of the image was larger than the max
            //   allowed to download, then flag a violation, and stop
            if (context.UrlInfo.ContentLength >
                Crawler.Settings.MaxContentLength) {
                Violations.AddImageTooLargeViolation(context,
                    "It is larger than the allowed download size");
                // Stop processing since we do not have all the content
                return;
            }

            // Load the image from the response into a bitmap
            using (Bitmap bitmap = new Bitmap(context.UrlInfo.ResponseStream)) {
                Size size = bitmap.Size;

                //--------------------------------------------
                // Augment the metadata by adding our fields
                context.UrlInfo.SetPropertyValue(ImageWidthField, size.Width);
                context.UrlInfo.SetPropertyValue(ImageHeightField, size.Height);
                context.UrlInfo.SetPropertyValue(ImagePixelFormatField, bitmap.PixelFormat.ToString());

                //--------------------------------------------
                // Additional Violations:
                //
                // If the size is outside our standards, then flag violation
                if (size.Width > 1024 &&
                    size.Height > 768) {
                    Violations.AddImageTooLargeViolation(context,
                        String.Format("The image size is: {0}x{1}",
                                      size.Width, size.Height));
                }

                // If the format is outside our standards, then flag violation
                switch (bitmap.PixelFormat) {
                    case PixelFormat.Format1bppIndexed:
                    case PixelFormat.Format4bppIndexed:
                    case PixelFormat.Format8bppIndexed:
                        Violations.AddImagePixelFormatSmall(context);
                        break;
                }

                if (bitmap.VerticalResolution <= 72 ||
                    bitmap.HorizontalResolution <= 72) {
                    Violations.AddImageResolutionSmall(context,
                        bitmap.HorizontalResolution + "x" + bitmap.VerticalResolution);
                }
            }
        }

        /// <summary>
        /// Helper class to hold the violations
        /// </summary>
        private static class Violations {

            private static readonly ViolationInfo ImageTooLarge =
                new ViolationInfo("ImageTooLarge",
                                  ViolationLevel.Warning,
                                  "Image is too large.",
                                  "The Image is too large: {details}.",
                                  "Make sure that the image content is required.",
                                  "Images");

            private static readonly ViolationInfo ImagePixelFormatSmall =
                new ViolationInfo("ImagePixelFormatSmall",
                                  ViolationLevel.Warning,
                                  "Image pixel format is too small.",
                                  "The Image pixel format is too small",
                                  "Make sure that the quality of the image is good.",
                                  "Images");

            private static readonly ViolationInfo ImageResolutionSmall =
                new ViolationInfo("ImageResolutionSmall",
                                  ViolationLevel.Warning,
                                  "Image resolution is small.",
                                  "The Image resolution is too small: ({res})",
                                  "Make sure that the image quality is good.",
                                  "Images");

            internal static void AddImageTooLargeViolation(CrawlerProcessContext context, string details) {
                context.Violations.Add(new Violation(ImageTooLarge,
                        0, "details", details));
            }

            internal static void AddImagePixelFormatSmall(CrawlerProcessContext context) {
                context.Violations.Add(new Violation(ImagePixelFormatSmall, 0));
            }

            internal static void AddImageResolutionSmall(CrawlerProcessContext context, string resolution) {
                context.Violations.Add(new Violation(ImageResolutionSmall,
                        0, "res", resolution));
            }
        }
    }
}

As you can see in the BeginAnalysis the module registers three new properties with the Report using the Crawler property. This is only required if you want to provide either a custom text or use it for different type other than a string. Note that current version only allows primitive types like Integer, Float, DateTime, etc.

During the Process method it first makes sure that it only runs for known content types, then it performs any validations raising a set of custom violations that are defined in the Violations static helper class. Note that we load the content from the Response Stream, which is the property that contains the received from the server. Note that if you were analyzing text the property Response would contain the content (this is based on Content Type, so HTML, XML, CSS, etc, will be kept in this String property).

Registering it

When running inside IIS Manager, crawler modules need to be registered as a standard UI module first and then inside their initialization they need to be registered using the IExtensibilityManager interface. In this case to keep the code as simple as possible everything is added in a single file. So add a new file called "RegistrationCode.cs" and include the contents below:

using System;
using Microsoft.Web.Management.Client;
using Microsoft.Web.Management.SEO.Crawler;
using Microsoft.Web.Management.Server;

namespace SampleCrawlerModule {
    internal class SampleCrawlerModuleProvider : ModuleProvider {
        public override ModuleDefinition GetModuleDefinition(IManagementContext context) {
            return new ModuleDefinition(Name, typeof(SampleCrawlerModule).AssemblyQualifiedName);
        }

        public override Type ServiceType {
            get { return null; }
        }

        public override bool SupportsScope(ManagementScope scope) {
            return true;
        }
    }

    internal class SampleCrawlerModule : Module {
        protected override void Initialize(IServiceProvider serviceProvider, ModuleInfo moduleInfo) {
            base.Initialize(serviceProvider, moduleInfo);

            IExtensibilityManager em = (IExtensibilityManager)GetService(typeof(IExtensibilityManager));
            em.RegisterExtension(typeof(CrawlerModule), new ImageExtension());
        }
    }
}

This code defines a standard UI IIS Manager module and in its client-side initialize method it uses the IExtensibilityManager interface to register the new instance of the Image extension. This will make it visible to the Site Analysis feature.

Testing it

To test it we need to add the UI module to Administration.config, that also means that the assembly needs to be registered in the GAC.

To Strongly name the assembly

In Visual Studio, you can do this easily by using the menu "Project->Properties", and select the "Signing" tab, check the "Sign the assembly", and choose a file, if you don't have one you can easily just choose New and specify a name.

After this you can compile and now should be able to add it to the GAC.

To GAC it

If you have the SDK's you should be able to call it like in my case:

"\Program Files\Microsoft SDKs\Windows\v6.0A\bin\gacutil.exe" /if SampleCrawlerModule.dll

(Note, you could also just open Windows Explorer, navigate to c:\Windows\assembly and drag & drop your file in there, that will GAC it automatically).

Finally to see the right name that should be use in Administration.config run the following command:

"\Program Files\Microsoft SDKs\Windows\v6.0A\bin\gacutil.exe" /l SampleCrawlerModule

In my case it displays:

SampleCrawlerModule, Version=1.0.0.0, Culture=neutral, PublicKeyToken=6f4d9863e5b22f10, …

Finally register it in Administration.config

Open Administration.config in Notepad using an elevated instance, find the </moduleProviders> and add a string like the one below but replacing the right values for Version and PublicKeyToken:

Use it

After registration you now should be able to launch IIS Manager and navigate to Search Engine Optimization. Start a new Analysis to your Web site. Once completed if there are any violations you will see them correctly in the Violations Summary or any other report. For example see below all the violations in the "Images" category.

Since we also extended the metadata by including the new fields (Image Width, Image Height, and Image Pixel Format) now you can use them with the Query infrastructure to easily create a report of all the images:

And since they are standard fields, they can be used in Filters, Groups, and any other functionality, including exporting data. So for example the following query can be opened in the Site Analysis feature and will display an average of the width and height of images summarized by type of image:

<?xml version="1.0" encoding="utf-8"?>
<query dataSource="urls">
<filter>
    <expression field="ContentTypeNormalized" operator="Begins" value="image/" />
</filter>
<group>
    <field name="ContentTypeNormalized" />
</group>
<displayFields>
    <field name="ContentTypeNormalized" />
    <field name="(Count)" />
    <field name="Average(iWidth)" />
    <field name="Average(iHeight)" />
</displayFields>
</query>

And of course violation details are shown as specified, including Recommendation, Description, etc:

Summary

As you can see extending the SEO Toolkit using a Crawler Module allows you to provide additional information, whether Metadata, Violations or Links to any document being processed. This can be used to add support for content types not supported out-of-the box such as PDF, Office Documents or anything else that you need. It also can be used to extend the metadata by writing custom code to wire data from other system into the report giving you the ability to exploit this data using the Query capabilities of Site Analysis.

IIS SEO Toolkit Extensibility

CarlosAg — Mon, 23 Nov 2009 21:46:56 GMT

The IIS SEO Toolkit includes a lot of functionality built-in such as built-in violation rules, processing of different content types (like HTML, CSS, RSS, etc) and more, however it might not do all the things that you would need it to do, for example, it might not process a set of documents that you use, or it might not gather all the information that you are interested in while processing a document. The good news is that it includes enough extensibility to let you build on top of its rich capabilities and provide additional ones easily using .NET.

There are three main extensibility points in this first release, including:

Crawler Module. This extensibility point allows you to provide your own code to hook to the process of crawling a Web site in the Site Analysis process. Using this extensibility point you can extend the built in set of violation rules with your own. You can also gather additional information such as links or any metadata of a resource, whether directly extracted from the content or from an external system.
Site Analyzer Extension. This feature allows you to provide your own set of tasks to be exposed in the Site Analysis user interface. These tasks will be displayed in the main menu bar in the report dashboard.
Sitemap Extension. This class allows you to provide your own set of tasks to be exposed in the Sitemaps, Sitemap and Sitemap Index user interface.

This is the first of a series of extensibility blog entries for the IIS SEO Toolkit where I will cover all of the extensibility points mentioned above.

Slides for IIS – Web Application Gallery presentation at DevConnections

CarlosAg — Mon, 23 Nov 2009 20:55:44 GMT

Two weeks ago I presented at DevConnections the talk "AMS10: Developing and Deploying for the Windows Web App Gallery", here are the slides.

Download the Web Application Gallery Talk slides here.

A few final links:

Microsoft Web Platform: http://www.microsoft.com/web/

Download Web PI: http://www.microsoft.com/web/downloads/platform.aspx

Submit your Applications at: http://www.microsoft.com/web/gallery/developer.aspx

IIS SEO Toolkit - New Reports (Redirects and Link Depth)

CarlosAg — Wed, 18 Nov 2009 23:33:00 GMT

In the new version of the IIS SEO Toolkit we added two new reports that are very interesting, both from an SEO perspective as well as from user experience and site organization. These reports are located in the Links category of the reports

Redirects

This report shows a summary of all the redirects that were found while crawling the Web site. The first column (Linking-URL) is the URL that was visited that resulted in redirection to the Linked-URL (second column). The third column (Linking-Status code) specifies what type of redirection happened based on the HTTP status code enumeration. The most common values will be MovedPermanently/Moved which is a 301, or Found/Redirect which is a 302. The last column shows the status code for the final URL so you can easily identify redirects that failed or that redirected to another redirect.

Why should you care

This report is interesting because Redirects might affect your Search Engine rankings and make your users have the perception that your site is slower. For more information on Redirects see: Redirects, 301, 302 and IIS SEO Toolkit

Link Depth

This is probably one of my favorite reports since it is almost impossible to find this type of information in any other 'easy' way.

The report basically tells you how hard it is for users that land in your home page to get to any of the pages in your site. For example in the image below it shows that it takes 5 clicks for a user to get from the home page of my site to the XGrid.htc component.

This is very valuable information because you will be able to understand how deep your Web site is, in my case if you were to walk the entire site and layout its structure in a hierarchical diagram it would basically be 5 levels deep. Remember, you want your site to be shallow so that its easily discoverable and crawled by Search Engines.

Even more interesting you can double click any of the results and see the list of clicks that the user has to make it to get to the page.

Note that it shows the URL, the Title of the page as well as the Text of the Link you need to click to get to the Next URL (the one with a smaller index). So as you can see in my case the user needs to go to the home page, click the link with text "XGrid", which takes it to the /XGrid/ url (index 3) which then needs to click the link with text "This is a new...", etc.

Note that as you select the URLs in the list it will highlight in the markup the link that takes you to the next URL.

The data of this report is powered by a new type of query we called Route Query. The reason this is interesting is because you can customize the report to add different filters, or change the start URL, or more.

For example, lets say I want to figure out all the pages that the user can get to when they land in my site in a specific page, say http://www.carlosag.net/Tools/XGrid/editsample.htm:

In the Dashboard view of a Report, select the option 'Query->New Routes Query'. This will open a new Query tab where you can specify the Start URL that you are interested.

As you can see this report clearly shows that if a user visits my site and lands on this page they will basically be blocked and only be able to see 8 pages of the entire site. This is a clear example on where a link to the Home page would be beneficial.

Other common scenarios that this query infrastructure could be used for is to find ways to direct traffic from your most common pages to your conversion pages, this report will let you figure out how difficult or easy it is to get from any page to your conversion pages

IIS SEO Toolkit - Start new analysis automatically through code

CarlosAg — Wed, 18 Nov 2009 06:00:28 GMT

One question that I've been asked several times is: "Is it possible to schedule the IIS SEO Toolkit to run automatically every night?". Other related questions are: "Can I automate the SEO Toolkit so that as part of my build process I'm able to catch regressions on my application?", or "Can I run it automatically after every check-in to my source control system to ensure no links are broken?", etc.

The good news is that the answer is YES!. The bad news is that you have to write a bit of code to be able to make it work. Basically the SEO Toolkit includes a Managed code API to be able to start the analysis just like the User Interface does, and you can call it from any application you want using Managed Code.

In this blog I will show you how to write a simple command application that will start a new analysis against the site provided in the command line argument and process a few queries after finishing.

IIS SEO Crawling APIs

The most important type included is a class called WebCrawler. This class takes care of all the process of driving the analysis. The following image shows this class and some of the related classes that you will need to use for this.

The WebCrawler class is initialized through the configuration specified in the CrawlerSettings. The WebCrawler class also contains two methods Start() and Stop() which starts the crawling process in a set of background threads. With the WebCrawler class you can also gain access to the CrawlerReport through the Report property. The CrawlerReport class represents the results (whether completed or in progress) of the crawling process. It has a method called GetUrls() that returns an instance to all the UrlInfo items. A UrlInfo is the most important class that represents a URL that has been downloaded and processed, it has all the metadata such as Title, Description, ContentLength, ContentType, and the set of Violations and Links that it includes.

Developing the Sample

Start Visual Studio.
Select the option "File->New Project"
In the "New Project" dialog select the template "Console Application", enter the name "SEORunner" and press OK.
Using the menu "Project->Add Reference" add a reference to the IIS SEO Toolkit Client assembly "c:\Program Files\Reference Assemblies\Microsoft\IIS\Microsoft.Web.Management.SEO.Client.dll".
Replace the code in the file Program.cs with the code shown below.
Build the Solution

using System;
using System.IO;
using System.Linq;
using System.Net;
using System.Threading;
using Microsoft.Web.Management.SEO.Crawler;

namespace SEORunner {
    class Program {

        static void Main(string[] args) {

            if (args.Length != 1) {
                Console.WriteLine("Please specify the URL.");
                return;
            }

            // Create a URI class
            Uri startUrl = new Uri(args[0]);

            // Run the analysis
            CrawlerReport report = RunAnalysis(startUrl);

            // Run a few queries...
            LogSummary(report);

            LogStatusCodeSummary(report);

            LogBrokenLinks(report);
        }

        private static CrawlerReport RunAnalysis(Uri startUrl) {
            CrawlerSettings settings = new CrawlerSettings(startUrl);
            settings.ExternalLinkCriteria = ExternalLinkCriteria.SameFolderAndDeeper;
            // Generate a unique name
            settings.Name = startUrl.Host + " " + DateTime.Now.ToString("yy-MM-dd hh-mm-ss");

            // Use the same directory as the default used by the UI
            string path = Path.Combine(
                Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments),
                "IIS SEO Reports");

            settings.DirectoryCache = Path.Combine(path, settings.Name);

            // Create a new crawler and start running
            WebCrawler crawler = new WebCrawler(settings);
            crawler.Start();

            Console.WriteLine("Processed - Remaining - Download Size");
            while (crawler.IsRunning) {
                Thread.Sleep(1000);
                Console.WriteLine("{0,9:N0} - {1,9:N0} - {2,9:N2} MB",
                    crawler.Report.GetUrlCount(),
                    crawler.RemainingUrls,
                    crawler.BytesDownloaded / 1048576.0f);
            }

            // Save the report
            crawler.Report.Save(path);

            Console.WriteLine("Crawling complete!!!");

            return crawler.Report;
        }

        private static void LogSummary(CrawlerReport report) {
            Console.WriteLine();
            Console.WriteLine("----------------------------");
            Console.WriteLine(" Overview");
            Console.WriteLine("----------------------------");
            Console.WriteLine("Start URL: {0}", report.Settings.StartUrl);
            Console.WriteLine("Start Time: {0}", report.Settings.StartTime);
            Console.WriteLine("End Time:   {0}", report.Settings.EndTime);
            Console.WriteLine("URLs:       {0}", report.GetUrlCount());
            Console.WriteLine("Links:      {0}", report.Settings.LinkCount);
            Console.WriteLine("Violations: {0}", report.Settings.ViolationCount);
        }

        private static void LogBrokenLinks(CrawlerReport report) {
            Console.WriteLine();
            Console.WriteLine("----------------------------");
            Console.WriteLine(" Broken links");
            Console.WriteLine("----------------------------");
            foreach (var item in from url in report.GetUrls()
                                 where url.StatusCode == HttpStatusCode.NotFound &&
                                       !url.IsExternal
                                 orderby url.Url.AbsoluteUri ascending
                                 select url) {
                Console.WriteLine(item.Url.AbsoluteUri);
            }
        }

        private static void LogStatusCodeSummary(CrawlerReport report) {
            Console.WriteLine();
            Console.WriteLine("----------------------------");
            Console.WriteLine(" Status Code summary");
            Console.WriteLine("----------------------------");
            foreach (var item in from url in report.GetUrls()
                                 group url by url.StatusCode into g
                                 orderby g.Key
                                 select g) {
                Console.WriteLine("{0,20} - {1,5:N0}", item.Key, item.Count());
            }
        }
    }
}

If you are not using Visual Studio, you can just save the contents above in a file, call it SEORunner.cs and compile it using the command line:

C:\Windows\Microsoft.NET\Framework\v3.5\csc.exe /r:"c:\Program Files\Reference Assemblies\Microsoft\IIS\Microsoft.Web.Management.SEO.Client.dll" /optimize+ SEORunner.cs

After that you should be able to run SEORunner.exe and pass the URL of your site as a argument, you will see an output like:

Processed - Remaining - Download Size
       56 -       149 -      0.93 MB
      127 -       160 -      2.26 MB
      185 -       108 -      3.24 MB
      228 -        72 -      4.16 MB
      254 -        48 -      4.98 MB
      277 -        36 -      5.36 MB
      295 -        52 -      6.57 MB
      323 -        25 -      7.53 MB
      340 -         9 -      8.05 MB
      358 -         1 -      8.62 MB
      362 -         0 -      8.81 MB
Crawling complete!!!

----------------------------
 Overview
----------------------------
Start URL:  http://www.carlosag.net/
Start Time: 11/16/2009 12:16:04 AM
End Time:   11/16/2009 12:16:15 AM
URLs:       362
Links:      3463
Violations: 838

----------------------------
 Status Code summary
----------------------------
                  OK -   319
    MovedPermanently -    17
               Found -    23
            NotFound -     2
 InternalServerError -     1

----------------------------
 Broken links
----------------------------
http://www.carlosag.net/downloads/ExcelSamples.zip

The most interesting method above is RunAnalysis, it creates a new instance of the CrawlerSettings and specifies the start URL. Note that it also specifies that we should consider internal all the pages that are hosted in the same directory or subdirectories. We also set the a unique name for the report and use the same directory as the IIS SEO UI uses so that opening IIS Manager will show the reports just as if they were generated by it. Then we finally call Start() which will start the number of worker threads specified in the WebCrawler::WorkerCount property. We finally just wait for the WebCrawler to be done by querying the IsRunning property.

The remaining methods just leverage LINQ to perform a few queries to output things like a report aggregating all the URLs processed by Status code and more.

Summary

As you can see the IIS SEO Toolkit crawling APIs allow you to easily write your own application to start the analysis against your Web site which can be easily integrated with the Windows Task Scheduler or your own scripts or build system to easily allow for continuous integration.

Once the report is saved locally it can then be opened using IIS Manager and continue further analysis as with any other report. This sample console application can be scheduled using the Windows Task Scheduler so that it can run every night or at any time. Note that you could also write a few lines of PowerShell to automate it without the need of writing C# code and do that by only command line, but that is left for another post.

Announcing: IIS SEO Toolkit v1.0 release

CarlosAg — Tue, 17 Nov 2009 21:06:21 GMT

Today we are announcing the final release of the IIS Search Engine Optimization (SEO) Toolkit v1.0. This version builds upon the Beta 1 and Beta 2 versions and is 100% compatible with those versions so any report you currently have continues to work in the new version. The new version includes a set of bug fixes and new features such as:

Extensibility. In this version we are opening a new set of API's to allow you to develop extensions for the crawling process, including the ability to augment the metadata in the report with your own, extend the set of tasks provided in the Site Analysis and Sitemaps User Interface and more. More on this on a upcoming post.
New Reports. Based on feedback we added a Redirects summary report in the Links section as well as a new Link Depth report that allows you to easily know which pages are the "most hidden pages" in your site, or in other words if a user landed at your sites home page, "how many clicks does he need to do to reach a particular page".
New Routes Query. We added a new type of Query called Routes. This is the underlying data that powers the "Link Depth" report mentioned above, however it is also exposed as a new query type so that you can create your own queries to customize the Start page and any other kind of things, like filtering, grouping, etc.
New option to opt-out from keeping a local cache of files. We added a new switch in the "Advanced Settings" of the New Analysis dialog to disable the option of keeping the files stored locally. This allows you to run a report which runs faster and that consumes a lot less disk space than when keeping the files cached. The only side effect is that you will not be able to get the "Content" tab and the contextual position of the links as well as the Word Analysis feature. Everything else continues to work just as any other report.
HTML Metadata is now stored in the Report. By leveraging the Extensibility mentioned in bullet 1, the HTML parser now stores all the HTML META tags content so that you can later use them to write your own queries, whether to filter, group data or just export it, this gives you a very interesting set of options if you have any metadata like Author, or any custom.
Several Bug Fixes:

Internal URLs linked by External URLs now are also included in the crawling process.
Groupings in queries should be case sensitive
Show contextual information (link position) in Routes
The Duplicate detection logic should only include valid responses (do not include 404 NOT Found, 401, etc)
Canonical URLs should support sub-domains.
Several Accessibility fixes. (High DPI, Truncation in small resolutions, Hotkeys, Keyboard navigation, etc).
Several fixes for Right-To-Left languages. (Layout and UI)
Help shortcuts enabled.
New Context Menus for Copying content
Add link position information for Canonical URLs
Remove x-javascript validation for this release
Robots algorithm should be case sensitive
many more

This version can upgrade both Beta 1 and Beta 2 version so go ahead and try it and PLEASE provide us with feedback and any additional things you would like to see for the next version at the SEO Forum in the IIS Web site.

Click here to install the IIS SEO Toolkit.

IIS SEO Toolkit Presentation at DevConnections

CarlosAg — Wed, 11 Nov 2009 16:41:00 GMT

Yesterday I presented the session "AMS04: Boost Your Site’s Search Ranking with the IIS Search Engine Optimization Toolkit" at the ASP.NET Connections, it was fun to talk to a few attendees that had several questions around the tool and SEO in general. It is always really interesting learning about all the unique environments and types of applications that are being built and how the SEO Toolkit can help them.

Here are the IIS SEO Toolkit slides that I used.

Here you can find the IIS SEO Toolkit download.

And by far the easiest way to get it installed is using the Microsoft Web Platform Installer.

Please send any question and feedback at IIS SEO Toolkit Forums.

And by the way, stay tuned for the RTW version of IIS SEO Toolkit coming SOON.

IIS SEO Toolkit – Report Comparison

CarlosAg — Wed, 11 Nov 2009 00:09:00 GMT

One of my favorites features in the IIS Search Engine Optimization (SEO) Toolkit is what we called Report Comparison. Report Comparison basically allows you to compare two different versions of the results of crawling the same site to see what changed in between. This is a really convenient way to track not only changes in terms of SEO violations but also to be able to compare any attributes on the pages such as Title, Heading, Description, Links, Violations, etc.

How to access the feature

There are a couple of ways to get to this feature.

1) Use the Compare Reports task. While in the Site Analysis Reports listing you can select two reports by using Ctrl+Click, and if both reports are compatible (e.g. they use the same Start URL) the task "Compare Reports" will be shown. Just clicking on that will get you the comparison.

2) Use the Compare to another report menu item. While in the Dashboard view of a Report you can use the "Report->Compare To Another Report" menu item which will show a dialog where you can either select an existing report or even start a new analysis to compare with.

Report Comparison Page

In both cases you will get the Report Comparison Page displaying the results as shown in the next image.

The Report Comparison page includes a couple of "sections" with data. At the very top it includes links showing the Name and the Date when the reports were ran. If you click on them it will open the report directly just as if you had used the Site Analysis report listing view.

The next sections shows a lot of interesting built-in data such as:

Total # of URLs	This basically shows the total # of URLs found in both versions. When clicking the link you will get the listing of URLs based on the version of the report you choose.
New and Removed	These are the number of new URLs that were either added in the new version or removed from the old version. When clicking the added link you will get the listing of URLs based on the new version of the report and if you click the removed link you will get the listing based on the old URLs.
Changed and Unchanged	These are the number of URLs that were modified or not modified. These are calculated by comparing the hashes of the files in both versions. When clicking the links you will get a query that displays a comparison of both versions of URLs showing their content length. (See below)
Total # of Violations	This shows the total # of violations found in both versions.
New in existing pages and Fixed in existing pages	These are the number of violations introduced or removed on URLs that exist in both reports. When clicking the added link you will get the listing of violations based on the new version of the report and if you click the removed link you will get the listing based on the old violations.
Introduced in new pages	These are the number of violations introduced on URLs that are found only in the new report. When clicking the added link you will get the listing of violations based on the new version of the report.
Fixed by page removal	These are the number of violations that were removed due to the fact that their URLs were no longer found in the new report. When clicking the added link you will get the listing of violations based on the old version of the report.
Others	There are a number of additional reports which basically compare different attributes in URLs that are found in both reports. They compare things like Time Taken, Content Length, Status Code and # of Links. When clicking the links you will get the query that displays a comparison of both versions of the reports showing the relevant fields. (See below)

Whenever you click the links you get a query dialog that you can customize just as any Query in the Query builder, where you can Add/Remove columns, add filters, etc.

My favorite one is the "Modified URLs" source when you actually can add filters that compare URLs coming from the two different reports.

Note that when you double click or "right-click –> Compare Details" any of the rows you get a side-by-side comparison of everything in the URL:

Again, you can use any of the tabs to see side-by-side things like the Content of the pages or the Links both versions have or the violations, or pretty much everything that you can see for just one.

Finally, you can also right click on the Query dialog and choose "Compare Contents". This will launch whatever File Comparison tool you have configured using the "Edit Feature Settings". In this case I have configured WinDiff.exe which shows something like:

Summary

As you can see Report Comparison offers is a powerful feature that allows you to keep track of changes between two different reports. This easily allows you to understand over time how your site has been affected by changes. For Site managers it will allow them to query and maintain a history with all the changes. You can imagine that using an automated build process that runs IIS SEO Toolkit crawling whenever a build is made that keeps the report stored somewhere and potentially annotate it with the build number you could even keep a correlation of changes in code with Web site crawling.