Like any database driven site, those hundreds of thousands of pages can be boiled down to a few distinct templates, for me they are:

1. Home page
2. “Article” pages (FAQs, online help, about us, contact us, etc.)
3. Locality page (e.g. best rated Sydney metro businesses)
4. Industry + locality page (e.g. Sydney hairdressers)
5. Business listing page (e.g. Toni & Guy Bondi Beach)

The bottom three templates (in reverse order) represent the most unique pages per template and make up 99% of organic traffic to the site.

Currently, my URL structure is pretty systematic, so using Google search filters such as “site:abc.com inurl:business_listing” I am able to get total number of indexed pages on a template by template basis.

However, in the next few months I intend to improve some URLs from say /business-listing/toni-guy-bondi-beach/12345-54321.html to simply /toni-guy-bondi-beach/ which will make make it impossible to track total indexed pages using my current method.

But, if you’re in the same position there is a solution, which so simple that I feel like a post dedicated to it is overkill. Because it’s a database driven site, I store all the page aliases (e.g. “toni-guy-bondi-beach”) in my locality, industry and business listing database tables.

This allows me to update my script to separate each template into different sitemaps. Google Webmaster Tools will then show you the total # of URLs in the sitemap vs. the number indexed:

By dividing the numbers, I can easily see which templates aren’t performing as well and look to see if there is any factors causing it to be considered duplicate content or if it’s an internal linking issue, etc.

If you’re interested in learning more about large site indexation, there is an SEOmoz post by Rand on Google’s indexation cap.

Migrating to a new platform or server is always a risky time for any site that relies on organic traffic. There’s a big risk of pages going missing and redirects not working properly.

The new site featured a new design plus a different CMS platform but essentially had an identical URL structure, so ensuring existing URLs still worked was the primary goal.

I wanted to do the following on both the staging site and post-migration on the live site:

1. Get the URLs indexed by search engines
2. Batch testing redirections

Indexed URLs

On a database driven site with thousands of pages, it’s not always possible to get a complete list of possible URLs, so we need to prioritise the URLs that search engines are aware of.

For smaller sites (under 1,000 pages), GSiteCrawler does a reasonable job. The downsides are that it puts unnecessary load on you web server and secondly I find it crashes for larger sites.

My preferred method is to get it from a search engine index. Grabbing index data from the major engines can be a hassle. Scraping the engines is cumbersome and it’s a hassle when you get thrown a captcha.

I prefer to use Majestic SEO which provides data from a smaller search engine they run. It uses similar crawl algorithms to Google, so it’s going to be a very similar dataset, and best of all it’s free to use on your own site.

Once you’ve validated your site, go to Domain URLs > Download All and all the URLs you’ll need to redirect will be in the first column.

Note: I recommend against using the sitemap XML as it’s likely to be an incomplete picture.

Batch testing URLs

When migrating a site, the kinds of errors you don’t want to see previously working URLs giving are 404 not found, 401 unauthorized and 500 internal server errors.

I was using a sub-domain on the staging site, so once I had my list of URLs all I needed to do was search and replace “http://www.” with “http://dev.” in Excel then get a good cross section and run it through an HTTP header checker.

I put through 500 URLs at a time through my own batch HTTP header checker and fixed up any pesky 404s I found.

Post migration, I picked another set of URLs to test and again got positive results. To be 100% sure, I will be logging into Google Webmaster Tools tomorrow morning to check for 404s.

Good luck with your site migration!

Specifically, when you link AdWords and Analytics, Google will append a gclid to destination URLs which Google uses to match keywords, campaigns and other dimensions to give you richer data in GA.

However, most webmasters won’t implement the right directives to pass on the query string, so the user clicks on the redirected URL, the following happens and the gclid is lost in the redirection.

Are my redirects AdWords friendly?

There’s a very simple way to check if your redirect is AdWords and Analytics friendly, simply try going to the old destination URL and append “?gclid=testing” on the end. If it redirects to the new URL and keeps it on the end, you don’t need to be on the phone with your webmaster!

Fixing with Apache & .htaccess

If the above tip didn’t pass on the gclid for you, the good news is there’s an easy way to fix this in Apache:

Further reading on .htaccess

• Batch HTTP header check
• Advanced .htaccess tips & tricks