One of the more common emails we get is from people who think their website has disappeared from Google. If you’re in a similar situation, we have some ideas for you in this article.

If your previously high-ranking website seems to have fallen out of Google altogether, you are probably teetering somewhere between panic and desperation. As friendly consultants who care about your mental health, we want you to know this first:

Don’t panic! You probably didn’t get banned.
Give this a few hours and see if it works itself out.

We say that because it’s the most common scenario that we’ve seen.  But if you can’t bear to wait around to see if the problem will resolve itself, read on.

The first thing you need to do is clarify the problem. Specifically, has your entire site been removed from Google, or has it just been knocked down in the rankings?  To figure this out, you can do a site search, by typing this into Google:

site: www.myurl.com

(replacing “myurl” with your own URL, of course).  If Google doesn’t display any pages from your site, it’s likely that your entire site has been dropped.  Another way to check is to take a unique string of text from your website, one that is unlikely to be on anyone else’s site, put it in quotes, and search for it on Google. If Google displays the page from your site that contains this text, then your site hasn’t been dropped from Google.

If your site is still in Google, but your ranks have taken a sudden and dramatic nosedive, skip down to the “What Happened to my Ranks?” section, below. If, after several strategic searches, you still can’t find your site on Google, then yes indeed, your site has been removed from the index, and it’s time to figure out why.

Unintentional Spamming?
It’s possible that your site was found guilty of spamming in Google’s eyes, and got banned. Websites can be banned for SEO spam techniques such as “white on white” text, blatant keyword stuffing, or tricky redirects. J.C. Penney’s website was famously banned temporarily for using a link-buying SEO technique.  It’s not likely that your website would be banned if you’ve never done anything to intentionally trick Google into thinking that your site deserves a higher rank than is appropriate.

If you’re not sure about your status as spammer, SEO expert Jill Whalen has a forum that discusses “SEO No-nos.” Read it to get some indication of whether your site was spamming inadvertently.

You’ll also want to check all your messages in Google Webmaster Tools.  There, Google will sometimes (but not always) alert website owners that their site has been penalized for spam.

Victim of Foul Play?
Whether you’re into conspiracy theories or not, you should entertain the idea that someone else did the spamming for you, causing your site’s disappearance from Google. We’ve seen it more than once: a site’s domain expires for a short time, and during that period, the pages are taken over by a domain squatter that displays not-so-nice content. Or, your site may have been hacked and stuffed with nuggets of spammy badness (this is especially common with outdated WordPress versions, so keep your WordPress up-to-date, yo!).

The fallout from one of these scenarios is something we describe in our case study of Providence Restaurant. (It’s worth noting, by the way, that even though that site temporarily displayed spam content, it suffered with terrible ranks, but wasn’t entirely dropped from Google’s index.)

Again, Google Webmaster Tools can help you diagnose this problem.  Check your messages to see if there have been any alerts describing malware or other problems on your site.

Another form of devious doings on the web is content scraping, which is when other websites steal content from your site and then republish it, often multiple times.  If this has happened to you, it is possible that Google banned your site along with the “bad guys” who copied you. Try doing some searches on other search engines, looking for strings of text from your website.  You just might find an unexpected match.

Redirects or Canonical Tags Gone Wrong?
We once had a client who inadvertently added  a canonical tag on every page of his site pointing to a different domain.  Google followed this instruction diligently and removed every page of his site from its index.  Yeowch!  The good news: once the errant canonical tag was removed, ranks and indexing were back where they belonged within a couple of weeks.

We have also seen examples of sites that redirect their pages to another domain using a 301 redirect, and then are surprised when their domain is dropped from Google.   A 301 redirect is like a permanent change-of-address for search engines.  If you redirect your entire domain to somewhere else, Google is not going to continue to display your domain in its search results.

Robots.txt or Robots Meta Tag Deindexing?
Have you double-checked that you didn’t accidentally deindex your entire site with the robots.txt file or robots meta tags?

Here’s the line you really don’t want to have in your robots.txt file:

User-agent: *
Disallow: /

And here is a tag that really shouldn’t be on every page of your site:

<meta name=”robots” content=”noindex”>

Go on… check it now.  We won’t judge you.

Server Outage?
Your server might have hiccupped just when Googlebot came to visit your site recently, but that shouldn’t cause your site to be dropped from the index.  We believe that Google will check a site at least a couple of times before dropping it.   But, if the site went down for an extended period (days, weeks), this could be an explanation.

Most likely, the bot will come back and rediscover your site, and you’ll find your site back where it belongs after that.  But how long will you have to wait?  It depends on how many links there are to your site and how often Googlebot visits it. While there’s no sure-fire way to trigger a visit from Googlebot, getting some new inbound links, refreshing your content a bit, and submitting to the free URL submittal page probably won’t hurt.  You can submit an XML Sitemap via Google Webmaster Tools, which has the nice fringe benefit of telling you how many of your URLs are indexed.

In the future, if your site has anticipated downtime, set a 503 server status so that search engines will know it’s just a temporary glitch.

Reinclusion Request
If you are certain your site has been penalized for spam or malware, you can submit to Google’s request for reinclusion page. Matt Cutts of Google has posted some information on how to file a reinclusion request. But before you file a request, be very, very sure that there’s nothing spammy on your site.

What Happened To My Ranks?
If Google hasn’t actually removed your site, but you just noticed a sudden and substantial drop in rank for your favorite keywords, there are several possible explanations:

• Google could have made an intentional change to its results or its algorithm.  Panda and Mayday are a couple examples of famous algorithm shifts that blew some sites out of the water.  SearchEngineLand.com is a good place to look into industry chatter about algorithm updates.  See also our post about recovering from Panda.

• Google could be experiencing some temporary weirdness that makes the results different from what you’re used to. As we mentioned at the top of this article, we’ve seen this so many times with so many clients that this is probably the most likely scenario. In this case, just check your ranks again in a day or so . You might once again see that good rank you were used to.

• Your competitors could have gotten their acts together. OK, if your rank dropped by a few hundred spots, it’s probably not your competitors, as that would mean a lot of competitors getting their acts together at the same time!  But if you’re just being outranked by legitimate competitive activity, then it’s time for you to work on your site. And this work includes optimizing for, and tracking, additional keywords.

• Is there something funky going on with your homepage, but not other pages?  We’ve seen sites that have suffered big rank drops because their homepage had been deindexed due to redirect or canonical issues, and no other page had as much power as the homepage, so no other page on the domain was making it into those previous ranking spots.  If you’re seeing only a portion of your site in the index, run through the diagnostic steps above to see if you can figure out why.
• Is your website new?  Sometimes, websites show up in search results briefly after being launched, but then seem to drop dramatically out of the rankings after a couple of days, and are stuck in the depths for months.  It’s a phenomenon that some have called the “Google Sandbox.”  How to get out the sandbox? Get some inbound links to your site, and be patient.

• Is it possible that something has dramatically changed on your website? Say, you intended to update a single page but accidentally changed the page titles on every page of your site?  Don’t forget to check your site and make sure everything is in order.
• Could you be seeing different results because your personalization has changed?  We’ve known people who thought their ranks were phenomenal – only to be brought down to earth when they realized that the ranks they were seeing were personalized just for them by Google.   Read more about Google search personalization.

Hopefully we’ve given you some ideas about how to track down the source of your site’s disappearance from Google. Regardless of what you find, it’s always a good idea to make sure you’re offering lots of great content that makes people want to link to your site, and do some link building. In this way, you’ll have other sources of traffic that makes a Google problem like this less devastating.

Upon bouncing back to Google from a search result, you may see a notice allowing you to block that domain from future Google results.  Or you may see a notice encouraging you to +1 that domain.  It all depends on the amount of time you spent on the page.

Google is well aware of what listings searchers click after performing a Google search.  But did you know Google continues to care what a user does after clicking on a listing?  The search engine monitors whether the user clicks right back to Google, and even pays attention to the length of time the user spent on a site.

Here’s an example.  After searching for “eco-friendly down comforters” I clicked a page on the domain www.beddingdowncomforters.com, reviewed it briefly, and clicked the Back button to return to Google.  This is known as a G-bounce. When I landed back on the Google search results page, Google had added the message ” – Block all www.beddingdowncomforters.com results” to the listing:

Click on the “Block…”  link, and you’ll never be bothered by this website’s listings again.

Later, I was searching for “how to decant wine” and clicked on wineintro.com.   I spent a long time on the page and eventually clicked the Back button to return to Google.  Upon landing back on my Google results page, I saw a popup suggesting that I +1 the site on Google:

This experience got me wondering:  How long do I need to spend on a website before Google stops suggesting I block it, but instead suggests that I +1 it?  What is that magic length of G-bounce time that Google deems to represent a successful website experience?

I tested it out by G-bouncing after various lengths of time, and here’s what I found:

• 15 seconds — block
• 30 seconds — block
• 45 seconds — block
• 1 minute — block
• 1 minute 15 seconds — block
• 1 minute 30 seconds — block
• 2 minutes — no message
• 3 minutes — no message
• 4 minutes — no message
• 7 minutes — no message
• 7 minutes + 30 seconds – no message
• 7 minutes + 40 seconds -  +1 message
• 8 minutes – +1 message

So apparently, according to Google, the 1-1/2 minute mark is where a site goes from hateworthy to just OK, and at the 7-1/2 minute G-bounce mark, the site has clearly served you well and deserves a +1 for its trouble.

I anyone else seeing these same times?  Different ones?

When you’re launching a new app, naturally you want everyone in the world to see it!  But many of the app review sites you’ll find in Google are spammy or require a payment for submittal.  Try this list for free app review submittals.

Before you submit, do a little prep work.  You’ll need a promo code if your app is not free.  And your app will have a much better chance of getting reviews if you create a fantastic demo video.  That way reviewers don’t have to download your app to see just how fun and fantastic it is!

When you’re ready to get started, here are your marching orders:

11 Great App Review Sites to Submit Your App For a Free Review

• http://www.macworld.com/info/appsubmit.html
• http://appadvice.com/appnn/contact-info-developers/
• http://www.appcraver.com/contact/
• http://www.appolicious.com/
• http://www.appstoreapps.com/contact/
• http://www.tuaw.com/contact/appreviews
• http://www.iusethisapp.com/submit-app/
• http://www.theiphoneappreview.com/submit-your-app/
• http://www.slapapp.com/request-a-review
• http://www.appchatter.com/submit/app-for-review/
• http://forms.whatsoniphone.com/forms/woip_submitanapp

Readers, do you have any more to suggest?

Q: I just hired a developer for my small-budget site development project. Is there a Top 10 list of things I should insist on SEO-wise to be built into the site?

A: As we all know by now, SEO consists of several different endeavors:

• Technical SEO: ensuring a search-friendly and social-friendly site from a structural/technical standpoint
• Content/Editorial SEO: creating content that aligns with your target audience’s search behavior and keywords
• Link appeal and link-building

It’s obvious that your web developer’s work influences the technical side of SEO.  But structural issues and CMS capabilities will also have long-term effects on your ability to maximize your content/editorial SEO and link power.   Here are my top 10 developer requirements for SEO:

• Each page on the site must exist on a separate, unique, permanent URL that is indexable by search engine robots
• All pages on the site must be accessible by clicking standard HTML links, preferably no more than 3 clicks from the home page
• Text content on the site must be search-engine-indexable, standard, visible HTML text
• Search engine robots are like site visitors that do not accept cookies, do not fill out forms, do not log-in, and have limited javascript and CSS capabilities.  These robots should be able to read versions of each page that match as closely as possible what a human visitor would see.
• Every page on the site must have a unique, descriptive HTML title.   The CMS should allow the site owner to edit the HTML title separately from editing other elements of the page, such as the page heading.
• We recommend that every page on the site include a Facebook Like, Google +, and Tweet button.
• The CMS should allow the site owner to edit OG tags on each page.
• If there is any chance that identical pages will be seen via more than one URL (creating potential duplicate content in search engines), the CMS should allow the site owner to include/edit a canonical tag on each page.
• URLs should be meaningful, human-readable and include the page title or relevant keywords
• Images that are clickable should have ALT and TITLE tags that provide a text description of the destination page being clicked to.

Granted, every one of these won’t apply to every project, but for the majority of small budget jobs, this list will form a strong foundation on which you can build your SEO efforts.

And one last piece of advice: communicate like crazy and stay on ‘em.  If you don’t have an outside SEO vendor involved in the project, you will need to serve the role of SEO watchdog during development.

Q: I have an ecommerce site containing a useful library of product information that the manufacturer allows me to use on my site.  My site was hammered in Google’s Panda update, and I think it was because of this “duplicate” content.  What should I do now?  I don’t want to remove this library because it is useful for my customers.

A: Even though this library of content is genuinely helpful for your site visitors, it could be similar to the low-quality, “stub”, or content farm materials that were punished in Google’s Panda update.  To make matters worse, the product information library you’re describing is probably also present on many other sites as well.  These two factors are both likely to be viewed negatively in Google’s eyes.

To play it safe, the easiest thing you can do is to deindex all these pages using either the robots.txt file or a robots meta tag on the concerning pages.  This way, the materials can still help your website visitors, without putting your site at any risk for Panda issues.

If deindexing the content is not an option, then you should work on improving the content, with an eye toward improving the user engagement on these pages.  What can you do to make these pages more unique, more interesting, and more likely to be a satisfying experience for searchers who land on these pages?

Not interested in rewriting hundreds or thousands of pages? Here’s a hybrid approach you might try:

• Deindex any pages that have a 90% or higher bounce rate
• For pages that have better (lower) than 90% bounce rate, work on improving content.  Possible ways to improve content:
• manually add your own links to related materials elsewhere on your site.  Make them intriguing or attractive enough to encourage visitors to click on them; remember your goal is to engage visitors and prevent bounces.
• manually add commentary/analysis that makes these materials more useful.  Be sure this is unique and well written.
• Proofread.
• add a prominent Google +1, Facebook Like, and Tweet button
• In addition to these steps, you should also pursue other SEO efforts, especially link-building, in order to gain stature in the Post-Panda search world.

Like any SEO effort, approach your Panda recovery with an iterative mindset.  Make some changes, watch for results over a several-week period, and repeat.  And a final word of wisdom: Keep the user experience a priority, and you can’t go wrong.

Tongue twister, anyone?  Pagination canonicalization is one of the trickier challenges for the technical SEO set today.  We’ve been closely monitoring the hints and tips that occasionally drop from our favorite Googlers’ lips (sure, we all adore Matt Cutts, but let’s save a little love for JohnMu and Maile Ohye, shall we?).  Here’s a roundup of good advice we’ve heard, along with our analysis and resulting recommendations.

Pagination is commonly seen on ecommerce sites when a category contains more products than can be listed on each page:

The question often arises:  How should an SEO handle all of these pages?  Allowing all of these pages to be indexed separately in search engines means indexing a lot of duplicate or near-duplicate content on a website, which dilutes SEO power and creates an undesirable user experience, since nobody particularly wants to click through from Google to a deep paginated page.

Here is the wisdom we’ve gathered from our favorite Googlers on the subject:

JohnMu on paginated pages and the canonical tag (Feb, 2010):

Pagination: this is complicated, I personally would be careful when using with rel=canonical with paginated lists. The important part is that we should be able to find all products listed, so at the very least those lists should provide a default sort order where we can access (and index) all pages. Since this is somewhat difficult unless you really, really know what you are doing, I would personally avoid adding rel=canonical for these pages. One possible solution could be to use JavaScript for paginated lists with different sort orders, for example, that way you would have a single URL which lists all products.

Our interpretation: Google needs to be able to see all of these paginated pages so that it can click through and get to all the individual links (product pages) that are listed on all the pages.  JohnMu suggests creating a default page that contains all of the links, so that Google can get to them.   Trouble is, what if you have a thousand of these links?  Or even more.

At SMX West (March 2011), Maile Ohye noted that you shouldn’t use canonical tags for paginated pages. (We’ve heard her say this since 2010) Here are notes from Barry Schwartz (SEORoundtable) on this session:

Maile explained that since the results on pages 2, 3, 4, and 5 are different from page 1, you should not use the canonical tag here.
Not only that, if you do, Google may ignore it because Google uses methods to determine if the canonical tag command is actually something valid for that case. So if you canonical page 2 to page 1 and page 2 is not similar enough to page 1, Google may ignore your canonical tag.

Our interpretation:  Since paginated pages aren’t identical to each other, and they aren’t subsets of each other, they shouldn’t be canonicalized to each other.  Google is pretty smart and can probably figure out when it’s dealing with page=1, page=2, page=3, and Google may ignore your canonical tags in this case.

The one exception to the no canonical tags for pagination rule is when you have a “view all” page.  As Brian Ussery of SEO firm Nine by Blue notes:

If paginated content uses “view all” page if it loads well you can put rel canonical on those URLs.
If you don’t have a “view all” page 2 isn’t a subset so you can’t use rel canonical.

At the end of the day, our two favored pagination approaches are the following:

• noindex,follow
  Place a robots meta tag on all deep paginated pages instructing the robots to “noindex, follow” the page.  This will allow Google to visit the deep pages, follow links on these pages, but not index these pages.
  
• “view all” + canonical
  Create a “view all” page showing all of the links on your list.  Then, on all other paginated pages (including page 1), add a canonical tag identifying the “view all” page as the canonical form.   We think this solution is best used when the “view all” page is also the version of the page that is linked from elsewhere on the site.

UPDATE:  Google got its act together and created a better way to deal with pagination canonicalization.  Read all about it at Google’s webmaster blog.

Q: I’m confused. What are Schema.org tags and are these the same as semantic tags for rich snippets? Which ones should I use?

A:  Have you ever wondered how some websites’ listings in Google and Bing get those great-looking review stars, price ranges, and other enhancements, like the ones shown here?

This is all achieved through the magic of semantic tagging.  Semantic tagging allows website owners to put tags around their content to help search engines understand the meaning of that content.  Say a website owner put a <price> tag around the price of a product:  <price>14.99</price>.  That would help a search engine understand that it’s looking at the price of the product.  Then, the search engine can use this information to improve its listings for your site.

Semantic tags for rich snippets have been supported by Google for years, but the new Schema.org tagging expands the type of information that can be tagged, and is also supported by Bing.  Here’s a quick overview:

• Semantic tagging per Schema.org.   This was recently announced by Google and Bing and is remarkable for being supported by both of the rival search engines.   Because Schema.org tagging is still very new, we don’t know exactly how it will be used by Google and Bing, but we expect that it will be used to improve search engine listings with rich snippets, and could help with positioning in Shopping search.

• Semantic tagging for rich snippets using microformats or RDFa.   This is an older form of semantic tagging that has been supported by Google for at least a couple of years.  It uses a different format than Schema.org tagging but provides some of the same information.  It is more limited than Schema.org tagging and only provides a few pieces of information about a product, including name of product, price, availability, and the number of review stars.

Which one should you use?  We recommend Schema.org because it is fully supported by both major search engines, Google and Bing.

Are there any possible downsides to using semantic tagging?  Yes.  Semantic tagging could help search engines create content pages that give visitors the information they’re looking for without clicking through to your site. Many website owners already feel that Google News, Google Places and Google Shopping have intercepted potential visitors to their site.  Some content creators think search engines are becoming no better than content scrapers that steal content from their websites. Weigh this concern against the benefits of having enhanced listings on search engines.  As SEOs, we continue to lean toward doing everything we can to feed quality information to search engines, but a dose of healthy skepticism never hurt anyone.

Readers: are you using rich snippets?  Have you seen more traffic, or are you concerned about search engines scraping your content?

Q: We have tons of great videos on our website. How can we get those little video thumbnails that sometimes show up in Google search results?

A: When Google understands that there is a video presented on a web page, it will often display a thumbnail as part of the search result for that page, as shown here:

(The exact layout of your results may vary, since Google is constantly tweaking its video result display.)

For the best chance at getting video thumbnails for your pages, you should embed videos on your site in a way that Google can easily recognize AND send Google as many signals as possible indicating that you have a video on the page. Here are some tips that can help:

• Include each video on a separate URL
• Embed your video in Quicktime or a standard Flash video player.  Google can’t crawl through a complicated Flash file, so if you use Flash, so we suggest using the built-in Flash video component for FLV playback using a hard coded URL path to the FLV streaming source (it’s OK if this is an off-domain location).
• Create a Google Video Sitemap and submit it to Google Webmaster Tools: click here for Google’s video sitemap instructions.
• To add even more signals that Google can read, tag the videos using RDFa: click here for Google’s video RDFa instructions.
• Use Video OG tags.  These are intended for social media, but Google sees them too, and we’ve seen clear evidence that Google will recognize video information in OG tags. See more about how to use Facebook OG tags.
• It is OK to host your videos on streaming service such as Brightcove.  However, you will probably be on your own when it comes to generating a video sitemap or well-formatted OG tags.

Video SEO is changing fast! About two years ago, video thumbnails were dominated by YouTube and Metacafe.  These days, YouTube still has a dominant role, but other domains are presented among the featured thumbnails more and more.  Watch this blog for updates in this quick-moving space.

Q: I am a Hollywood agent and we are not happy with some of the images that Google shows when you search for names of some of our clients.  Where does Google get these images and how can I request that they remove them?

A: We’ll use the search results for the actor Paul Rudd as an example.  Here is a screenshot showing the image thumbnails on Google  when you search for the phrase “paul rudd”:

In this example, the image results are sourced in a variety of sites, including wikipedia.com, exposay.com, thehunkies.com, movieeye.com, biographyworld.net and so on.  You can see where the images come from by rolling over each thumbnail and looking at the URL.

If there are offensive or problematic images in your clients’ results, here are some approaches you can take to improving them:

• Compete for these positions with your own images on your own website. Gaining a position among these thumbnails  requires that your site compete within image listings in the same way that webpages compete for web search rankings.
• Contact the owners of the site that posted the offending images, and request that the images be taken down. (This is sometimes called a “takedown notice”).  You can challenge them based on copyright violations or other applicable violations.
• Google will only remove listings in extremely limited circumstances: if another site publishes confidential information such as social security number or bank account number.  See Google’s help page on this topic.

If you decide to compete for these positions, here are some basic image ranking factors that will help your website’s images rank well:

• Images must be accessible to the search engines.  Make sure you aren’t disallowing search engines from indexing your images with the robots.txt file.
• Images filenames should contain target keywords.  In this example, a filename such as paulrudd or paul-rudd.jpg would be a good idea
• Image captions and other text on the page should contain target keywords
• The images should be placed on individual pages – one page for each image – and there must be links pointing to these pages
• A domain with a high level of authority, based on overall inbound links from high-quality websites, will have a better chance of gaining high ranks in image search .

We hope it goes without saying that getting a high rank for a celebrity name search is going to be highly competitive.  Rather than reinventing the wheel, you may want to simply work on making sure that your clients’ preferred pics are nicely distributed to as many legit webmasters as possible.  That way, the other sites will do the job for you!

Q: I am optimizing a site that has content filled with search terms but a lot of the content is in a wordpress document that is password protected for the user.  Is that content still searchable?

A:  Pages that are password-protected will not be accessible to the search engines. Search engine robots typically can’t log in to crawl pages, so content behind a login will not make it into the search index.

When possible, we recommend creating teaser pages for password-protected materials.  On the teaser page, the site owner can include a title and brief description of the content that will be accessed behind the password.  This teaser content will be accessible by search engines, allowing them to index this limited version of your content.  We also believe that teaser content is helpful for the user experience.

There is one available workaround: A prominent example of hidden content is when news organizations, such as the New York Times and the Wall Street Journal, require a login in order to see their articles.  These organizations have set up a “first click free” agreement with Google, which allows people to click on a search engine listing and view a single article without logging in.  Click to read more on Google’s First Click Free program.