Let’s start. There are 3 ways to check keyword search volume with Google: broad, phrase, and exact match. What the hell is the difference? And which one should you be checking search volume with?

Broad Match

Broad match is the default setting when using the Keyword Tool. What broad match means is that any time any part of the keyword shows up in a search phrase, that data gets added to the reported total searches per month. For instance, if the keyword is “sunflower seeds”, then any search with either “sunflower” or “seeds” or “sunflower seeds” in it will be added into the total volume. So the actual search terms “sunflower seeds” might only get 100 searches, while “seeds” gets 2000 searches and “fruit seeds” gets 600 searches.

So broad match search volume is an aggregate of all searches involving any combination of any of the keywords as well as other words you didn’t account for. It’s an inflated number when estimating search volume.

Phrase Match

Phrase match keyword search volume works like this: if you check phrase match search volume for “sunflower seeds” then the number returned will include all traffic for the main phrase as well as phrases like “huge sunflower seeds” and “sunflower guide for seeds”. The number returned is the aggregate of all search phrases that must include the entire set of main keywords, no matter if they are split up by other words or not. This number is definitely more accurate than broad match, but can still be slightly inflated.

Exact Match

Finally, there is exact match keyword data. The volume of searches returned when checking exact match keywords is simply for only that single keyphrase. So there are no aggregates involved, it’s only the total number of searches for just that exact phrase, typed exactly into Google, with no other words included. This is the most accurate number to reference when researching niches, as it doesn’t aggregate traffic you might not actually rank for and get. You should always check your search volumes with this setting on.

Finishing Up

The point of this post is that it is easy to be deceived by broad and phrase match data, so you should use exact match data when searching for niches. When trying to get ranked for certain keywords, you cannot guarantee that you will be able to rank for all variations and combinations of keywords, so the most realistic estimate of possible traffic is the exact match search volume data.

I’ll post more this weekend, I have some cool ideas in the works.

Til then,

D

Related posts:

1. The Power of Micro Niche Sites with Exact Match Domain Names
2. Interpreting the Market Samurai SEO Competition Module Data

The script is designed for anyone that has a lot of different SEO sites and is constantly checking their traffic. It is also built around Google Analytics so that is definitely a pre-requisite for use. I need to finish up a few functions and then style it up a bit to make it nice and clean, but as soon as that is done, I’ll post it up here along with some installation instructions (it’s basic, no worries).

Til then, peace,

D

No related posts.

Of course, that means it nuked my Ruby install as well as my local dev environment. I really should have scripts written to automatically do all this crap when I need to, but since it’s only every few months, I don’t care that much. I also don’t care enough to remember all of these commands, so I just have a bookmarked list of guides that I follow when I need to.

So without further adieu, here’s what I just did to install a LAMP server, PHPMyAdmin, and configure local domains, to make local development easy as pie.

Installing LAMP

So Ubuntu makes this really fucking easy to do. All you need to do is open up a terminal and type the command:

and then select LAMP and follow the prompts. You can read a more through guide here

Installing PHPMyAdmin

Some purists might complain about abstracting into a MySQL manager tool, but I don’t care. This thing makes it easy to play around with database tables and get things rocking initially. It’s a good idea to install locally to save some of your sanity. This is an easy install as well, so just follow this guide and you’ll be good (start with the Installed PHPMyAdmin section).

Setting Up Local Domains

So this is something a lot of people don’t do, which surprises me. Most people work off of subfolders of their localhost directory, so like http://localhost/project1. That isn’t a good representation of your final deployment environment, which will probably be at the root level. So a good idea is to setup local domains that allow you to use something like http://project1.local as your address. It’s a good idea to use something other than .com/.net/.org/.whatever (any real tld’s) so that you don’t unintentionally block yourself from the real domain down the road. I use .local for my projects.

What this does is tell your system to route whatever local domains you have to your own IP address. Makes life easy. Follow this guide to get them all setup.

One other note, I like to keep my dev sites in a www folder in my account directory (so /home/username/www). This keeps everything under one location and makes it easy to back things up when it all gets nuked from a botched install

Finishing Up

This was a quick little post, I know. I just wanted to get all of this stuff in one place, and I know one or two of you will find it useful. I’m going to go into some more details about my development tips and tricks soon, and I’m tossing around the idea of doing a screencast of me doing a slice and dice job on a Photoshop design and the process of making it a functional Wordpress site.

Let me know if that sounds interesting to anyone. I’m going to go get some fresh air for a bit now because it’s amazing outside, but I’ll be back later to write another post that continues on with the Ultimate Niche Product Sites Guide tonight.

Easy,

D

No related posts.

So around the end of March, I resurrected this old blog domain of mine and ripped out all of the old posts and decided to start fresh with a whole new set of topics and posts. Since then, things have gone pretty well. I got ridiculous traffic from that post I did on the simple Watir bot, which was sweet. I definitely need to man up and write up some more posts on that topic, people were obviously very interested in it. After that post, things settled down quite a bit, and I’m back to about 75-100 visitors a day pretty consistently. I’m really happy about that, blogging without readers is always difficult

So enough with the babbling, here are my analytics screenshots for the first month of Layered Thoughts:

Traffic Overview

As you can see in there, I got a little over 4000 visits, about 6500 page views, and an average time on site of a minute and a half. Not too bad. You can also see the traffic breakdown per page, as well as the splits amongst my referrers. Search traffic actually makes up a small portion of my traffic this month, all because of that post sending a huge amount of traffic through Hacker News, Wickedfire, and random other sites. I would imagine my search traffic will be a much larger percentage this month.

That Watir Post got me just under 1000 visitors (980, damn it) the day after it was posted.

Content Stats

Standard stuff…watir post was number 1, my main url #2, and the niche product sites guide #3. The last two are most definitely because of my forum signature at Wickedfire.

Keyword Stats

This data is fun. It’s always interesting to look at the keywords people find a higher traffic site with. You get an idea of all of the stuff you never would have seen/planned for when doing keyword research. I have a whole post planned about my rankings for the keywords “web bots”. I’m also planning on doing some actual link building down the road to try and get up at the top for those keywords (I’m currently at around #11-13). I’ll case study that stuff for sure.

Visitor Stats

Obviously a savvy crowd reads this site, as Internet Explorer makes up 4.71% of the browsers used. That is not a normal number for most sites, it is usually much higher. But this data is typical of nerdy sites

Subscription Stats

49 subscribers is pretty awesome. Anyone who wants to seriously blog should focus on his subscribers count, as this number is a count of people who will immediately see when I post without having to check back on the site. Good stuff, thanks to everyone who subscribed!

Coming Up

I’m going to do a post like this every month, because I know that when I was getting started with all of this stuff, I always wanted to see numbers and graphs and real data.

Coming up, I have a few random posts I want to write as well as I’m planning to finish up the niche product sites guide. There are some great topics coming up with that guide, including keyword research, content creation, and link building, so definitely stay tuned, it’s going to get fun.

Like I said in the beginning of this post, I’m going to try and post every other day in May. Posting every day is too cliche . I also want to try and keep pushing this blog further and start getting more and more traffic for it to see what level I can get it to.

Finally, I’m planning on doing either some interviews with some of the more popular IM/SEO guys or some guest blog posts from there, or a mix of both, so if there is anyone you want to see talk here or anything you want me to ask people, let me know in the comments section.

Til next time,

D

No related posts.

Finding Niches

Go to WalMart. Seriously. Just go to WalMart and look around. Everything in that store can probably be built as its own niche site, targeted around specific keywords. For example, take sewing.

Sewing is a topic that is nearly entirely targeted at women, and generally, adult women. This is a good thing. These are people who are a good demographic to target for making online purchases. So fire up your keyword tool and just start off by typing in “sewing” and see what comes up.

You want to look for long tail keywords that get pretty solid traffic. You then want to switch down to exact match and make sure that that traffic still holds up. If it does, you’re looking good.

Now, you’re going to want to identify a good amount of solid keywords from this broad keyword. These will be important for the next step, which is choosing a keyword.

What Are The Best Niches?

The niches that you know something about. The site I’m building and writing this guide around is a topic that I know a good amount about. I’m not an expert, but I know enough to have a voice on the topic.

This is important for me, because I’m going to be publishing and selling my own guide on the site. If I didn’t know a damn thing about the niche topic, I could still fake it and probably do ok, but you’re in a much safer position if you do actually understand the topic. Plus, and this is especially important when you’re bootstrapping early on, if you know about the topic, writing your own content is a hell of a lot easier.

Oh Yeah, Target People Who Can Have Credit Cards

Kids suck for this stuff, so don’t go after kid keywords. If you can monetize in other ways, these sites might work for kids, but for the most part, target adults in niches where they normally pay money to participate anyways. People who sew don’t sew for free, so chances are you can get them to buy something with the right targeting.

I know that’s a simple reminder, but it’s important.

Go Forth And Make Keyword Lists

So go punch in your broad keywords into your keyword research tools and generate some keyword lists. Use the criteria I outlined from my last past about filtering out the low traffic volume keywords, then follow the guidelines in my next post about what to look for with competition. Then look and see what the keywords look like for creating or selling a product.

That’s pretty much it. It really is a simple topic. Just look around at a store and you can find niches. The keyword research takes intelligence and effort though, but I’ll go over specific details in the next post that will outline exactly what you need to do to succeed.

Good luck and good night,

D

Related posts:

1. Goals, Keywords, & Some Specifics About Our Niche Product Site
2. The Difference Between Broad, Phrase, and Exact Match Search Volume Data

I’m going to start with the overarching goals and expectations you can expect from these kinds of sites, and then go into the details of how we’re going to get there, and what we can do once we get there. These topics will all be expanded on in detail in the upcoming posts in this series.

What We (Hopefully) Want To Get From These Micro Niche Product Sites (Hopefully)

$10 a day. That’s it. Of course, more is always better, but at the very least, $10/day. Now, we’re not going to be doing this with Adsense or CPA, this will be with selling our own products. So I’ve priced the product I’m going to be selling at $9.99. So that means I need one conversion a day to reach my goal.

How much traffic do I need for one conversion a day? No fuckin idea. BUT, until I have real data, I’m going to say that I’m going to get a 1% conversion rate. So for every 100 unique visitors to my site, I’ll sell one copy of my product. This could be really high, really low, or just right, I really have no idea yet. We’ll find out soon enough though.

So if we know we are shooting for a 1% conversion rate, that leads us perfectly into search volume goals for our target keywords.

Click Through Rate Matters

A few years back, AOL released a shit ton of search data. From that data, people were able to calculate the average Click Through Rate for each position on the front page of the SERPS. So using the graph on the left, if you rank number 1 for a keyword that gets 100 exact match searches a day, you can expect about 42 visitors. If you’re number 2, that drops down to about 12 visitors.

Now obviously, these numbers are dependent on a lot of variables. If you have more compelling meta data than people above you, you might snag a few more clicks. But for the most part, this is spot on.

So what does that mean for us? It means that we can work backwards and find the minimum daily searches we want to shoot for to make $10 a day from our sites.

Let’s Calculate Our Minimum Monthly Searches We Need

Yay for math time. Here we go.

We need 100 unique visitors a day (not pageviews, actual visitors). Knowing that, and knowing the average CTR of the front page of the SERPS, that means that if we rank #1, we will get 42 visitors for every 100 searches.

So to get to 100 uniques a day, we would need to rank #1 for a keyword that gets about 240 exact searches per day.

(Remember Algebra?)

Ok, so that’s all fine and dandy, but what about if you can’t get to #1? Well, then you’re going to need a keyword that gets about 850 searches a day. That’s definitely doable if you dig deep enough, but it’s significantly harder than if you ranked #1.

Now this is all assuming there is one single keyword that you could ever receive traffic for. The beauty of SEO is that you get ranked for all of the related keywords as well, so you’ll oftentimes get a lot of traffic that you weren’t even expecting. But for simplicity’s sake, we’re just going to go with 240 as the minimum searches a day that we’re shooting for.

I’m Cheating For This Though

The keyword I’m basing this series off of actually only gets about 150 exact searches a day. It’s not that big of a deal, I just need to either a) raise my conversion rate or b) get more traffic. Or else I can expect to make $10 every other day.

You’re not always going to find the right domains/right keywords/right volume. Cut corners and be smart and you can make up for it in other ways.

What’s the Site Going to Be Like?

So for this project and site, I’ve built out a long landing page style home page, with about 3000 words of content on it, all highly optimized. I’ve also added about, privacy, and contact us pages, that are all linked to in the footer. I’m also going to include about 4 or 5 articles that are linked to throughout the page to increase the content site wide.

This site is fully geared towards sales, so it has large calls to actions and is very optimized. I’ll be posting a wireframe of where I put things on the layout so you guys can get an idea of how it looks.

I’m going to be using Buy Now buttons that go directly to a PayPal Buying page. Quick and easy, and well trusted by mostly everyone online.

Where’s the Traffic Going to Come From?

Organic SEO all the way. Standard link building rules apply here, I’m going to utilise a variety of sources and shoot for very high quality links at all times. Here’s an overview of what I’ll be using:

• Social Bookmarks, from red_virus at Wickedfire. This guy is just damn amazing, does great work
• Link wheel with full bookmarking, RSS submissions, etc
• Profile Links
• Lots and lots of links to each hub of the link wheel (if you catch my drift)
• Really high quality blog comments

The blog comments stand to give the biggest boost. I have a system down where I can scrape a huge amount of blogs, then filter the list down to High Page Rank, Low Out Bound Link Blogs. Those are the perfect targets for getting comments on, so in order to make them work as best as possible, I’ll be manually posting comments. Probably about 10-20 a day, depending on how many blogs I can scrape total. I’ll cover all this later.

What’s Getting Sold?

A 20 or so page book. It’s a how to guide, something that women/moms will probably enjoy a lot. It will delivered as a PDF with their name and email stamped on the bottom so they are afraid to share it with friends. It is actually very high quality and probably pretty valuable to people who want to learn how to do what it is about.

I designed a 3d cover for it using a template I found online, and am just formatting it in Word. Basic formatting, table of contents, nice cover, border, image on each page. Nice big colorful fonts and pages. Easy stuff, like you’re writing a report for school.

Coming Up Next

Alright, so I need to figure out how to nicely tie together all of this posts for easy browsing. All blog post series plugins suck, so I’ll hack something together later to use.

Next I’m going to write about how to pick a niche. After that, we’ll do some keyword and competition research. Then it’s on to how to build the site and how to get it ranked high.

Til next time,

D

Related posts:

1. How to Choose a Niche For Your Niche Product Site
2. The Difference Between Broad, Phrase, and Exact Match Search Volume Data

Why are you telling people about this?

Honestly, because I like writing about how to do things. I think a lot of things that people wonder about will get covered in this, and I know that most people, like myself, are always dying for case studies instead of hypothetical numbers. So this will hopefully serve as a good place for people to reference when getting started with these types of sites.

No seriously. Why are you telling people about this?

Alright, you got me. There’s honestly two reasons why I felt compelled to put together this guide. The first reason is that I’ve been meaning to kick myself in the ass and get a bunch of these sites rolled out anyways, so what better way to get moving than to write a damn guide about them. 99% of the reasons why people fail at Internet Marketing is simply because of execution. You can be the smartest guy in the world when it comes to this stuff, but if you simply don’t build and test and try, you’ll never make a dime.

Secondly, and this goes along with the last point, is that I’m really not too worried about a bunch of people leeching these ideas and beating me to the punch. Nothing I’m about to write about is ground breaking in any way whatsoever, all I’m really doing is putting a bunch of information in one place and adding in my own knowledge where I can. An industrious individual could find everything I’m about to write in an afternoon of searching.

What is a Niche Product Site?

Niche product sites are simply sites that are optimized around a certain product/service/offer and are pretty limited in scope. These are not portal sites and are not really trying to rank for more than a few keywords at the most (oh yeah, did I mention this was an SEO case study?).

Generally with these sites, you are searching for a high search volume exact match keyword domain name. That’s right, a high search volume exact match keyword domain name. These are the diamonds in the rough of this type of site. I’ll go into the details of what constitutes one of those domains and how to go about finding them in later posts in this series.

When it comes to the product you’re trying to sell, this can be a number of things. The main things you can promote are:

• Affiliate Offers
• Third party products (wholesale or drop sale)
• Your own products

For this guide, I will covering how to sell your own products. I won’t go too much into the ins and outs of how to create your own product in this guide per se, but I will cover how to sell digital products, what to look for in content and creation of the product, and how not to create a bullshit product that will get you flamed on all of the boards.

Dude. Really. An eBook?

Yeah, a damn ebook. For these sites, I will not be creating bullshit “Hey, you don’t have a degree unlike all of the PHD’s at Google, but we still want you to work for us!” Make Money Online offers. These are going to be specific guides and books about certain niche markets that will add value to the market and be useful to the people who buy them. I’ll cover this in more detail later as well.

What Next?

I have an internal list going on right now in terms of planning out the next few posts. This whole series will probably span a dozen or so posts, each one hopefully covering what you guys care about in as much depth as possible. In order to prevent any data skewing, I will not be outing my niche for this guide. Maybe one day.

I am using this write up as a proof of concept for about a half dozen other sites that I have domains for and am looking to build out. Once you get the process down and dial in your work ethic, you can seriously pump these out. We’ll see what I can pull off.

No related posts.

What I’m going to go over here is how truly effective an exact match keyword domain name can be when used on simple micro niche sites. I guess I’ll first go into what both of those concepts mean, then I’ll jump into the subject of this analysis.

What is an Exact Match Domain Name?

This is simply a domain name that has the EXACT keyword that you want to target in it. That’s it. No extras “the” or “at” or extra words at the beginning or end of it. Simply the exact keyword someone would type in at Google to come to your site, without the spaces. Technically, hyphens don’t count either, but sometimes domains with them can work out well.

So if you wanted to rank for “sunflower seeds”, having the domain www.sunflowerseeds.com, .org, or .net would be the exact match domain name. Something like www.sunflowerseedsstore.com wouldn’t cut it.

You might have seen people talk about the advantages of an exact match domain name around different forums and blog posts before, so this might not come as a surprise, but having your exact keywords in your domain name can give you a HUGE boost in the SERPS. Google eats up sites like this, and in low to moderate competition niches, you can pretty easily rank on the front page with minimal backlinking. Additionally, if you see an exact match domain name in the top ten, you better be prepared to really beat them in terms of linkbuilding and on page SEO, because they will have an inherent advantage over you because of the keywords in their domain.

What is a Micro Niche Site?

The best way to describe a micro niche site is that it is a site that is targeting a market with moderate monthly searches, super low allintitle competition,low to medium front page competition, and generally long tail keywords. These sites will often be very small, sometimes just 1-4 pages, and are usually about just a single product/list/offer.

You generally don’t need an active blog or a bunch of content on these kinds of sites, as the competition is so low that it doesn’t take much to rank for the keywords. The real trick to these types of sites is that they are somewhat hard to find. Often times, long tail keywords won’t have enough monthly searches to really be worth pursuing with a site. So you really have to dig in there and find a few keywords that have low enough competition coupled with enough monthly searches to convert to a point that’s worth your time.

Here are some rough numbers I look for when I research micro niches:

• allintitle competition of less than 10-30,000
• Front Page Competition with average PR of 3 or less
• Low backlinks (<3-500 max) for each front page site
• At least 5000 searches a month
• Exact match domain name available for .com, .org. or .net

Now, techie readers will realise that that all of those things can be found out basically automatically with the right keyword lists and some know how. But that will be something I probably won’t go into too much here at least

Analysis of Micro Niche Site

So recently, this interview was posted to Hacker News. In the interview, Patrick McKenzie, the creator of Bingo Card Creator, goes into some pretty great details about what I consider “active white hat SEO”. He’s not doing anything shady, or actively building any links for that matter, but he is paying attention to his SEO and encouraging others to build links to his sites. Additionally, he’s utilising micro niche sites with exact match domains to promote his seasonal bingo cards, with a bit of success it seems. He says in the interview:

I don’t want to say that microsites are the right solution for everyone.  The problem with microsites is that they cause you to split your domain equity and link equity.  The basic idea behind PageRank (the core of Google’s original search ranking algorithm) is that any link to a page is a vote that that page and site is trusted and should appear higher in the search engine rankings.  Separate from PageRank is what is called the exact match domain bonus — if your domain (.net, .com, .org, or country-level TLD like .co.uk if the searcher is in the UK) exactly matches — and I mean character-for-character match when you strip out the spaces — you get a super bonus.

For example, if you registered patricksfunnydomain.com , and tried to rank for [Easter Bingo Cards], you might need hundreds of links or dozens of links until you’re competitive for that search result.  f I was putting it on Bingo Card Creator, which has hundreds of links, it would probably rank on the top page but near the bottom of it, until I built a significant amount of links directly to that page.  The exact match bonus causes easterbingocards.com to rank in the top three with a handful of links in a matter of weeks — a link from my blog, a few from directories, that is all it takes really.

There’s a lot of gems in what he’s saying right there. Let’s go into one of his micro sites.

Examination of a Micro Site

Here’s one of Patrick’s micro sites that has an exact match domain:

Really basic site, has a theme that matches the topic, a few content pages (home + 4 subpages) and a hell of a lot of links to his money site (bingocardcreator.com). The entire point of this site is to act simply as a landing page, capturing visitors who were searching for “halloween bingo cards”, and sending them to his actual bingo card site to get them to buy cards.

What he’s doing here though is leveraging the fact that he owns www.halloweenbingocards.net and because of that ranks #1 for that keyword. So he’s getting a bunch of clicks without doing much of anything. Some simple linkbuilding and that domain name, and he made it to the top.

I want to go over some of the numbers behind his site to show you what I mean in terms of a micro site and how effective the exact match domain name can be in terms of ranking. Here’s the keyword research data (I know, I haven’t gone over this topic yet for Market Samurai, I will next week probably):

About 120 searches a day, with an allintitle (SEOTC) competition of just 1,930. Those search numbers are a little low, but they are averaged out throughout the year. Searches in Halloween time would probably be significantly higher (I believe he said he got somewhere around 30,000 visitors during that week).

Next is the front page competition data:

This is where you can truly see how effective an exact match domain name is. As you can see, his site is firmly at #1, while his stats are pretty much terrible compared to his competitors. He has the lowest domain backlinks of everyone, pretty weak page backlinks, moderate Page Rank, and doesn’t even have his keyword in his meta description.

But none of that matters because he has an exact match keyword domain name. This is a perfect example of why these sites work so well. Just look at the PR spread of his backlinks (all 57 of them):

He only has 25 PR 1 or greater backlinks and yet is rock solid at the number 1 spot. You could sneeze and get that many backlinks in a day.

His on page setup is good, take a look:

You can see his halloween bingo cards keyword in each of his header tags as well as his meta title (not description though). He has a few subpages, but honestly, he probably doesn’t even need those.

Finishing Up

So nothing truly earth shattering here, just a really good example of how to leverage your exact match domain name to rank high and fast. I have a few plans to further pursue this topic, both from the research side of things as well as from the case study side, so I’ll try and keep this topic fresh as we move forward.

I’m planning on launching a one page (well technically, it will have About, Privacy, and Contact pages) site over the weekend to sell a little info product (not a bullshit make money online ebook, something that has nothing to do with the web and that is well written and pretty useful to people who care). I have an exact match .org domain and the site design/lander ready, and am now just finishing up the content.

My plan is to pretty thoroughly document the linkbuilding I do and the rankings that follow from them. I’m going to be using a mix of a few popular techniques so it should be interesting.I’ll try and be as candid as possible without giving away the niche. I also might (I stress the might here) do a bit of Facebook Advertising for this site as there is a specific demographic I’m targeting that I can pretty cheaply it there.

I’m going to start with organic traffic though.

Stay tuned.

D

Related posts:

1. The Difference Between Broad, Phrase, and Exact Match Search Volume Data
2. Goals, Keywords, & Some Specifics About Our Niche Product Site
3. Interpreting the Market Samurai SEO Competition Module Data

Why Ruby? Why Watir?

For anyone who has had the joy of switching to Ruby from other languages, this question should be a no brainer. Ruby is a fun language, it’s clear and concise, reads like English, and has a great collection of plugins (gems) that allow you to significantly expand the capabilities of your apps. Ruby obviously came to fame because of Rails, which is a really nice and powerful framework, but is a whole different beast from web bots. Ruby is a great general purpose language that can be used for a lot of different projects.

As for Watir, here’s the basic run down. Watir is a browser based web app front end testing tool. Automated testing is a big deal for guys who work on large code bases, but when it comes to doing assertions on the visual side of things, there’s no substitution for simulating a browser. What Watir allows you to do is directly control your Firefox (or Chrome, Safari, or IE even) through Ruby code to simulate a user on a site to make sure your site doesn’t break down if a user clicks a button 10,000 times for instance.

So Watir exists for noble causes, but there are obviously other ways you can utilise the power that it gives you. Admittedly, it’s not the fastest botting solution out there, cause Firefox is heavy as hell. Also, running concurrent (threaded) bots becomes difficult because of Firefox’s heft as well. But no other solution allows you to so quickly and easily get up and running with a fully functional, robust bot. Plus, you can watch the thing run as you’re testing in your browser, so you can see exactly what you need to do next. It’s sweet.

Let’s Get Started

Ok, so there’s a couple things you need to do to prepare to write bots with Watir. First, you need to get Ruby installed. On Windows, there are things like the One Click installer or just the actual Ruby installer. For Linux, like Ubuntu (what I’m running), you can use apt-get to install everything you need. I’m not going to go over how to install everything, I’m just going to point you to Watir’s installation page: Install Watir

Next, you should install Firefox. You could technically run this bot in any browser, but there are a couple of plugins that make this a lot easier and you can only use those in Firefox. So go do it.

Once Firefox is installed, you need to get the JSSH Plugin installed as well. The instructions for that are on the Install Watir page as well.

Next, you need to install those Firefox plugins. Here are the ones that we’ll use in this guide:

• Test-Wise Recorder – This plugin is amazing. It records your interactions with a site and spits out Ruby Watir code. It’s not 100% all of the time, but it is a damn good starting point.
• Firebug – If you want to write web bots but don’t know what Firebug is, just, uhhh, come on. Go get it. Go love it.
• FireXPath – This lets you right click any item on the page and grab the exact XPath string to access it through code. It is super handy for a lot of these projects.

Finally, you need to install the Ruby Gem Nokogiri. Your best bet is to actually install Mechanize, because Mechanize comes with Nokogiri, and you’re going to need both of them eventually, so you might as well. Should be as simple as:

If that doesn’t work right, fire up Google and get it to work.

Party Time

So what are we going to build? There are obviously a lot of dubious projects that you can do with the tools described in this guide, but for now, I’m going to keep things nice and white hat. So today, we’re going to build a bot that will automatically scrape the forecast for a location 7 days from today. I know, super useful.

So anywho, fire up your browser and go to weather.com. You should see something like this:

Now, when botting, you want to run through (often times, a lot) the process to get a feel for exactly what you’re going to do. So go ahead and type in your location in the search box up top (I did San Diego, CA for this guide). You should now see this:

Ok, so to get the forecast for a week from today, you need to get over to the “10-Day” page, so click that link on the blue menu bar:

Finally, what we want is the 8th row of that table (it includes today, so 7 + 1 = simple math). That box looks like this:

And what we want to pull out from that box is the text underneath the rainy clouds, Showers.

Seems easy enough, right? Let’s get started.

Writing The Bot

So now, go back to weather.com’s home page, and fire up “iTest2 Recorder Sidebar” in Firefox. A menu will pop up on the left side of the screen, that looks like this:

So what we want to do is click the Watir tab on that, then start interacting with the website. As you click around, enter form fields, etc, it records what you do and spits out Ruby code. Really cool.

So go ahead and click into the search box, and type in that location you used when we were exploring the site earlier. Then click “Search”.

Now, click on the “10-Day” link in the blue menu bar. Your iTest Panel should now have this code in it:

Now, right click in that box, select “Copy All”, and fire up your text editor and paste that code into there. You can close up iTest now.

Next, we’re going to get the appropriate require’s into your code so we can run this initial script. At the top of your ruby file, add these lines:

All we’re doing there is getting the right gems loaded up, then making sure that when Watir starts running, it uses Firefox by default. Your code should now look like this:

Save that file out as weather.rb, then drop down into the command line/terminal and navigate to where you saved the file and run it by entering:

If everything was setup correctly, you should see Firefox pop up, visit weather.com, type in your location, then click the 10-Day link. Sweet!

Scraping the Forecast with Nokogiri and XPath

So now that we’re on the right page, we need to get that specific forecast for one week from today. In Firefox, right click the “Showers” forecast in the table row for 7 days from today (in this case, it’s April 11th) and click “Inspect XPath”.  You should see this:

Copy the “.//*[@id='tenDay']/div[8]/div/div[2]/div/p” code out and paste it into your weather.rb file.

So now we’re going to load the current page into Nokogiri, and using the .xpath method, parse out this row of the table and get the text within that <p> tag.

Just under your “require ‘watir’” line, you need to add this:

Next, after the current last line of your script (the line that gets us onto the 10-Day forecase page), you’re going to add this:

What the hell is that? Well, the first line is instantiating the variable “page_html” and setting it to the result from parsing the current page (accessible via browser.html) with Nokogiri. The current page for Watir is always stored in browser.html, fyi. What this is basically doing is getting it locked and loaded in Nokogiri so you can parse out the relevant information from the DOM that you want.

The next line is printing to the screen (that’s what “puts” does, it’s like “echo” in PHP) the inner_text of whatever is located at the XPath location that we copied from FireXPath earlier. Let me break that down a bit.

I’m not going into what XPath is here, but it’s basically a quick way to traverse the DOM. What this XPath says is “go through the whole page, find any element with the id of ‘tenDay’, find the 8th div inside of that, find the first div inside of that, the 2nd div inside of that, the first div inside of that, then the first p inside of that”. If we left out the .inner_text call, this would spit out all of the html code inside of that <p> tag.

But by adding .inner_text, you are telling Nokogiri to ignore all of the HTML in that <p> tag and just print out whatever text is located in there, which in this case is “Showers”.

So that’s basically it. Your final code should look like this:

Run that with “ruby weather.rb” again, and it should run and stop on the 10-Day forecast page. Check your terminal, and it should print out “Showers”.

Boom. You just botted Weather.com.

Where To Go From Here?

Ok, so I’ll admit it, that was a really simple and White Hat bot. But, there is A LOT of potential with what I just wrote up if you take some time to plan out some good targets. So start tweaking this code and try botting some other sites. Once you’re comfortable, start thinking about how you can expand your efforts.

I want to write some more guides like this for different efforts, but I probably won’t ever write anything very Black Hat just because that’s a job for you to figure out anyways . There are a few other useful topics revolving around these concepts though that I will write about soon, so stay tuned.

Good luck,

D

No related posts.

According to SEOMoz, having your keywords in your meta title (and description) is the 4th most important thing you can do to help with your rankings. This is something I definitely agree with, but there is also the human side of why you should focus on these elements that I really want to cover. Your meta tags are your mini sales force on Google, and you need more than just keywords to get increased traffic.

First, just to clarify things, your meta title and description are what show up on the Google Search Results pages. Your meta title is the blue link, your description is the block of text underneath. For this post, I’ll continue on with the sunflower seeds keyword from before.

Here’s an example of a meta title and description in use on the SERPS for that keyword:

Nothing crazy or new here, the meta title is the blue hyperlink, and the meta description is the short block of black text underneath.

So what’s the big deal?

There’s two main things that you need to focus on with your meta information.

1. As has been mentioned above, is that you must absolutely have your exact keyword in both your title and description. That doesn’t mean you can have all the right words there, in different order or with words in between. It needs to be exactly what you’re targeting if someone were to type it into Google. Sometimes that can be tricky to form a coherent phrase around, but put your brain into it and you’ll get something.
2. There are humans seeing (and presumably, clicking) on your meta title. There needs to be sales copy in both of those spots. You have a little over 200 characters to convince someone that your link is the best one on the page. I think this is something a lot of marketers forget about and it can definitely start to impact your Click Through Rate if you neglect it too much.

Ok, so the two big main points for these elements are out in the open. Now what?

Meta Titles and Descriptions from the SEO Side of Things

There’s no need to go into too much detail here, it’s relatively simple. I’ll just lay out some bullet points for clarity though:

• Use your exact-match keyword
• Meta titles should be no more than 67 characters long (Search Engines won’t show more than that)
• Meta Descriptions should be no more than 160 characters long
• You should never use a meta title or description more than one time throughout your site. Every page needs it own unique set of meta information. There will be overlap for keywords and content, but you should never have duplicates

Meta Titles and Descriptions from the Sales Side of Things

Now, from the sales side, some other considerations need to be taken into account. Most people fail to realize that when you’re targeting Organic SEO Traffic (meaning, the search engines are sending you visitors), your Meta Title and Description are the single thing standing between you and a new user. User’s don’t care that you have the exact keyword in the title and description (other than for checking the relevance of the result, at least), they will click on what they see that tells them to click.

You absolutely need to focus on making your meta titles and descriptions into Calls To Action. What does that mean? It means making the user want to click on your Search Result hyperlink because of compelling copy in both the title and description.

Your meta title needs to be the driving force. Everyone has been conditioned to read that blue text, so you have 67 characters to convince them to click your link. You need to target action words and draw them in with that title tag.

For your meta description, you need to use it as a support for your call to action meta title. It needs to prop up your meta title and give the user even more of a reason to click through to your site.

Example Time

From that sunflower seeds page, here is an example of a good title and description pair versus a bad one:

You’ll notice in the first title tag (GIANT), they have their company name, the type of product, and a little sales copy (Giant Taste). Their meta description then quickly goes into summing up what the site is and why you would want to go there (to buy sunflower seed products).

For the Bad example, all they have listed in the title is the company name, with no details on why they are the best sunflower seeds company. For their description, this is actually a perfect example of what happens when you don’t list one. Google will try and guess what should go there, which in this case, they chose what seems to be their drop down menu. No CTA support, no reasons given as to why I as a user should care.

How to Put These Concepts into Action While Doing Niche Research

So Google provides a pretty nice site of Search Tools that you can use to get less obvious information about a site. One thing that is useful is the site: operator, which allows you to return only results for that specified site. There are a lot of uses for that, but one of the best ones is to check on the quality of both your own meta titles and descriptions as well as your competitor’s.

When you run a site: check on your competitor’s site, you can pull a few pieces of useful information out. The first is just to get an idea of what kinds of keywords they are targeting across all of their pages. That’s pretty simple and easy to figure out on your own.

But the other thing you can do is much more powerful. When you do that site: search, you get to take a peek at how hard they are trying to optimize their pages for ranking in the Search Engines. If you do that search and you see a bunch of similar or identical meta titles and descriptions, you know that they aren’t trying that hard and you can probably give them a run for their money simply by focusing on making very targeted and effective titles and descriptions.

Here is Bigs.com’s (the Bad example from before) site: search:

As you can see, they seem to be either a) auto generating their meta titles or b) don’t care about them.

Having seen that, you now know that they have a weakness you can attack by writing keyword targeted meta titles and descriptions that are full of compelling ad copy. These things take a little bit of effort to do well, but they can really pay off in the end.

Where to Go From Here

Hopefully this helped explain some less discussed aspects of Meta Titles and Descriptions and make things a bit clearer in terms of what you should focus on to do them correctly. As you start researching niches, start paying attention to these things to try and get a sense for how your potential competitors are focusing their efforts.

PPC Advertisers spend a lot of time and money writing out high quality ad copy, so it’s worth taking a look at what they are doing as well. There’s no reason why you shouldn’t put as much effort into your meta information as they do.

Good luck,

D

No related posts.