A Wordpress Widget is a dynamic element that can be placed in the sidebars (or anywhere that is declared a sidebar) in your Wordpress theme. For example on this blog there is a right sidebar with these widgets: Search It, Recent Entries, Categories, Related Sites, Recent Visitors (by Feedjit), Recent Readers (by MyBlogLog). A widget can be any small modular piece of html (usually a list but not always) that can be stuck into a sidebar.

Widgets Make Your Plugin More Flexible

If you write your plugin as a widget your users will easily be able to move your plugin to which ever sidebar they want and arrange it with their other widgets how they want to. And they can do all of this without touching their theme’s code. This is great for users who don’t know how to modify a Wordpress theme template.

Sidebars Are Not Just Sidebars

Though they are called sidebars, they can actually be any block element of the page. A sidebar could be a heading, a footer, a section within a particular page, anything you want. It’s easy to declare a sidebar. Usually sidebars are declared within a theme’s functions.php file.

Declaring Your Sidebars - From Within A Theme

Update: There’s a great tutorial just on making your theme widget ready over here.
For example in a theme I’m working on I have 3 sidebars, normal left and right sidebars, and a top navigation bar that I’m also using as a dynamic sidebar for widgets. In my functions.php file I have:

register_sidebar(array(
    	'name'=>'top_sidebar',
        'before_widget' => '<li id="%1$s" class="widget %2$s">',
        'after_widget' => '</li>',
        'before_title' => '<h2 class="widgettitle">',
        'after_title' => '</h2>',
    ));
    register_sidebar(array(
    	'name'=>'left_sidebar',
        'before_widget' => '<li id="%1$s" class="widget %2$s">',
        'after_widget' => '</li>',
        'before_title' => '<h2 class="widgettitle">',
        'after_title' => '</h2>',
    ));
    register_sidebar(array(
    	'name'=>'right_sidebar',
        'before_widget' => '<li id="%1$s" class="widget %2$s">',
        'after_widget' => '</li>',
        'before_title' => '<h2 class="widgettitle">',
        'after_title' => '</h2>',
    ));

You can see some basic documentation on register_sidebar here. The name=>top_sidebar’ tells Wordpress what I’m calling my sidebar. This name will be displayed in the Admin interface under Design->Widgets->Current Widgets in the drop down list. You also use this name to place all the sidebar widgets within your them template. Users can choose to add widgets to any of the declared sidebars. The before_widget, after_widget, before_title, and after_title specify some html to wrap the widget in. This is handy if you need to style the widgets in a special way for your theme.

Using Dynamic Sidebars Within Your Theme

To use your sidebars within a theme you just call the dynamic_sidebar function. Here’s how I do it in my theme:

<?php if (function_exists('dynamic_sidebar')) {
dynamic_sidebar('right_sidebar');
} ?>

This tells Wordpress to call the display functions for all the widgets the user has added to the ‘right_sidebar’ sidebar.

A Hook To Hang Your Widget From

You’ll need to tell Wordpress about your plugin’s widget. To do this you declare a “plugins_loaded” action hook like so:

    function mycoolplugin_loaded()
    {
        $widget_ops = array('classname' => 'mycoolplugin_widget', 'description' => "A very cool widget for your sidebar." );
        wp_register_sidebar_widget('mycoolplugin_widget', 'CoolPlugin', 'mycoolplugin_widget', $widget_ops);
    {
    add_action('plugins_loaded','mycoolplugin_loaded');

In your widget you’ll want to replace mycoolplugin with the unique name you’ve chosen for your plugin. And you’ll want to replace “A very cool widget for your sidebar.” with the real description for your widget.

A Note For Uber-Geeks

Your Widget Display Code

In the code above we specified mycoolplugin_widget as the callback function for Wordpress to use when it wants to display your widget. So we’ll need to declare this function and use it to generate our widget’s HTML code.

function mycoolplugin_widget($args)
{
	extract($args); // extracts before_widget,before_title,after_title,after_widget
	echo $before_widget . $before_title . 'CoolPlugin' . $after_title . "<ul>";
	for ($i=0; $i<10; $i++)
	{
		echo "<li>List Item $i</li>";
	}
	echo "</ul>" . $after_widget;
}

My widget simply displays a header of “CoolPlugin” and then a list of ten items. Of course you’ll want to make your widget do something much cooler, but I’ll leave that to you!

Further Reading & Resources

Most of these resources a little out of date for Wordpress 2.5. Most of the screen shots show the admin widget pannel for 2.3 instead of 2.5.

• Widget API Documentation - documents the functions register_sidebar and register_sidebar_widget
• Making New Widgets - Recommended tips for designing plugin widgets.
• Widetizing Your Wordpress Themes
• Worpress API Actions - Documents action hooks like ‘plugins_loaded’ you can use to change Wordpress.
• Wordpress API Filters - Documents all the filters you can use to change Wordpress.
• Wordpress Widget Information - Basic information on widgets.
• Blog Post: Wodpress 2.5 Released - A blog post introducing Wordpress 2.5.
• Makebeta Wordpress Customization - A tutorial I wrote about how to customize Wordpress.

We’ve all seen those websites that show up in the SERPs that are just there because the domain has been around for years and has managed to pick up a few links along the way. They’re listed in dmoz, PR 4 or 5, but the content is really old. What if there was a way to dig through the list of sites in your niche and find the few that been abandoned by their owners? Don’t you think a some of those owners would be willing to sell you their websites far below their true value? They are! You just need to find them.

Old, Authoritative, Ranking Websites Are Waiting For You

I built a tool that will help you find sites you can buy that are old and authoritative in your niche. There are websites in almost every niche that have built up authority over years but have been abandoned by their owners. These absentee owners are often happy to unload their old websites on you for very little. The trick is finding these hidden gems amongst all the active websites. You can spend hours going through directories and SERPs looking for websites that are abandoned or you can use the tool I built to find them quickly and automatically.

Sitefinder301 - Find A Killer Deal On An Old Website.

Sitefinder301 uses the dmoz Open Directory and the Internet Archive to find sites that are relevant to your niche that haven’t been updated in a long time. If you’re lucky you can find a great site with an owner that is willing to sell it for very little.

How It Works

You supply a dmoz category URL and Sitefinder301 examines all the sites that are in it and finds the ones that haven’t been updated in a long time. It gives you PageRank and Whois information on the sites so you can quickly determine if it’s worth buying and who to contact. It’s a quick and easy way to come up with a short list of acquisition targets out of hundreds of possible websites.

Enough Already Where’s The Goods?

Ok here it is, the product of late night php hacking:

Benefiting From Your Acquisition

Here’s a couple ideas for what to do with your new site:

• Put up new content and rank quickly for valuable terms. Build upon the old sites authority and create something of real value.
• Put up a page that promotes your product or service and funnels visitors to your current website.
• Redirect the old site to your current site and benefit from it’s authority and traffic.

Share And Share Alike

Sitefinder301 is open source. You can install it on your SEO tools website or hack it for your own purposes. Download the source here: /makebeta/sitefinder301/sitefinder301code.zip
All I ask is you pass some link friendship back my direction :).

Beware Comparing Floating Point Values Can Be Hazardous!

“The curse of the .0000000001 and .9999999999!”

Here’s the problem. Say you’re comparing two floating point numbers a (71.00) and b (71.00) to see if they are the same. The problem is if you’ve done any calculations to arrive at these numbers they might actually be stored as 71.00000000001. Now if one of them is stored that way and the other isn’t and you compare the two to see if they are equal you’ll get a FALSE as the response, even though they should be the same.

This isn’t a bug, it’s how floating point comparisons are designed to work. It makes it so people writing complex software that requires floating point arithmetic can do what they need to. Unfortunately for the rest of us that usually are just using floating point numbers to represent things like currency (dollars in my case) it really makes life difficult!

PHP Floating Point Comparison Made Easy

Here’s my php function that compares floating point numbers:

function moneycomp($a,$comp,$b,$decimals=2) {
	$res = bccomp($a,$b,$decimals); // php function for comparing floating point numbers with a specified level of precision
	switch ($comp) {
		case ">":
			return ($res==1);
		case ">=":
			return ($res==1 || $res==0);
		case "<":
			return ($res==-1);
		case "<=":
			return ($res==-1 || $res==0);
		default:
		case "==":
			return ($res==0);
	}
}

// example usage: if ($total > $payments) ...
if (moneycomp($total,'>',$payments)) ...

// example usage: if ($debt <= $income) ...
if (moneycomp($debt,'<=',$income) ...

// example usage: if ($cost == $price) ...
if (moneycomp($cost,'==',$price) ...

So that makes life really simple. You just write your comparisons wrapped in the moneycomp() function.

Javascript Floating Point Comparison Made Easy

And here’s my javascript function that compares floating point numbers:

function moneycomp(a,comp,b,decimals) {
	if (!decimals)
		decimals = 2;
	var multiplier = Math.pow(10,decimals);
	a = Math.round(a * multiplier); // multiply to do integer comparison instead of floating point
	b = Math.round(b * multiplier);
	switch (comp) {
		case ">":
			return (a > b);
		case ">=":
			return (a >= b);
		case "<":
			return (a < b);
		case "<=":
			return (a <= b);
		case "==":
			return (a == b);
	}
	return null;
}

// example usage: if (total > payments) ...
if (moneycomp(total,'>',payments)) ...

// example usage: if (debt <= income) ...
if (moneycomp(debt,'<=',income) ...

// example usage: if (cost == price) ...
if (moneycomp(cost,'==',price) ...

Resources And Further Reading

• PHP Floating Point Numbers
• PHP Gotchas!
• PHP Manual: bccomp() function
• Javascript Math Functions
• Mozilla Documentation - Javascript 1.5 Math Object

Thanks for visiting. Stay tuned, I’m about to release a cool SEO tool I’ve been working on and using =).

In this tutorial you will learn how to build a PHP script that scrapes links from any web page.

What You’ll Learn

1. How to use cURL to get the content from a website (URL).
2. Call PHP DOM functions to parse the HTML so you can extract links.
3. Use XPath to grab links from specific parts of a page.
4. Store the scraped links in a MySQL database.
5. Put it all together into a link scraper.
6. What else you could use a scraper for.
7. Legal issues associated with scraping content.

What You Will Need

• Basic knowledge of PHP and MySQL.
• A web server running PHP 5.
• The cURL extension for PHP.
• MySQL - if you want to store the links.

Get The Page Content

cURL is a great tool for making requests to remote servers in PHP. It can imitate a browser in pretty much every way. Here’s the code to grab our target site content:

$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$html = curl_exec($ch);
if (!$html) {
	echo "<br />cURL error number:" .curl_errno($ch);
	echo "<br />cURL error:" . curl_error($ch);
	exit;
}

If the request is successful $html will be filled with the content of $target_url. If the call fails then we’ll see an error message about the failure.

curl_setopt($ch, CURLOPT_URL,$target_url);

This line determines what URL will be requested. For example if you wanted to scrape this site you’d have $target_url = “/makebeta/”. I won’t go into the rest of the options that are set (except for CURLOPT_USERAGENT - see below). You can read an in depth tutorial on PHP and cURL here.

Tip: Fake Your User Agent

Many websites won’t play nice with you if you come knocking with the wrong User Agent string. What’s a User Agent string? It’s part of every request to a web server that tells it what type of agent (browser, spider, etc) is requesting the content. Some websites will give you different content depending on the user agent, so you might want to experiment. You do this in cURL with a call to curl_setopt() with CURLOPT_USERAGENT as the option:

$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);

This would set cURL’s user agent to mimic Google’s. You can find a comprehensive list of user agents here: User Agents.

Common User Agents

I’ve done a bit of the leg work for you and gathered the most common user agents:

Search Engine User Agents

• Google - Googlebot/2.1 ( http://www.googlebot.com/bot.html)
• Google Image - Googlebot-Image/1.0 ( http://www.googlebot.com/bot.html)
• MSN Live - msnbot-Products/1.0 (+http://search.msn.com/msnbot.htm)
• Yahoo - Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
• ask

Browser User Agents

• Firefox (WindowsXP) - Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6
• IE 7 - Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30)
• IE 6 - Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)
• Safari - Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en) AppleWebKit/522.11 (KHTML, like Gecko) Safari/3.0.2
• Opera - Opera/9.00 (Windows NT 5.1; U; en)

Using PHP’s DOM Functions To Parse The HTML

PHP provides with a really cool tool for working with HTML content: DOM Functions. The DOM Functions allow you to parse HTML (or XML) into an object structure (or DOM - Document Object Model). Let’s see how we do it:

$dom = new DOMDocument();
@$dom->loadHTML($html);

Wow is it really that easy? Yes! Now we have a nice DOMDocument object that we can use to access everything within the HTML in a nice clean way. I discovered this over at Russll Beattie’s post on: Using PHP TO Scrape Sites As Feeds, thanks Russell!

Tip: You may have noticed I put @ in front of loadHTML(), this suppresses some annoying warnings that the HTML parser throws on many pages that have non-standard compliant code.

XPath Makes Getting The Links You Want Easy

Now for the real magic of the DOM: XPath! XPath allows you to gather collections of DOM nodes (otherwise known as tags in HTML). Say you want to only get links that are within unordered lists. All you have to do is write a query like “/html/body//ul//li//a” and pass it to XPath->evaluate(). I’m not going to go into all the ways you can use XPath because I’m just learning myself and someone else has already made a great list of examples: XPath Examples. Here’s a code snippet that will just get every link on the page using XPath:

$xpath = new DOMXPath($dom);
$hrefs = $xpath->evaluate("/html/body//a");

Iterate And Store Your Links

Next we’ll iterate through all the links we’ve gathered using XPath and store them in a database. First the code to iterate through the links:

for ($i = 0; $i < $hrefs->length; $i++) {
	$href = $hrefs->item($i);
	$url = $href->getAttribute('href');
	storeLink($url,$target_url);
}

$hrefs is an object of type DOMNodeList and item() is a function that returns a DOMNode object for the specified index. The index can be between 0 and $hrefs->length. So we’ve got a loop that retrieves each link as a DOMNode object.

$url = $href->getAttribute('href');

DOMNodes inherit the getAttribute() function from the DOMElement class. getAttribute() returns any attribute of the node (in this case an <a> tag with the href attribute). Now we’ve got our URL and we can store it in the database.

We’ll want a database table that looks something like this:

CREATE TABLE `links` (
`url` TEXT NOT NULL ,
`gathered_from` TEXT NOT NULL ,
`time_stamp` TIMESTAMP NOT NULL
);

We’ll a storeLink() function to put the links in the database. I’ll assume you know the basics of how to connect to a database (If not grab a MySQL & PHP tutorial here).

function storeLink($url,$gathered_from) {
	$query = "INSERT INTO links (url, gathered_from) VALUES ('$url', '$gathered_from')";
	mysql_query($query) or die('Error, insert query failed');
}

Your Completed Link Scraper

function storeLink($url,$gathered_from) {
	$query = "INSERT INTO links (url, gathered_from) VALUES ('$url', '$gathered_from')";
	mysql_query($query) or die('Error, insert query failed');
}

$target_url = "http://www.merchantos.com/";
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';

// make the cURL request to $target_url
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$html= curl_exec($ch);
if (!$html) {
	echo "<br />cURL error number:" .curl_errno($ch);
	echo "<br />cURL error:" . curl_error($ch);
	exit;
}

// parse the html into a DOMDocument
$dom = new DOMDocument();
@$dom->loadHTML($html);

// grab all the on the page
$xpath = new DOMXPath($dom);
$hrefs = $xpath->evaluate("/html/body//a");

for ($i = 0; $i < $hrefs->length; $i++) {
	$href = $hrefs->item($i);
	$url = $href->getAttribute('href');
	storeLink($url,$target_url);
	echo "<br />Link stored: $url";
}

What Else Could I Do With This Thing?

The possibilities are limitless. For starters you might want to store a list of sites that you want scraped in a database and then set up the script so it runs on a regular basis to scrap those sites. You could then compare the link structure over time or maybe republish the links in some sort of directory. Leave a comment below and say what you’re using this script for. Here are a few other things people have done with scrapers in the past:

• Build a search engine from the content you gather. Google
• Analyze a site to determine how well it is SEO optomized for keywords. SEO Book’s Keyword Density Tool.
• Republish free content dynamically on your website.
• Create an RSS feed from a website. Using PHP To Scrape Web Sites As Feeds

Is Scraping Content Legal?

There is no easy answer to this question. Many organizations scrap content from all over the web - Google, Yahoo, Microsoft, and many others. These companies get away with it under fair use and because site owners want to be included in the search results. However, there have been copyright infringement rulings against these companies.

The real answer is that it depends who you scrape and what you do with the content. Basic copyright law gives authors an automatic copyright on everything they create. But the same law permits fair use of copyrighted material. Fair use includes: criticism, comment, news reporting, teaching (including multiple copies for classroom use), scholarship, or research. But even these uses could be considered copyright infringement in some circumstances. So be careful before you claim “fair use” as your defense!

Here’s a couple sites that have granted you the right to use their content. They do require you to attribute the content to the author or the URL you scraped it from:

• Wikipedia - GNU Free Documentation License
• Open Directory Project - Open Directory License
• 
  Creative Commons - Creative Commons Attribution 3.0
  Many sites publish their content under some form of the Creative Commons license. You can search for creative commons licensed works here: Creative Commons Search. Remember that it’s your responsibility to verify the copyright rules for anything you use, even stuff found using the Creative Commons Search.

Facebook Is The New Black.

Everyone is talking about Facebook and their new Application Platform:

• Techcrunch - Facebook Launches Facebook Platform; They are the Anti-MySpace
• Mashable - Facebook Powertools: 150+ Apps, Scripts and Add-ons for Facebook
• Digg - The Impact of Facebook’s Platform
• Mashable - Facebook Platform: 30+ Awesome Applications for Facebook
• Valleywag - Developers, beware. Facebook really is the new Microsoft.
• Facebook’s Blog Post on the Facebook Platform
• Inside Facebook - Hottest Facebook Apps: July 28th

And I have to admit I think it’s pretty cool too. So I decided to create an application that uses the Facebook Platform. I’m writing the application in PHP and I thought it might be useful for others to know how to write their own Facebook applications with PHP. So here we are.

What Will I Learn?

I will cover the basics of using the Facebook PHP library and how to get your application started, including:

1. Installing The Facebook Developer Application
2. Downloading The Facebook PHP Client Library
3. Creating Your Application
4. Hello Facebook! Example
5. FBML - Facebook Markup Language
6. Using the Facebook API

Getting Started As A Facebook Developer

The first thing you need of course is a Facebook account. You can sign up for Facebook at www.facebook.com. Once you have your account you’ll need to install the Facebook developer application. This little tool will allow you to generate your application profile and get an API key (more on that later). Once you’ve logged into Facebook visit: www.facebook.com/developers/ or click below to install the developer application:

Add The Facebook Developer Application.

PHP Facebook API Client Library

Facebook has created a nice php library that allows you to use their API without writing a lot of extra code. You can grab the PHP version of the library at developers.facebook.com/resources.php. Download the ‘PHP (4 and 5) Client Library’.

Once you’ve downloaded the library unzip it into a folder that is accessible by your PHP scripts. So you would have something like /php_include_directory/facebook/ and in that folder you will have the entire Facebook PHP Client Library (3 folders: client, footprints, php4client). I’m using PHP5 so my examples will be using the “client” directory of the library. The footprints folder is an example application.

Creating Your Application Profile And API Key

Facebook requires that you register each application you make. Once you’ve logged into Facebook and installed the developer application go to the developer panel (or click here). Inside the developer application click “Set Up New Application”.

Choose a name for your application. This is important because it’s what users will see when they are browsing the application directory. Currently the name field is the only thing used when searching for applications. So it’s doubly important at this point.

Click on “Optional Fields”. Then fill out the Callback Url with the location of your script. This is the public URL on your webserver where the Facebook application will be.

Next fill out the “Canvas Page URL”. This is your application URL within Facebook. For example if the application was called “Makebeta Is Cool” then the application URL could be: “makebeta” which would make the full URL: http://apps.facebook.com/makebeta/.

Check Yes for Can your application be added on Facebook?

You should check the Developer Mode checkbox so that no one can add your application until you are done working on it.

Under Integration Points fill out Side Nav URL with the full Canvas Page URL. In the example above it would be http://apps.facebook.com/makebeta/. This allows users to add your application to their Facebook left side bar navigation.

All of these settings can be changed after the application has been created. But it’s a good idea not to change the Canvas Page URL or Name once you have users that have installed your application.

Further Reading: Starting your First Facebook App: Demystifying Application Form Field by Field

Get the API Key and Secret. You should now see your application listed with a unique API Key and Secret code. You’ll use these within your application.

Hello Facebook!

Let’s create a really simple first application that just says hello to the current Facebook user. Here’s the code for the Hello Facebook! application:

<?php
/* include the PHP Facebook Client Library to help
  with the API calls and make life easy */
require_once('facebook/client/facebook.php');

/* initialize the facebook API with your application API Key
  and Secret */
$facebook = new Facebook(YOUR_API_KEY,YOUR_SECRET_CODE);

/* require the user to be logged into Facebook before
  using the application. If they are not logged in they
  will first be directed to a Facebook login page and then
  back to the application's page. require_login() returns
  the user's unique ID which we will store in fb_user */
$fb_user = $facebook->require_login();

/* now we will say:
  Hello USER_NAME! Welcome to my first application! */
?>

Hello <fb:name uid='<?php echo $fb_user; ?>' useyou='false' possessive='true' />! Welcome to my first application!

<?php

/* We'll also echo some information that will
  help us see what's going on with the Facebook API: */
echo "<pre>Debug:" . print_r($facebook,true) . "</pre>";

?>

So what happens when a user hits the Canvas Page URL (from the example it would be: http://apps.facebook.com/makebeta/)? The require_login() call will produce a screen like this for the user:

If you change require_login() to require_add() the user will get a page that looks like this:

After the user logs into or adds the application they will get the canvas page with the “Hello…” text. It should look something like this:

Facebook Markup Language - FBML

Facebook has provided a bunch of built in tags that will render dynamic data inside your application. All you have to do is include the tags with the correct parameters. In the example above the fb:name tag is used to generate the user’s name on the canvas page. fb:name has a couple of parameters, one of which is uid. In the example above we set uid = $fb_user which is the unique ID of the current user. There are lots of FBML tags you can use, check them all out at: developers.facebook.com/documentation.php?doc=fbml

Facebook API REST-based Interface

The Facebook Client Library provides you with an easy to use wrapper for the Facebook API REST interface. All of the API calls are available under the $facebook->api_client object (after you initiate the $facebook object). Many of the calls will require that the user has either added or logged into the application. Here’s an example call that would retrieve the user’s About Me text from their profile:

$fb_user = $facebook->user;
$about = $facebook->api_client->users_getInfo($fb_user,'about_me');

There are a number of API calls, and a list of them can be found here:
developers.facebook.com/documentation.php

Resources and Further Reading

These are very helpful pieces of information that I highly recommend reading before you get too far into making your Facebook application.

• Basic Facebook Application Architecture
• How URLs Are Handled Within An Application
• Understanding Your Callback Page

And of course the official documentation / developer site:

• Facebook Developers Site
• Facebook Developers Wiki

Other Facebook tutorials and how-to articles:

• Tucows - Getting Started With Facebook Application Development
• 10 Things That Would Have Been Nice to Know When Starting My Facebook Application
• Developing with the Facebook Platform and PHP

I will no longer be hosting Spyjax.

It’s been fun and very interesting, but it’s time to call it quits. Spyjax is just a side project but it’s eating up server power so I’ve decided to turn it off. I kicked around the idea of turning into a real service / business but I’m just not that interested in it.

Thank You For Commenting and Linking!

It’s been really fun reading everyone’s comments here and on all the blogs that wrote about Spyjax. There were some great discussions around privacy and ethics. Some of the best comments where on the SEO Book blog: Spy on Visitor Browsing History for Competitive Research.

Full Source Code Available

Because I’ve decided to shutdown the service I’m giving away all the code as open source. Anyone can install the Spyjax service on their server. Someone else could even start the service up and let others user it. You can download the source here: Spyjax Source Code.
Spyjax uses PHP and MySQL so most web servers should be able to run it. You’ll need to do a little bit of configuration, mainly in the config.php file. There might be a place or two where you need to put your email address for sign up and feedback emails to work correctly. Also the urls.php file needs your MySQL database host, user name, and password. The service assumes that it’s using a database named “spyjax”. The schema for the database can be found in spyjax.sql. If you have any questions about how to install it contact me using the form on over here.

4,808,202 URLs Were Found

While Spyjax was running it tracked 282,542 visitors and collected 4,802,202 URLs from their browser history. Here’s the top 50 URLs found by Spyjax:

1. http://www.google.com - 59927
2. http://www.yahoo.com - 45591
3. http://www.myspace.com - 29869
4. http://www.youtube.com - 27057
5. http://www.msn.com - 22045
6. http://www.hotmail.com - 16969
7. http://www.ebay.com - 16123
8. http://www.facebook.com - 13855
9. http://www.mapquest.com - 11689
10. http://www.cnn.com - 9285
11. http://www.weather.com - 8634
12. http://www.amazon.com - 7852
13. http://www.wikipedia.org - 7401
14. http://www.aol.com - 7381
15. http://www.imdb.com - 6023
16. http://www.walmart.com - 5826
17. http://www.gmail.com - 5740
18. http://www.apple.com - 5549
19. http://www.flickr.com - 5197
20. http://www.chase.com - 5053
21. http://www.ask.com - 4919
22. http://www.google.co.uk - 4876
23. http://www.bestbuy.com - 4682
24. http://www.usps.com - 4607
25. http://www.craigslist.org - 4555
26. http://www.download.com - 4474
27. http://www.digg.com - 4389
28. http://www.paypal.com - 4038
29. http://www.dell.com - 3944
30. http://www.bankofamerica.com - 3846
31. http://www.bbc.co.uk - 3798
32. http://www.southwest.com - 3753
33. http://www.adobe.com - 3704
34. http://www.comcast.net - 3657
35. http://www.t-mobile.com - 3636
36. http://www.yellowpages.com - 3584
37. http://www.monster.com - 3578
38. http://www.nytimes.com - 3556
39. http://www.live.com - 3550
40. http://www.hp.com - 472
41. http://www.orbitz.com - 3456
42. http://www.whitepages.com - 3339
43. http://www.microsoft.com - 3310
44. http://www.capitalone.com - 3251
45. http://www.ticketmaster.com - 3228
46. http://www.target.com - 3157
47. http://www.realtor.com - 3114
48. http://www.ebay.co.uk - 3064
49. http://www.kbb.com - 3036
50. http://www.blogger.com - 3031

Your browser history is not private!

In fact with a few well crafted lines of Javascript, websites can examine your browser history and record what pages you have been to. Keep reading and I’ll tell you exactly how it’s done and introduce you to a service that any webmaster can put on their site to see what pages their users have visited. I’ll also tell you exactly what type of information can be retrieved, and how you can protect yourself.

How JavaScript Can Be Used To Steal Your Browser History:

With CSS website designers can make links a different color if they have been visited by the user. For example this link should be colored differently than this other link. The first link you have been to before (it’s the page you are on right now) while the second link you have never visited (because it is fictitious). Now you’re thinking “but how can this be used to steal my history?”. Let’s dive a little deeper.

Javascript Can Examine The Color Of Your Links = Steal Your Browsing History

Javascript can examine the rendered state of an HTML document, called the DOM. One of the properties that is available through the DOM is the current CSS attributes of a node (nodes are HTML tags, one of which is the <a> or link tag).
All a website has to do to see what pages you’ve been to is place a list of links on the page and examine the color of those links. Ajax can be used to retrieve a list of links to test and also send the results back to the server without the user ever knowing.
The code to do this examination can be a little tricky due to cross browser issues. Here is a snippet of Javascript that can do the evaluation (based on the Hey you! Where have you been? blog post by Peter van der Graaf and script from Jeremiah Grossman and Robert Cabri):
<pre>
function hasLinkBeenVisited(url) {
var link = document.createElement('a');
link.href = url;
document.body.appendChild(link);
if (link.currentStyle) {
var color = link.currentStyle.color;
if (color == '#ff0000')
return true;
return false;
} else {
link.setAttribute("href",url);
var computed_style = document.defaultView.getComputedStyle( link, null );
if (computed_style) {
if (computed_style.color == 'rgb(255, 0, 0)')
return true;
}
return false;
}
}
</pre>
The code above assumes that CSS rules are making links that have been visited red (#ff0000) and new links a different color.

Ajax Can Be Used To Examine Thousands Of Links Dynamically

A clever web developer can use Ajax to dynamically load a list of links for each new visitor. A couple hundred links can be grabbed at a time and examined without slowing down the page noticeably. If you spend just a few seconds on a website thousands of URLs can be checked.

The Limitations

This technique does not allow sites to read your entire browser history. It only allows a site to test a predefined list of URLs to see if you have visited any of them. It’s like the card game “go fish”, you can’t see the players cards but you can ask them if they have any particular card. Most likely the way this technology would be used is to examine a list of competing URLs. This could give a website valuable information on who their competitors really are and what information on those sites is being looked at.

How To Stop People From Spying On Your Browser History

There are two sure fire ways to stop people from stealing your browser history.

1. The nuclear option is to disable JavaScript within your browser. In Firefox you’d just go to Tools -> Options -> Content tab and then uncheck “Enable JavaScript”. This method is very limiting because you probably enjoy all the JavaScript goodness on the web.
2. Limit your browser history. The less browser history you store the fewer URLs someone can steal from that history. In Firefox you can change the amount of browser history by going to Tools -> Options -> Privacy and then either uncheck the “Remember visited pages” checkbox or change the number of days that history is stored for.

UPDATE: Spyjax Has Been Turned Off

I will no longer be hosting Spyjax. It’s been fun and very interesting, but it’s time to call it quits. Read more here.

Introducing Spyjax

One Line Of JavaScript And You Can Start Spying

Ok, now that I’ve explained how this all works and how you can protect yourself, I want to introduce you to a small piece of code that I wrote that makes it super easy for you to spy on your website visitors. It’s called Spyjax and here’s how it works.

1. Sign Up For An Account

   All that’s required is your email address and a password of your choosing. I promise I will not send you any unwanted email or give your email address away to anyone else. Sign Up For Spyjax

2. Add URLs To Look For

   You can add custom URLs, the top 12 Google results for any search, or just look for the home page of the top 10,000 sites on the web.

3. Put One Line Of Code At The Bottom Of Your Pages

   A simple <script> tag will insert all the JavaScript needed to spy on your visitors as well as communicate with the Spyjax service to record the results.

4. Optionally Add A Spyjax Widget To Your Site

   If you just want to have some fun and show people that you’re spying on them you can put one of three Spyjax widgets on your website. There’s one on this site on the right sidebar.

Update: Spyjax Only Gives You Anonymous Data

There have been some concerns raised since I first published this article and released Spyjax. So I just wanted to point out that the service does not link specific websites with identifiable user data. It simply tells you things like 36% of your visitors have been to http://www.google.com/. Basically all the data collected by Spyjax is anonymous and shown in aggregate form. Obviously this same technology could be used to track specific user’s history, especially if you’re on a site that records your identity in some way. In my humble opinion it’s much better to debate these issues in the open than to have this sort of technology floating around without people knowing about it.

So You Just Want The Code?

Well I’m not greedy, so I’m giving it away for free. You can do anything you want with it, just don’t blame me if it breaks or gets you in trouble.

You can download the code here: Spyjax Code. It’s got an open source Attribution Assurance License attached to it.

Check out these services by my company MerchantOS:

• POS Software - A point of sale and inventory control system for small retailers.
• Bike Shop Point of Sale - A full POS solution specifically designed for independent bicycle retailers.

Site: http://wordpress.org/

Description

“WordPress is a state-of-the-art semantic personal publishing platform with a focus on aesthetics, web standards, and usability. What a mouthful. WordPress is both free and priceless at the same time. More simply, WordPress is what you use when you want to work with your blogging software, not fight it.”

Benefits

• Easy to use back end for writing content and making other dynamic changes to your site.
• Simple setup, anyone with a little bit of server experience can install it.
• Themes and Plugins are numerous and mostly free.
• Easy to build your own theme or customize an existing one.
• Uses standard/free web server technologies: PHP & MySQL.

Using WordPress

1. Login to the administration features by going to: http://<your blogurl>/wp-admin/
2. To write a post click on the “Write” tab. You can write in a visual editor or you can write straight HTML. I recommend learning HTML and editing pages in it, you’re posts will come out a lot better and be easier to modify later. Choose your editor under My Profile -> “Use visual editor when writing”. If you have a normal sized monitor you may also want to turn up the number of lines you can see at once in the editor at: Options -> Writing -> “Size of the post box”, I recommend setting it to about 30 lines instead of 20.
   
3. General options for your blog are under: Options -> General.
   
4. An important option is: Permalinks. These determine the directory structure of your blog. You probably want to change it from default to custom with structure of something like: “/%category%/%postname%/”. This will help your site rank in Google and other search engines. %category% will be the category you choose for your post. Warning: For SEO optimization give each post only one category. If you use multiple categories Google will find your post in multiple locations and may discard it as duplicate content. Your %postname% comes from the “Post slug” field on the post writing screen. It will default to your post title but you can change it to anything you want, just make sure that it’s something that will be valid as a url (no spaces or weird characters). For example if we categorized our post as “Awesome” and made a slug of “this-is-the-most-awesome-slug” it would show up on our blog at /awesome/this-is-the-most-awesome-slug/ which would rank well in Google for phrases like “awesome slug”.
5. Your theme is chosen under Presentation -> Theme
   

B. WordPress Themes

Almost everything about Wordpress can be customized through your theme. Themes are where WordPress really starts to shine. Here are a few example themes:



You can browse and download thousands of themes from:
http://themes.wordpress.net/

How do I install a new theme?
Your themes are located in a subdirectory of your WordPress installation: <your WordPress installation directory>/wp-content/themes/. Each theme has it’s own directory, which is the name of theme. You’ll always start out with a
theme called “default” which is the standard WordPress theme. To add a new theme download the zip file for the theme and unzip it into your /wp-content/themes/ directory. That’s it! You should now see the new theme under the Presentation -> Theme tab in the administration back end.

C. Structure Of A WordPress Theme

Theme Templates

There are a couple different types of pages in WordPress. Each type of page can have a customized look and feel. This customization is done by templates. Templates are how themes work. Each template is a seperate PHP file inside the theme’s directory. Here are the standard templates:

• 404 Template = 404.php - It’s handy to create a custom 404 page (Page Not Found) template. You can list common links and ways for users to find what they are looking for.
• Archive Template = archive.php
• Archive Index Page = archives.php
• Comments Template = comments.php - This template defines how comments look under individual posts on the “Post Template”.
• Footer Template = footer.php - HTML that is placed at the bottom of each page, saves you time and produces less duplicate HTML code in your templates.
• Header Template = header.php - HTML that is placed at the top of each page, usually has the <head> section in it.
• Links = links.php
• Main Template = index.php - Usually this page lists your most recent posts. This is the “home” page of your site.
• Page Template = page.php - Used for single pages, instead of blog posts. Your “About” page uses this template. This is the default page template, you can make your own custom page templates. I’ll cover this more in detail later.
• Popup Comments Template = comments-popup.php
• Post Template = single.php - This is where you customize pages that display single posts. Click on a post title (permalink) and you will be at it’s single post page using the post template.
• Search Form = searchform.php
• Search Template = search.php
• Sidebar Template = sidebar.php - Usually the right hand side bar of the page. Some themes have more than one side bar so you’ll see templates like left_sidebar.php.
• Stylesheet = style.css - Default place to put your CSS. It also contains the name and description of the theme. You must always have this file with the name and description of your theme. Many themes will have multiple CSS files that are linked to from the header.php file or the individual page templates to customize individual pages with different style rules.

D. Making Your New Theme

1. Find a theme you want to base your new theme off of. A good place to start is the default theme, but you may also want to just tweak another theme that’s closer to the look/feel you are going for.
2. Copy the existing theme to a new directory in your themes directory for example if I was making a theme called “webdesign_meetup” I might copy the “default” directory into a new directory called “webdesign_meetup”.
3. In your new theme directory edit the style.css file. You should see something
   like this in that file:
   /*
   Theme Name: WordPress Default
   Theme URI: http://wordpress.org/
   Description: The default WordPress theme based on the famous Kubrick.
   Version: 1.6
   Author: Michael Heilemann
   Author URI: http://binarybonsai.com/
   */
4. Edit the theme information. This is the information that will show up under Presentation -> Theme in the administrator.
5. Go into the WordPress administrator and change your current them to the new one you just created.
6. Now you can start customizing and see the affects in real time on your site!

E. Modifying The Header

The first thing you might want to do when customizing your theme is change the header. Most sites have a custom header image, usually a company logo or a picture that has to do with the theme of your site. To edit the header just go into your theme directory and edit the header.php file. The header HTML will usually be stored in this file. In some more rare cases theme authors may have placed header information in the individual page templates (if they did you’ll have to hunt around a bit more!). You’ll usually see a piece of HTML under the opening <body> tag that’s something like lt;div id=”header”>…header content…</div>. You can customize this content with your new header.

F. Blog / Post Templates

Maybe you want to display an ad at the bottom of each post, or perhaps you’d like to add or remove the author information under the post header? The place to do these things is in the single.php file. This file controls the layout of your individual post pages.

Let’s examine the components of this file for the default WordPress theme:

<?php get_header();
?>

<div id=”content”
class=”widecolumn”>



        <?php
if (have_posts()) : while (have_posts()) : the_post(); ?>

<div class=”navigation”>

    <div
class=”alignleft”><?php previous_post_link(’&laquo; %link’)
?></div>

    <div
class=”alignright”><?php next_post_link(’%link &raquo;’)
?></div>

</div>

<div
class=”entry”>

    <?php the_content(’<p
class=”serif”>Read the rest of this entry &raquo;</p>’);
?>

So how do I add some author information and maybe date of the post to this?
It’s easy! There’s a lot of information stored with each post that is accessible within “The Loop”. To see a list of all the functions you can use to get at this information visit this documentation page: http://codex.wordpress.org/Template_Tags. There you’ll see all the template functions that are available to you. So to get the author of the post we’d just call the_author();. Of course we’d want to put some HTML formating around it so we might do:

Now to put the date the post was written we could put:

<?php the_date(’Y-m-d’,
‘<h2>Published On ‘, ‘</h2>’); ?>

There are tons of stuff you can do with posts so check out: http://codex.wordpress.org/Template_Tags and get started.

G. The Post Loop

So you want a custom loop? The loop is powered off of the current page query. The query is generated by WordPress, typically from your URL. So if your on your site and your at /category1/a-blog-post-with-id-12 the query to get blog post id 12 in category 1. The internal mechanics of how this happens are fairly complex, so we won’t go into them. It’s enough to know that each page has a query generated for it that will grab the correct content for that page. So if you want to customize your loop and grab something besides what WordPress would do by default you need to modify the query before you call your loop.

Let’s say you want to put just the 3 most recent blog posts on a page, maybe under some other content. On my site the home page has static content and then displays some recent blog posts. Here’s how you do it:

<?php query_posts(”showposts=3″);
?>

This does a new query to grab the 3 most recent posts to the blog. I can then display these posts in a normal loop:

<?php if (have_posts()) {
?>

    <h3>Recent
News</h3>

    <?php while (have_posts()) :
the_post(); ?>

        <div
class=”blog_post”>

       
    <?php the_content();
?>

       
</div>

    <?php endwhile;
?>

<?php } ?>

There are almost infinite ways you can write queries to get the posts you want for your loop. Visit these two pages for more information:

http://codex.wordpress.org/The_Loop
http://codex.wordpress.org/Function_Reference/WP_Query

H. Page Templates

Page templates are used to format specific pages. For example if you want your About page to have a specific template you can assign it a page template. Posts can not have templates, only pages (under Write -> Write Page). To start a new page create a new php file, for example we could call our new file about_us.php. In the file put this code at the top:
<?php

/*

Template Name: About
Us

*/

?>
WordPress will see this file as a page template and will display it in the list of page templates you choose from when you are editing a page. To create a page in the admin interface go to Write -> Write Page. Then on the right side bar you can pick a custom page template. Pick your new “About Us” template.

You probably want a number of standard template elements on your custom page template. Here’s a good framework to start you template with:

The loop on your page template will by default grab the content your write for your page in the admin interface.

I. Sidebars

Most templates have one or sidebars. Sidebars usually contain navigational links, but they can also include search boxes, lists of related pages, and other widgets (like mybloglog etc.). It’s really easy to edit your sidebars. Just find the sidebar.php file and open it your editor. You’ll see the HTML that creates the side bar and puts the content within it. There are a few template functions that are helpful within the sidebar like wp_get_archives(); which will display a list of time periods that you wrote blog posts (think August (8), September (2) which are links to those months posts). Most sidebar functions are meant to be placed within <ul></ul> tags because they format their output as lists. So in your side bar you might have this code:

This code would generate two lists, one of archive months, and one of categories you have posted under.

You can find more of these functions at:
http://codex.wordpress.org/Template_Tags

J. Meta Data / Custom Fields For Page Customization

Meta data can be added to any blog post or page. It’s located under the edit box on the Write Post and Write Page sections of the admin interface. Look for the section called “Custom Fields”, that’s your post meta data. Each piece of meta data has a key and a value. You can use meta data to store almost anything, and you can retrieve it within the page using some template functions/tags. Here some code I use so that each page can have a custom title, description, keywords, and style rules. This code is placed within my <head></head> tags in my header.php file:

The first section of the code loads the meta data. For example to get the title meta data I call $title = get_post_meta($wp_query->post->ID, “title”, true);. I can then use the $title variable within my code to display the title information I retrieved. The styles meta data I retrieve is a little
more complex because it is grabbing an array of styles. So if there are more than one meta data elements with the title “style” I will get all these styles within one array. The last parameter of get_post_meta determines if an array is to be returned or just single value. I loop through the array and display each style using this code: <?php foreach ($styles as $s) echo $s . “\n”; ?>.