I know it’s been a while since posted anything on this blog, but I’ve had a lot going on and it hasn’t really been at the forefront of my mind. So, what’s been going on?

Firstly I’ve shut TwitApps.com down. I won’t go into the reasons because they’re well documented in the blog post announcing it. I’ve also released the code so anyone can make a clone if they feel so inspired. One big side effect of this is that I’m open to offers for that domain name and a bunch of others I don’t see myself using anymore. Please only contact me with serious offers.

Secondly I’ve been doing a lot of soul searching lately, and one of the results of that has been my new blog over at stuartdallas.com where I’m writing for my benefit rather than as a service to others, which is how I view this site. Check it out and let me know what you think.

Finally I have been all over the place in terms of my career lately, which I’ve written about on my new blog so I won’t repeat it here. Ultimately it means I’m on the lookout for interesting projects to get involved with. If you think I may be able to help you or your company please let me know.

Hopefully that explains where I’ve been. I plan to continue publishing tech and development posts on this site for the foreseeable future, but how often depends on how much my focus gets split in my new reality. I also plan to change the layout of this site – it’s starting to depress me because I never really finished it.

One of the pillars of a scalable website is ensuring that only activity which is required to build a page should be performed during the processing of a page request. Activities that fall under this category commonly include sending emails, recording statistics and general housekeeping such as removing temporary files.

Back when I started working on sites big enough for these activities to cause a problem I went down the obvious route of making a PHP CLI script for each job that needed doing and getting it to run using cron. This worked for a while but as the sites I was working on got bigger and more complex it quickly became clear that this was becoming difficult to manage, so I started to consider alternatives.

At the time all this happened I was mainly working with a site that ran on Zend Platform. One of the recent additions at the time was a module called Job Queue which appeared to do exactly what I needed. Unfortunately after spending a fair amount of time developing the infrastructure required to make it run the jobs I discovered that it really wasn’t very well tested; it was far from production quality [1] and nowhere near reliable enough so I went back to the drawing board.

Time passes…

After thinking about what I needed from such a system and what I had available I came up with an arrangement I’ve been using ever since with great success.

The core of the system is a DB table and a PHP script. The table contains the definition of jobs that need running and the script, erm, runs them.

I won’t go into the details of the table because the only important parts as far as this system go are the run_at, processor_pid and schedule_* fields.

The run_at field is a timestamp that simply indicates the time when that job should be executed.

The processor_pid field is an unsigned integer that defaults to 0 and will indicate the process that’s running a job if any.

The schedule_* fields specify how often the job should be executed. There are a number of ways of organising these depending on what your requirements are, but they fall into two general categories.

Before I move on to the script I apparently need to cover something that I missed when I first wrote this. You need to specify what to run in this table! I though this was pretty obvious but based on initial feedback I was wrong.

This can take any number of forms and will depend greatly on your specific application. Over the years I’ve used a number of different methods including a PHP script name, a CLI command and a method in a static class. Whatever you use you’ll want to make sure you do sufficient checks to ensure it’s secure.

Ok, on to the script which is designed to be run by cron according to a schedule that allows it to keep up with the size of the job queue you anticipate.

The following is a list of the basic steps the script performs. For this example I’m continuing with the assumption that the job queue is a table in a database.

1. Get the pid and put it in $pid.
2. Execute this SQL query…
   update `job_queue`
   set `processor_pid` = “$pid”
   where `run_at` <= unix_timestamp()
     and `processor_pid` = 0
   order by run_at asc
   limit 1
3. Check mysql_affected_rows() and exit if 0.
4. Execute this SQL query…
   select * from `job_queue`
   where `processor_pid` = “$pid”

   Note that these SQL statements (step 2 and this one) atomically grab a job and lock it. If you’re using a different storage system for your queue you’ll need to lock it while you select a job to run and then mark it as in progress.

5. Run the job. As mentioned already this can mean any number of things and will depend on your particular application. One of the useful things you can do here is to set up a clean, safe environment for the job to run in, along with ways to capture errors and other outputs so you can do something useful with them.
6. As soon as the job has finished executing we mark it as completed and record the success or failure status.
7. If the job has a schedule (i.e. it’s a recurring job) we create a new job by effectively cloning the job we’ve just run. We now calculate the time it should be run according to the schedule definition and save that in the run_at field of the new job. Finally we set the processor_pid of the new job to 0 so it’s then available to step 2 of this process.
8. Depending on the job configuration and status we now either remove the completed job from the queue or archive it complete with errors and output for later inspection.
9. If this processor has been running for > 60 minutes it exits, otherwise it goes back to step 2 and looks for another job to run.

To get this to do something useful we configure it to run via cron every n minutes where n depends upon the anticipated size of your job queue. For example running it every minute will automatically scale it up to 60 concurrent jobs at any one time. Running it every 5 minutes will reduce this to 12, and so on. There’s also nothing stopping you putting more than one line into the crontab so it runs two processes every minute which increases concurrent processors to 120.

Assuming your job queue is network accessible this system also scales across multiple machines with minimal changes. In fact the only change that’s required it to incorporate a machine identified into the processor_pid field. This could be as simple as <machine>_<pid>; the key thing is that it’s guaranteed to be unique to a given process across your entire infrastructure.

Crashed jobs

One problem you may need to deal with is how to handle crashed jobs. This will happen, you can’t get away from it and you’ll need a way to detect and deal with it when it does. Luckily the job queue makes detection fairly straightforward.

On a single machine you can implement a script (either run via cron separately or indeed run by the job queue) that will check that for each job that has a processor_pid > 0 there is a PHP process running with that PID.

If and when you’ve scaled across multiple machines this script essentially remains the same except that you need to run it on every machine that runs the job queue and filter the PID’s you check.

As far as what to do when you find a crashed job that’s really something you need to consider on a case-by-case basis. At the very basic level the script could simply reset the processor_pid field to 0 so it gets run again. At the other end of the spectrum in a very flexible system you could have a way to run a job with a flag to indicate that it had previously crashed; each job can then deal with crashes in their own custom way.

Potential additions

The components described above is just the core of a job queue system; you can add a lot of useful stuff above and around it to make it easier to manage and provide better feedback from your periodic tasks.

• Performance metrics
  Since you’re running all your jobs from a central script adding code around the actual execution to record execution time, load and possibly memory usage too is pretty simple. You then have the ability to compare the time a job took to previous executions of that job to detect potential errors.
• Management interface
  Since you have all the information regarding the jobs in the queue, what’s running right now and the status of jobs that have previously been executed it’s a pretty small leap to build a UI that will let you view and manage the whole thing. This can be especially useful for presenting error messages and performance metrics.

Final thoughts

I hope that’s useful to someone, I’ve certainly found it applicable to most web applications I now deal with. As I mentioned a few times you can implement the various parts in a number of ways, in particular how jobs are specified and how the scheduling works.

As an example of this flexibility I’ll make a passing mention to one implementation I’ve done that only needed to be able to execute scripts daily, hourly or every 15 minutes. To accomplish this I simply created a folder for the scripts, and three folders named 15min, hourly and daily within that. The processor script uses a custom locking system and CLI arguments to run each set of scripts according the the directories they’re in.

This system has proven to work very well and will continue to do everything that site needs until we need to execute something at a specific time; a requirement that has not yet surfaced.

One of the many side projects I’m working on is a reusable version of this system. At the moment it’s a fairly messy combination of scripts that doesn’t work very well so far, but as soon as I have something worth sharing I’ll definitely do so on this blog. Stay tuned.

If you have an questions or suggestions for improvement please don’t hesitate to leave a comment or contact me privately.

[1] This was a few years back and I’ve heard that the Job Queue module has received some attention since then so is now a lot better, but I no longer use Zend Platform so I’m not in a position to comment. If budget is not an issue for you I’d recommend checking it out.

A new phishing attack against Facebook has surfaced recently using the domain fbaction.net. Whenever I see these things I always reset my browser and try signing in with fake details to see what happens. In this case I got a nice surprise. When I got redirected to Facebook I got this…

This is fantastic. Every site should do this. It’s a really simple check to do and immediately informs the user that they have probably been compromised complete with advice as to what they should do next.

Top marks to Facebook!

It’s 5am and I can’t sleep. When this happens I usually fire up Boxee and find something to watch, so that’s what I did. I was happy to see that since the last time I used it Boxee appear to have added a bunch more internet sources, which is great. However, the following 15 minutes were nearly the most frustrating of my life so far.

I understand regional restrictions. I don’t necessarily agree with the reason they’re there, but I understand why.

What I can’t understand is why they’re not shown up-front when browsing available content. On sites like Joost, Hulu and others you have to actually start streaming content before you get told you can’t view it. They clearly know where you’re coming from but they still insist on not telling you what you can and can’t watch until you actually try.

Maybe there’s a technical reason why it works this way (I can’t think of one that couldn’t be easily worked around), or maybe there’s a licensing reason for it. Whatever the reason it makes attempting to use any of these sites more and more frustrating the more region-limited content they contain.

So here’s my plea… please will these sites make it clear from the highest level possible in the browsable structure whether content is viewable from my current location. That’s all I ask.

Boxee themselves don’t help out with this problem. Take theWB for example. You can’t access the website at all if you’re outside the US (see below), but you can if you use Boxee. Right down to the content level, but when you start streaming you then get a message telling you it’s restricted.

So, Boxee and co… sort it out!!

Not sure if I believe this is real or not, but either way it’s a pretty cool idea.

I had a request recently to release some of the code from the TwitApps bots. The core of both bots is actually a shared email handler and I have no problem sharing it with the world.

Twitter-related stuff » Handling email notifications

On the assumption that I’ll eventually get around to sharing some more Twitter-related stuff I’ve created a new section in the projects area of this site. If there’s anything related to Twitter integration that you’d like to see me cover there please let me know and I’ll be happy to help.

A couple of days ago I released a new TwitApps tool called Follows. I know I haven’t talked about TwitApps on this blog yet – that post is coming – but if you follow me on Twitter you should be aware of it.

The Follows service monitors your followers on Twitter and sends you a daily, weekly or monthly email telling you who’s followed and who’s unfollowed you since your last email. Twitter itself has the facility to send you an email every time someone starts following you but does not offer any sort of notification when they stop.

I can see why Twitter have done it this way. When someone starts following you it’s likely you’ll want to check out their tweets and you might decide to follow them back. The same logic doesn’t really apply to when people stop following you, or does it?

I built the Follows service primarily as a technical exercise, but also because I was curious to get some visibility on people who stop following me.

A while back a service called Qwitter appeared on the scene that offered an unfollow notification service so I signed up. Unfortunately it stopped working shortly after that for unknown reasons, but in the few days while it did work it was quite enlightening. By paying attention to both follow and unfollow notifications it was possible to spot some interesting trends.

By a strange coincidence it would appear that shortly after I launched my Follows service Qwitter started working again, but so far it doesn’t appear to be particularly reliable. It attempts to match an unfollow event to one of your tweets, effectively trying to find the cause of the action. This is daft primarily because there’s not always a single reason why people stop following you but also because they can’t possibly monitor your followers continuously – the Twitter API has limits that would prevent that – so it’s likely the tweet they attribute the action to was not your latest tweet at the time.

Since Follows went live I’ve been keeping track of what people tweet about it and Qwitter and I’ve noticed some interesting behaviour in response to notifications.

Quite a few users react to people unfollowing them by calling them out with a public tweet asking them why. I can understand the desire to get an explanation but doing it publicly seems a bit … well I dunno really, but it certainly isn’t the best way to go about asking the question.

The other common reaction is the realisation that their self-esteem can’t take being informed when people stop finding them interesting. Again I can understand this to a certain extent but it’s not something you can take personally and remain well-balanced. It’s the constant battle between the need for validation and the bliss of ignorance – part of the “human condition”.

Another interesting side-effect has been some comments from other Twitterers to the effect that such a service is pointless and/or purely egotistical, but I’m convinced there’s great value in getting visibility of the ongoing changes to your follower list.

I use the daily emails I get from Follows to do two things. First of all I check every new follower to see if it’s worth following them back, just like I did when I was getting the individual notifications from Twitter but now I only do it once a day which is far more efficient.

The second thing I do is have a look at every ex follower and if I’m following them I consider whether it’s worth continuing to follow them despite their decision to stop following me. I would never ask them about it because if someone has a problem with what I’m saying, whether it’s frequency or content, I would hope they’d tell me about it. It’s difficult to learn without feedback.

For me Twitter is not a popularity contest and it’s not an ego broadcasting outlet. It’s a conversation. If someone decides to stop following me I take that to mean they no longer want to participate in conversations with me. That’s fine, but it doesn’t necessarily mean I no longer want to participate in conversations with them, or that I’m no longer interested in what they have to say, but knowing they have made that decision is empowering.

What do you think? Do you see value in knowing if and when people stop following you or is it too much information?

Absolute genius!

Twitter promised OAuth support a l-o-o-ong time ago, and it would appear to finally be here. Alex Payne sent a “Call for OAuth beta participants” to the developers list yesterday and had an overwhelming response. This predictably triggered a deluge of tweets and blog posts, but the one that caught my eye was on ReadWriteWeb titled “Why Twitter’s New Security Solution Could Pave the Way to a Future Web of Mashups“.

A number of things disturbed me about this post so I posted a comment or two but I believe the issues involved deserve more attention.

First of all the post asserts that sharing our Twitter password with other websites and applications “makes a lot of us very uncomfortable”. If doing that makes you feel uncomfortable, don’t do it. And this is my main criticism regarding this post. OAuth is being hailed as a solution to sharing your password with third parties, and it is, but it doesn’t protect your account once you’ve given a third party access. And surely the more important message is that if you care at all about the security of your Twitter account you should not be sharing your password with anyone but the Twitter site, an even then only after you’ve checked and double-checked that you’re on the actual Twitter site.

OAuth partially solves the problem in that third parties don’t get your password, but they still get access rights to your account. They’ll still be able to read your direct messages and post tweets and direct messages on your behalf.

The point here is that even with OAuth you are still giving a third party access to your account albeit slightly more limited access. We’re yet to see any details regarding the Twitter implementation of OAuth, and a lot will depend on how fine-grained the permissions system is and how their side of the user experience looks and works, but I think my point is easier to explain with an analogy.

The common analogy used with OAuth is that of a valet key for your car. These are special keys that only allow your car to be drive a few miles at most – just enough to be parked and then brought back to you, essentially giving limited access to the valet. However, consider that you’re still leaving your car with the valet unattended for maybe a few hours. They can get into the car. They can drive it on to a car transporter. They can then take it anywhere they want where they can take it apart to disable whatever security you have until they get full access. You’ve not made your car “secure”, you’ve limited the damage that can be done and made it harder to take it away from you, but if they’re determined you still may never see your car again.

Ok, so the analogy doesn’t completely work since you can’t take a persons Twitter account away with OAuth and gain full access given enough time, but the basic point remains. You’re still giving someone access to your account therefore implying you trust them.

This is known as security theatre which is another way of saying it gives people the illusion of security where the benefits are actually minimal.

OAuth comes with risks as well as benefits, and these are rarely covered. Let’s say I’m a “bad guy” and I want to collect Twitter users passwords. Pre-OAuth it’s pretty easy, you just come up with a viral application and work hard to get people using it. Post-OAuth it’s a little more difficult but how much more depends on how Twitter have imlemented it.

Let’s take the worst-case scenario in which Twitter have used a plain login page for OAuth authentication. I as the “bad guy” write my application so it appears to be taking users to Twitter to authenticate, but in actuality the login page is still on my website. It looks like Twitter, it acts like Twitter, and most enough users will believe it’s Twitter to make it work. Once they enter their username and password I can simulate a login against the actual Twitter site to confirm they’re correct, store them away and take the user to my application just as if they’d signed in on the real Twitter site.

The simple fact that OAuth redirects the user to the Twitter site for authentication allows this phishing attack to be pretty successful.

Now, Twitter can prevent this from happening using a personalisation feature such as allowing the user to upload a secret image that’s then shown to them on the login page. Because only Twitter has that image when a user sees that they know they’re on the real site – various OpenID providers use this system. Let’s hope Twitter has implemented something similar.

Ok, so having read all that you might get the impression I’m anti-OAuth. I’m certainly not. I believe that OAuth will be a great thing for Twitter, but I feel it’s important that all the coverage it’s getting also highlights that there’s still trust involved between the user and the third party. It’s about setting realistic expectations, because believing something is secure can be far more dangerous than it actually being insecure.

I’ve been accepted into the OAuth private beta and I’m excited to see what Twitter have implemented.

Finally I just want to state that I have great respect for Marshall Kirkpatrick and the work he does over at RWW. I wish I could blog as often as he does and stay interesting. Maybe in 2009 I should try it. Hmmm…