Andy's Notebook

.net and ISA Server - a match made in hell?

2010-01-19T22:24:01+00:00

I've been looking at the bandwidth consumption of a .net based webapp and working with the developers to implement many of the common recommendations for improving front-end performance - minifying, gzipping, cache-control headers etc - we've noticed some improvements in performance but we've also noticed some oddities.

.net has the ability to embed static resources e.g. javascript, css etc., in a .dll and then these can be served to the browser using a special handler - ScriptResource.axd for the Microsoft AJAX framework and WebResource.axd for others.

Typically these appear in a web page looking something like this:

(The d in the query string uniquely identifies the resource, and t is a timestamp)

The WebForms javascript that's served via WebResource.axd, hasn't been minified or gzipped but at least it has some cache-control headers. The scripts served out but ScriptResource.axd are somewhat better, the release versions are minified, gzipped if the browser supports it, and served with a long expiry time, so should be ideal for caching.

Why Microsoft chose URL handlers instead of nice plain and simple static files is beyond me (but then most of the .net framework seems to be about over complicating simple things)

The traffic to our production servers is monitored via Oracle REUI which allows us to slice and dice the information in it's cubes so we can see response times, which URLs are requested most frequently, what's consuming the most bandwidth etc.

After making several changes to try to reduce bandwidth consumption, ScriptResource.axd was still at the top of the pile of URLs by bandwidth and it wasn't clear why until we started looking at our users environments in a bit more details.

Most of our customers are schools and they generally have a gateway cache of some form, increasingly this appears to be ISA Server.

By default ISA Server removes the accept-encoding header so users will get the uncompressed versions of scripts and other compressible resources (Steve Souders covers this in Even Faster Websites). There are some scripts around that will change this setting but it appears not many people have implemented them.

Also ISA Server doesn't cache URLs with query string parameters (I can see the point of this) but it means that by default all those .net resources that are served using WebResource.axd and ScriptResource.axd aren't being cached.

There are ways of using the asp:ScriptManager control to replace the ScriptResource.axd URLs with references to static files but there appears to be no way of doing that for WebResource.axd

It appears Microsoft have managed to develop a web framework and a proxy caching product that don't work well together... the words arse and elbow spring to mind...

(Over the next few weeks I'll be spending some time looking further into this and also comparing the ISA Server behaviour with the Squid based proxies that some of the other schools in the UK have. I'll report back when I know more)

From RSS to delicious via ping.fm and twitterfeed

2009-01-26T23:06:44+00:00

One of my continuing frustrations in the love/hate relationship I have with delicious is there's no easy way of getting items from an RSS feed into delicious.

I've got stuff I've 'liked' in FriendFeed, starred items in Google reader, favourites on Slideshare, Twitter etc, that I want to pull directly into one place and that place is delicious, but it seems delicious aren't interested in making my life easy.

Apart from bitching on the delicious forums, I've been fiddling with various ways of filling the gap for a while and finally I think I've found a way of partially filling it using ping.fm and twitterfeed. It's got a few limitations but it may be good enough for now.

Here's a list of the steps needed to get things up and working:

If you don't have a ping.fm account already, head over and create one

Add your delicious account so that ping.fm can post to it (warning you have to give them your delicious password)

Add some form of micro-blogging/blogging account too. This is needed for the link from twitterfeed to work, I used tumblr (but again I have to give my tumblr password, has no-one heard of OAuth?)

Now head over to twitterfeed, and create an account if you need one.

Create a new feed for ping.fm

You'll need your ping.fm API key which you can get from ping.fm/key, and you'll also need to pick a posting method - I chose 'microblogging' ( but I wasn't able to make any choice until I added the tumblr account)

Add in the details of the feed you want to push into delicious along with how often the feed should be checked - I choose to have the feeds checked as often as possible and have as many items as I can pushed across.

By default twitterfeed shortens URLs using tinyurl, you can either stick with this or choose an alternative (I used bit.ly, but that's just a personal preference)

Finally in the field 'Prefix each tweet with' add '@de ', this tells ping.fm to forward the item to delicious

Repeat the above step for as many feeds as you want to pull into delicious

Hopefully, if everything is right you should soon have items arriving in your delicious account.

The approach does have a few drawbacks though:

twitterfeed shortens the URLs, and delicious has no way of expanding them, so the tags from other users can't be applied to bookmark. The original URL being destroyed will also be a problem for people who use delicious to keep track of others who bookmark their blog posts.

The first time twitterfeed does it's thing it only collects the most recent RSS item so all the other items remain uncollected.

twitterfeed only collects five items at time from each feed so it may be possible to build up a backlog if you added a lot of items to your feed

You have to give ping.fm your delicious login details and probably login details for another service too

The bookmarks bypass my delicious inbox and I'm not sure whether this is a good thing or not - I don't always want to keep all the items I like of Friendfeed, or star in Google Reader so having them arrive in the inbox for a final last filter seems like a good idea

If a item already has tags in the feed delicious can't make use of them.

As a word of warning I've also got no idea what happens if one of the items already exists in my delicious collection and ping.fm tries to add it from a feed, so perhaps I need to check that.

It's an imperfect solution but I'm going to give it a go for a while to see how it works, ultimately this is something delicious and other social bookmarking sites should provide, indeed if one of the other sites introduces the ability to add bookmarks from RSS I'm likely to jump ship and leave delicious behind.

Finally if you do decide to go down this route think about making a donation to twitterfeed, Mario offers a great service for free.

(Oh and thanks to Scoble, who's broadcast of his plan for world domination led me to checkout ping.fm)

If anyone comes up with any improvements or alternative ways of tacking this, I'd be very interested in hearing them.

Alas, twitter cripples SMS in the UK (and elsewhere)

2008-08-14T11:36:23+01:00

Woke up this morning to an email from twitter saying they were scrapping outbound SMS messages for the portion of the world that charges for sending rather than receiving text messages i.e. anywhere that's not the US, Canada or India.

The same message is on their blog too - Changes for Some SMS Users—Good and Bad News - and despite the title there was no good news in the post, only the vague promise that they'd introduce new local SMS numbers for unspecified countries within Europe in the coming weeks and months.

I can understand why they've done this - the costs of sending all the messages must burn through money - but by removing the service before a replacement is available I think they've screwed up.

Unlike the US, Canada and India, most of the world charges people for sending text messages but they also offer a way to charging people for receiving messages via SMS reverse billing.

In my view, twitter should have had this option up and ready to go before scrapping their existing SMS service. I guess the danger for them is that their users will continue to migrate to other service (http://identi.ca/ for example) or other ways of being notified e.g. RSS on phones, damaging their future revenue streams.

Perhaps the fact twitter are US based limits their understanding as to how people use mobile phones and how the networks work in the rest of the world?

Also as twitter is pretty open (via rss and the API), perhaps there's also an opportunity for someone to set up a third party service to provide and charge for SMS updates?

Of course this screws up my traffic bot (uk_m5_traffic) as it relies on twitter to send out the SMS notifications. In the short term I can set up the feed reader on my phone to update from the RSS feed but longer term I guess I'll need to see what twitter do and what other options I've got.

Have you ever launched a product you've been ashamed of?

2008-05-22T23:17:47+01:00

In a few weeks time the product I've spent the last eighteen months on will launch. I've built and launched products before and despite their quirks and issues I've been proud of them all, but this time it's different, this time I have an almost overwhelming feeling of sadness.

I'd describe myself as a pragmatic perfectionist (I like things done right but the closer the deadlines get the more pragmatic I become), but this time I seem to have run out of pragmatism.

Part of me (I'm not sure how much) doesn't care about this product any more and I find that really sad as I care about what I do but the effort it's taken to get it this far and the frustrations involved in development have taken their toll. When I told my wife about how I felt she just put her arms around me as she understood the magnitude of what it meant (no chance of me ever complaining that my wife doesn't understand me!).

Functionally the product appears to work OK, but there's so much that's clunky and inconsistent about the way it's been built, behaves and looks, and in the time we've had for development many of them could have been addressed.

The problem as I see it is that the software development side of the product was outsourced and I don't think the company I work for picked the right partner or have learnt from their previous experiences with this partner.

In general I believe product development is far too important to outsource, as "no-one cares about your product like you do". I knew the development was being outsourced when I joined, but at the time I just finished a contract job and needed another so was in the position of "beggars can't be choosers"

The product had already had a few false starts before I arrived but between us we finally got it really moving early last year some of the early prototypes were promising but the immaturity of our partner quickly started to show through.

Their web development techniques seemed to be stuck at around five years ago, all tables and no semantic markup or styles, where there were styles they tended to be in-line rather than an in a stylesheet so you could never be sure that a change in style would be reflected across the product. Their javascript was quite frankly appalling (badly structured and indecipherable) and I'm sure they wouldn't have accepted the same quality of code in the web-services components of the product.

I've spent the last year teaching a development team about everything from web development techniques (semantic markup, how to use css, why they should use a library like jquery (or prototype etc.) instead of writing their own DOM manipulation code), through web infrastructure (how caching can make a huge difference to the performance of our product - we're serving quite a lot of media rich content, the role load-balancing has in scaling and high availability) though to how we can't make users decisions for them (if the user wants to delete something, let them just warn them of the consequences before they do).

Even now I'm not sure how much of what I've tried to teach them has stuck, sometimes I see glimmers that it has and sometimes I see the old ways are back, all I know is that this product has just been all take with no return and I need something back, so it's time to find something else to move on to...

(I will try to write more about this product, what's gone wrong and what I think I've learnt in the next couple of weeks, just need a bit more time to reflect first)

Feed readers and social bookmarking - a missing link?

2008-04-14T22:12:51+01:00

Why isn't there an easy way of getting items from a feed reader into a social bookmarking service?

Like many people (or is it just the geeks?) I use Google Reader to keep up-to-date with news and opinions from around the web, and del.icio.us to bookmark the interesting things I come across. A couple of weeks ago MichaelSearcy's analysis of Google's Broken Bookmarking Systems popped up and got me thinking about how I could get more out of the Reader/del.icio.us combination and what it would take.

In Reader I tend to mark items I want to read later or keep for the future with a star, sometimes I get around to reading them again, and sometimes I get around to adding them to del.icio.us but sometimes despite my best intentions I don't and slowly I'm building up two parallel sets of items I want to keep, one in Reader and one in del.icio.us.

What I need is an easy way to get items I want to keep from Reader into del.icio.us - currently I either have to open the item up in a new browser tab and then use the del.icio.us toolbar to add it, or use a Greasemonkey script to add the ability to post straight to del.icio.us from Reader and it's not always how I want to work.

What I want to be able to do is mark something as shared in Reader and have it appear in my del.icio.us inbox and from there I can choose to save it (being able to delete items from the inbox would be helpful too).

Reader already produces a feed for shared items (and other the labels if you know how to get at them), but del.icio.us doesn't have any way of consuming feeds and they don't seem too keen on the idea of adding it. I've had a brief scan of some of the other social bookmarking sites and they don't seem to support getting bookmarks from a feed either, so perhaps there's an opportunity for someone?

For the second part of the puzzle I'd like a way of having any tags I attach Reader to be included in the feed, this is something del.icio.us does with its feeds so Reader could do it via something like xFolk microformat, then not only will my shared items arrive in my del.icio.us inbox but they'll also arrive with tags already attached.

Data portability and how private data can be shared between social services is a hot topic but right now, I'd settle for services that can share my public information!

Ideas for improving my twitter traffic bot

2008-03-25T12:43:29+00:00

The bot I built for keeping me up-to-date with traffic conditions on the M5 has proved handy a few times but it also suffers from a couple of drawbacks...

It only covers the M5, and the whole of the M5 at that, if someone wants updates on the M4, or just part of the M5 there's no way for them to choose the information that's relevant to them.

TwitterFeed also introduces a time delay and even on it's most frequent setting it will only check a feed every 30 minutes - as we found out on Friday morning when we ran into a jam just north of the M4/M5 interchange only to hear our phones get the text message telling us there was a hold up!

So I've been thinking about building a new bot that would allow someone to choose the roads (and parts of) they want to receive updates for.

The bot would allow people to send it messages about the roads they're interested in, grab the traffic feed from the Highways Agency (and perhaps other feeds) on a regular basis, and when there's something to tell people message them via twitter.

I'm thinking of something that's quite similar to mytrade's twitter bot in concept but needs to be more frequent in it's updates. The language used to communicate with the bot also needs to allow richer concepts as in stock watching terms it needs to support not just broadcasting prices, but broadcasting when a price crosses a user set threshold.

The things I'm looking at right now are:

The syntax for communicating with the bot - something similar to the syntax of mytrade's bot might be good enough but I need to have a hunt around to see what alternatives are about. (I'm not sure I like twitter's format for direct messaging and wonder whether !username might be fit better with the @username format used elsewhere on twitter, but there's not a lot I can do about that!)

Whether to use the REST API or Jabber / XMPP for communicating with twitter. The API pages suggest using Jabber / XMPP for bots that use direct messages so perhaps I'll follow their advice to begin with.

So time to brush off my coding skills and as I (hopefully) make some progress I'll add some updates here.

Some other online tools for filtering RSS

2008-03-07T12:19:45+00:00

UK_M5_Traffic currently uses Yahoo Pipes to filter and manipulate the RSS feed it's based on but I came across a post on Read Write Web that lists some other online tools that might be worth exploring. ZapTXT looks to be the most interesting but won't send text messages outside the US and Canada at the moment.

In the short term I'll probably carry on using Pipes as I'm now using it's regex features to cut down the message to make it a bit more punchy and readable.

Roll your own traffic alerts using twitter and Yahoo Pipes

2008-03-05T14:55:08+00:00

In the early hours of Tuesday morning there was a three lorry smash on the M5 just north of Bristol resulting in diesel and olive oil being spilt all over the road and a closed motorway. A work colleague sent Nic (my wife) a text message to let her know but unfortunately it arrived just after she had got on the motorway and so the fifteen (ish) miles to work took three hours!

During her three hours stuck on the motorway we had a few calls and of course there were questions about why wasn't there a way of letting people know about traffic problems so they could take alternate routes came up (either via signs before you get on the motorway or via text messages etc.)

A quick Google found a few sites that would send a text message when there were traffic problems but they wanted to charge a minimum of 50p a text!

So ever curious, I wondered how hard could it be to roll a free version of my own?

In the UK, the Highways Agency conveniently provides RSS feeds for traffic incidents - http://www.highways.gov.uk/traffic/11278.aspx - now all that I needed was a way to filter the feed and fire off a text message.

I've played with Yahoo Pipes in the past so it was my first port of call to filter the whole of the UK feed for just incidents affecting the M5. If you're interested the pipe's here: http://pipes.yahoo.com/andydavies/m5traffic (Tim later pointed out that I could skip this step and just use the M5 feed instead, but I might add a date and time to the message so I haven't changed from Pipes yet).

So now the feed contains just the incidents affecting the M5, what ways are there of firing off a text message? A quick email to underscore about RSS to SMS conversion and back came the suggestion of using Twitter and it's API (again, thanks to Tim).

After playing for a while with trying to drive Twitter's API from Yahoo Pipes and deciding against it, I found twitter feed. twitter feed does what it says on the tin - takes a feed and uses it to update a twitter account

So create a new twitter account - imaginatively named UK_M5_Traffic - use twitter feed to link it to the Yahoo Pipe, set my main twitter account to follow it and send me a text message when it's updated and there it is, a free automatic alert system for traffic incidents on the M5.

The first message through was a bit on the useless side as it just used the title from the item and the title doesn't contain enough information about where the problem is, but switching twitter feed to use the description fixed that. As twitter can only display 140 characters, the end of the message gets cut off but there's enough information for it to be useful.

If anyone wants to keep an eye on the M5, the final result is here: http://twitter.com/UK_M5_Traffic