Andy Davies

Adding iOS agents to a WebPagetest Instance

2013-03-05T07:49:00+00:00

Back in September I explained how to create a private instance of Web Page Test running IE, Firefox and Chrome on Windows 7.

Recently I needed to add some iOS agents, after a bit of trial and error this is the approach I used.

I’ve added the mobile agents to my existing ‘all-in-one’ WebPageTest instance so if you’ve not read the original post it’s worth having a quick scan first: Configuring an ‘all-in-one’ WebPageTest Private Instance

At a minimum you’ll need access to a Mac with Xcode and iOS Simulator installed. If you want to use a real device as a test agent you will need an iPhone or iPad and an iOS developer account.

Building the Mobitest Agent

Confusingly there’s some source code for a mobile agent in the WebPageTest SVN respository but as I discovered that’s not the source we want!

We want the code for the Mobiletest agent that Guy Podjarny & Co created at Blaze (now part of Akamai).

Download the Mobitest source https://code.google.com/p/mobitest-agent/source/checkout and build it using Xcode.

Install and Configure the Agent

Using Xcode launch the Mobitest App on the iOS Simulator or a physical device.

Mobitest has a set of default settings that need updating to match the WebPageTest instance, for example:

Default settings on left, settings for my install on right.

Update Jobs URL 1 to match the URL of your WebPageTest instance, and set the Unique Agent Name and Location (location must match the location you configure in WebPageTest).
Unless you are using using keys you’ll need to remove the default Location Key too.
I also enable Restart After Each Job and Auto-Poll (scroll down for these settings)
Go back to the iOS home screen, launch Mobitest, press Poll Now and the agent should now poll the server for jobs.

Add the agent to WebPageTest

Once the test agent is up and running we need to update WebPageTest’s configuration so jobs can be scheduled for the new agent.

Edit settings\locations.ini

Add the new agent under the relevant location section ([Local] and 3=Local-iPhone in the example below)

[Local]
1=Local-URLBlast
2=Local-WPTDriver
3=Local-iPhone
label=Local

Add a section for the agent itself.

[Local-iPhone]
browser=iPhone
connectivity=3G
label="Local"

N.B. Section name must match the name set in [Local] and the Location field of Mobitest Agent settings

Save locations.ini

Check it’s working

I find getLocations.php to be the easiest way of checking agents are configured correctly and requesting jobs.

Go to http://192.168.0.12/getLocations.php (replace IP address with your WebPageTest URL) in your browser and you should see a list of all the current test agents, their status, and the number of jobs in their queues e.g.

List of test agents and their current status (priorities 2-8 omitted from table)

If either Idle Testers or Being Tested columns are greater than zero then the test agent is successfully communicating with the WebPageTest instance.

You should now be able to use the normal WebPageTest interface to run tests on the device you’ve just configured.

Shaping the Connection

On http://webpagetest.org, the mobile agents connect via WiFi to a Linux bridge that shapes the network connection.

For my testing I use the Network Link Conditioner that’s available on iOS devices registered for development (Settings > Developer > Network Link Conditioner).

If you’re using the iOS Simulator for testing then unfortunately the Network Link Conditioner isn’t installed. You can use the OSX version of the Network Link Condition instead but that will shape the connection for the whole of OSX.

In either situation you can force the WebPageTest connection dropdown to only show a single label by adding a connectivity entry to the location in settings.ini

In the example below I’ve added a label that matches the Network Link Conditioner settings for 3G.

[Local-iPhone]
browser=iPhone
connectivity=3G (780/330 Kbps 100ms RTT)
label="Local"

Unlike the PC based agents this setting doesn’t have any influence over the actual connection settings used and they will always need to be configured separately.

Some other things to note

The Mobitest Agent uses a UIWebView rather than Safari, so any JavaScript on the page won’t use the JIT and this may be an issue on Javascript heavy pages. This blog gets roughly the same number of visitors inApp on iOS i.e. UIWebView, as it does from iOS Safari so the UIWebView experience matters.

The Mobitest Agent ignores the physical orientation of the device and works in portrait only.

Pat Meenan reports that an Airport Express seems to be the most reliable base station for iOS devices. He also has a monitor reboots the Mobitest App via SSH when it stops responding.

Questions or Comments?

If you’ve got any questions or spot any mistakes feel free to leave a comment or drop me an email.

If you want to discuss private instances further or need help WebPageTest Forum for Private Instances is the place to go.

Generating HAR files for iOS Safari

2013-02-07T01:04:00+00:00

If you want to generate page load waterfalls for iOS Safari the current options are Mobitest / WebPageTest or Safari on OSX.

Mobitest and WebPageTest are great tools (essentially the same tool underneath) that I often use but they have a few limitations:
- there’s a limited number of test locations so latency can complicate testing
- uses a UIWebView so any javascript in the pages doesn’t have access to the JIT

Safari on OSX is OK but there seems to be no way of generating a HAR file from the waterfall view in developer tools.

These limitations sometimes frustrate me so I decided to write a generator that creates HAR files from Safari running on the iOS Simulator (actual physical iPhones and iPad support will hopefully come too).

The generator is a node application that uses the Safari remote debugging protocol to talk to Safari running on the iOS simulator.

Although the debugging commands for Safari are essentially the same as Chrome’s they’re wrapped up as binary plists in an undocumented RPC protocol.

If you want to know more about the communications side node-iosdriver is the repository you need.

Installation

Clone ios-har-builder
Run npm install to install dependencies

Usage

Load the iOS Simulator, launch Safari and then run the HAR builder from the OSX command line e.g.

bin/ios-har-builder -o example.har http://m.guardian.co.uk

There should now be a HAR file ready for import into HAR viewer etc.

Issues

It’s fresh code, and a bit rough and ready in places so I doubt these are the only issues, if you come across any more feel free to raise an issue on GitHub or better still send me a pull request!

Cookies aren’t currently supported in the exported HAR file (this is just a matter of writing the code)
Code needs cleaning up and refactoring in several places, particularly around the network events in client.js
Host and Port parameters don’t work
Only works on iOS Simulator (for now)
Validation needs to be disabled to display the watefalls in the HAR viewer
Limited timing information due to the lack of resource timing in Safari

Credits

Tom Ashworth and Remy Sharp at Left Logic did a lot of the work involved with being able to communicate with Safari on iOS.

I was going to start from the netsniff.js example included with phantomjs but then Andrea Cardaci released chrome-har-capturer which saved me some time.

For now ios-har-builder is a separate fork as there are some differences in behaviour between Chrome and Safari e.g. lack of navigation / resource timing in Safari, Safari wraps the WebKit debug protocol in RPC etc.

Implementing Sparkicons

2013-01-14T03:00:00+00:00

A few days ago, Mark Boulton bounced around the idea of Sparkicons, small inline icons at the end of links, that indicate what’s on the end of a hyperlink.

Based on meta-data attached to a link, the Sparkicon could indicate the link is to a PDF, or a video, or a discussion thead. It could also include addition information such as how long a video is or how many comments there are in a discussion thread.

This is one way they could be implemented…

Adding meta-data to <a> element

As there aren’t any suitable standard attributes to store Sparkicon meta data in, I used data- attributes:

data-sparkicon - content type e.g. video, music, pdf, comments
data-sparkicon-plus - optional extra detail to display e.g. length, size, number of comments

Displaying the Sparkicon

The Sparkicon is inserted using the CSS :after pseudo selector, so it should degrade gracefully in browsers that don’t have CSS content support:

a[data-sparkicon$="media"]:after {
  content: " >" attr(data-sparkicon-plus);
  font-size:xx-small;
  vertical-align:top;
  font-family: sparkicons;
}

@font-face {
    font-family: "sparkicons";
    src: url("data:font/truetype;base64, base64 font data goes here, 
      but you don't have to use a dataURI..."");
}

WARNING: Supporting webfonts across all browsers isn’t as simple as it might be so while a dataURI is great for demo purposes slightly more complex syntax will be need in real life!

The HTML is straightforward, with the data- attributes added to the anchor element:

<a href="http://www.youtube.com/watch?v=9sSc5ZPK7P8"
  data-sparkicon="media" data-sparkicon-plus="12:02">Rugby World Cup Highlights</a>

The font used has just numbers, a colon and an icon for video (mapped to >).

Ta Da and Boo!

The final result looks pretty good:

The combination of font-size: xx-small and vertical-align: top results in the underline on link under the Sparkicon being disjointed in WebKit browsers (vertical-align: super doesn’t improve things). Firefox 12 and IE9 don’t suffer from this issue.

If you want to look at the result or play with the code it’s on jsFiddle: http://jsfiddle.net/andydavies/GKdDy/

DRYing Out and Fixing the Baseline Problem

Another issue with the example above is that each Sparkicon type requires its own CSS selector and as the number of Sparkicons rises so does the number of CSS selectors.

CSS pre-processors can help manage this and make it maintainable but font ligatures provide another alternative.

Using ligatures for each Sparkicon type e.g. chat, video, music, allows data-sparkicon to be used directly.

a[data-sparkicon]:after {
  content: " " attr(data-sparkicon) attr(data-sparkicon-plus);
  font-family: sparkicons;
}

Creating a superscript font based on GG Superscript Sans simplifies the CSS further and fixes the underline issue in WebKit too.

Using a single CSS selector we can now specify different Sparkicons for multiple links:

Again the code is on jsFiddle http://jsfiddle.net/andydavies/Cg5RQ/

Final Thoughts

I like the idea of Sparkicons, I guess what we need now is real world usage to understand how it affects visitor experience.

This is a prototype, so if you’re planning to take this approach further there’s a few things to keep in mind.

It needs a thorough test across multiple browsers, some browsers don’t support the CSS content pseudo selectors e.g. IE8 but can be polyfilled (or left to degrade gracefully). I tested in on Chrome, Safari and Firefox on OSX and IE9 on Windows (dataURI definition isn’t correct for IE9 though.)

I could’t find any way to keep the underline on the main link but remove it from the Sparkicon, text-decoration: none seemed to have no effect. Somewhat confusingly I could specify a different colour for the Sparkicon though!

In the future it might be possible to use normal fonts and do away with the superscript font. CSS3 Fonts specifies font-variant-position: super for superscript but it’s not implemented by any browser yet.

Finally, icons are really not my forte so it’d be great if someone could produce a demo with decent icons!

Think Twice Before Using matchMedia to Conditionally Load Stylesheets

2012-12-29T21:30:00+00:00

Christian Heilmann recent started exploring how matchMedia could be used to conditionally load resources based on viewport size and other mediaqueries - Conditional Loading of Resources With MediaQueries

I’ve experimented with using matchMedia to restrict the loading of social media buttons to larger viewports so I think using matchMedia to conditionally load content is a great idea but I was less convinced by the benefits of Christian’s example of conditionally loading CSS.

Ilya Grigorik had similar concerns when Scott Jehl created eCSSential, a loader that attempts to optimise the loading of CSS based on viewport size and other factors.

The question is what impact, if any, does conditionally loading CSS have on the performance of sites in real life?

For testing I used a static copy of Enoch’s Fish and Chip Shop as built by Matt Wilcox, hosted on an EC2 micro-instance in Dublin.

Enoch’s is typical of many of the sites showcased on mediaqueri.es as it uses separate CSS files for each breakpoint.

Other than changes to the CSS outlined below, the only other change made was enabling gzip using the configuration from HTML5 Boilerplate.

Three scenarios were tested:
1. Multiple stylesheets with browser determining load order
2. Multiple stylesheets with matchMedia used to load only those that apply
3. Single stylesheet containg all stylesheets merged into one

Each scenario was tested using the iPhone agent on WebPageTest with the median of 9 runs used for comparison.

Default iPhone Behaviour

Webkit based browsers prioritise the loading of CSS. Stylesheets whose mediaqueries match the current viewport as loaded first and any others deferred until later in the page.

The waterfall below demonstrates this split (highlighted in green) between the loading of CSS needed to render Enoch’s homepage and other CSS files.

Waterfall of network activity for Enoch’s Fish and Chip Shop on iPhone produced via webpagetest.org

Conditional Loading of Stylesheets with matchMedia

Unfortunately other browsers don’t behave in the same way as webkit and attempt to load all stylesheets before starting to render the page, so using matchMedia to conditionally load the stylesheets might offer a way to reduce the number of requests and so speed up page load times.

One drawback of using javascript to load CSS is the order resources are retrieved might be altered - the browser isn’t aware of the CSS resources until the script updates the HTML link elements and while the script is executing the lookahead preloader will schedule other resources for download, so the download of these resources may delay downloading the CSS.

This behaviour can be seen in the waterfall below, where other resources (highlighted in red) start to download before screen.320.css, and so delay its download.

Overall the loading pattern Christian proposes results in the page starting to render marginally later (1.80s vs 1.63s) but reaching document complete sooner (7.62s vs 10.94s).

Waterfall of network activity for Enoch’s Fish and Chip Shop on iPhone with stylesheets conditionally loaded using matchMedia.

There are other disadvantages to this approach too:

The sample code needs to be enhanced to allow for changes in viewport size or orientation so mediaqueries on the elements can be re-evaluated and appropriate resources loaded when necessary.

Browsers are continually trying to improve the way they load pages so it’s likely other browsers will adopt the same approach as Webkit and may not even load stylesheets that don’t apply or defer them until after DOMContentLoaded.

Finally not all browsers support matchMedia so some will need to use Paul Irish’s matchMedia polyfill.

Merged Stylesheets

Merging all the stylesheets together into a single downlaod is perhaps the simplest optimisation that could be applied. In this case it also results in the fastest start render (1.57s vs 1.63) and document complete (7.21s vs 7.62s) times.

Waterfall of network activity for Enoch’s Fish and Chip Shop on iPhone with stylesheets merged into one.

Comparison of Tests

Raw timings don’t always give a complete picture, so I always compare tests visually too.

Comparing the tests side-by-side demonstrates the test with the merged stylesheets is both first to start displaying ‘real’ content and also the first to complete.

The test using matchMedia completes before the original site most likely due to a lower number of requests but it is the last to start rendering ‘real’ content.

N.B. Watch out for the unequal timescale!

Filmstrip comparison of the three test scenarios.

WebPageTest also provides a comparison of visual progress and this re-inforces the matchMedia test is the slowest to start displaying ‘real’ content.

Illustration of progress towards visual completion for each test scenario.

Conclusion

In this case the simplest optimisation i.e. merging the stylesheets in one, produces the fastest result.

The combination of less requests and most likely the availability of all styles earlier in the pageload allows the page with the merged stylesheet to start rendering sooner, even though the combined stylesheet (14KB) is slightly larger than the two stylesheets downloaded by the matchMedia test (10.1KB).

Even the original page starts to render before the one that uses matchMedia, with DOMContentLoaded being delayed by the download of stylesheets that aren’t applicable. The WHATWG spec allows for the loading of stylesheets that aren’t applicable to be deferred until they are are needed so further optimisations may be possible.

Based on these tests and the likelihood of other browsers adopting behaviour similar to Webkit’s, I wouldn’t advocate approaches that use matchMedia to determine which stylesheets should be loaded.

Other Considerations

We should ‘poke holes’ in all tests and examine them for weaknesses, this test is no different and there are some areas for further exploration.

CSS is very repetitive (IDs, class names and CSS syntax itself) so it compresses well, dataURIs are the exception to this. The CSS on Enochs doesn’t contain any dataURIs so sites using dataURIs may see different results.

The iPhone test agent uses a UIWebView and execution of the javascript is impacted by the lack of a JIT. Delays in the download of screen.320.css were also seen when the matchMedia test was repeated in Safari on both an iPhone and the iOS simulator. The delays were not a great as those on the WebPageTest agent but they were long enough to allow other resources to start downloading.

The tests need repeating in other mobile browsers e.g. Firefox, Internet Explorer, Opera etc., and for other sites. I attempted to repeat the tests using Android 2.3 but even with a polyfill the test using matchMedia failed (a similar test using eCSSential works, so it’s not quite clear where the issue is at the moment).

Scott has seen performance improvements when eCSSential is used so we need more detail to be able to understand the cases where it makes a difference.

Raw Test Results

If you want to take a look at the raw results yourself, here’s the relevant tests on WebPageTest:

Default iPhone Behaviour
webpagetest.org/result/121228_66_a9708983b3a994bc3345e4f66858c59c/

Conditional Loading of Stylesheets with matchMedia
webpagetest.org/result/121228_BA_52d204278ed082e081c4cdffac110358/

Merged Stylesheets
webpagetest.org/result/121228_JZ_c100646a583ed1d3789db3d4e37629c5/

Visualising Webpage Resource Dependencies

2012-10-24T14:37:00+01:00

Waterfall charts are great but they show a timeline rather than the relationships between the resources on a page and I often want to understand what triggered a resource to load.

At Velocity EU, Rajiv Vijayakumar of Qualcomm demonstrated a tool they’re building to map the resource dependencies in webpages highlighting blocking paths etc.

Currently it’s not clear whether Qualcomm are going to opensource the tool, or run it as a service (like WebPageTest), so during a moment of curiosity I attempted to produce something similar using WebKit dev tools and Mike Bostock’s D3.js.

The results are interesting but unfortunately not quite right, as this chart for The Guardian demonstrates:

The green nodes in the above chart are CSS files which include some definitions with background images but you might notice there’s no images (purple nodes) linked to them.

This is due to Chrome reporting the initiator as the HTML line that uses the selector and so triggers the download e.g. in the example below the a element would be reported as the initiator.

#logo { background: url('/content/images/web/logo.png')' }

<a href="http://andydavies.me/" title="Logo" id="logo">Logo</a>

Marco Tolk has found other scenarios where Chrome reports the initiator incorrectly so we’re putting together a set of test cases to determine what works and what doesn’t (for our needs).

I’ve got lots of ideas on how the charts could be enhanced but I’m holding off until the data is right, in the meantime I’m back in C++ land debugging and exploring what information Chrome can give me.

If you want to try it out for yourself the code I used to generate the charts is on GitHub.

Configuring an ‘all-in-one’ WebPageTest Private Instance

2012-09-18T12:51:00+01:00

WebPageTest (WPT) is a great tool and the public instances cover many scenarios but sometimes I need a private instance, for example when testing sites that aren’t publicly accessible yet or debugging a script.

Setting up a private instance is reasonably straightforward but there are a few gotchas and some settings aren’t always obvious the first time around.

I’ve been asked by a few people how I set up instances, so these notes explain the steps I follow to create an ‘all in one’ installation with the server and test agents on same machine.

The notes are based on WPT install guide and can also be used as a guide to creating separate a server and agents, although I’d use a Linux variant for the server in that case.

Base Windows Configuration

Start with a fresh Windows 7 install (I used 32-bit Professional, the virtual machines Microsoft make available for browser testing are another option but stick with 32bit)

Create a user with local administrator rights and set them to log in automatically when the machine starts up. (this has security implications and some network managers may not like it)
Turn off UAC
Turn off any screensaver
In the power saving settings choose never as the options for turning off the display and putting the computer to sleep
Set firewall rules to allow port 80 & 443 TCP inbound
Add shortcuts for Shutdown and Restart

When accessed via Remote Desktop (RDP), Windows 7 doesn’t display the Restart and Shutdown options on the Start menu, so I add shortcuts for them:

Shutdown : shutdown -f -s -t 1

Restart : shutdown -f -r -t 1

I also add Notepad++ and 7zip so I have a decent text editor and archiving tool

Server Installation

Install Apache

Download and install Apache. Apache 2.2 is currently the most recent version with a Windows binary.
Edit C:\Program Files\Apache Software Foundation\Apache2.2\conf\httpd.conf
Update the configuration to include the following:

DocumentRoot "C:/wpt-www/"

<Directory />
    Options FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>

<Directory "C:/wpt-www/">
        AllowOverride all
        Order allow,deny
        Allow from all
</Directory>

<IfModule dir_module>
    DirectoryIndex index.html index.php
</IfModule>

Uncomment these modules

LoadModule expires_module modules/mod_expires.so
LoadModule headers_module modules/mod_headers.so
LoadModule rewrite_module modules/mod_rewrite.so

Add the PHP module (installed in next step)

LoadModule php5_module "C:\Program Files\PHP\php5apache2_2.dll"

Install PHP

Download PHP from http://windows.php.net/download (I used the zip of 5.4.7 VC9 Threadsafe)
Extract and copy contents to c:\program files\php
Copy c:\program files\php\php.ini-production to c:\windows\php.ini
Edit c:\windows\php.ini

Update the extension_dir path, and uncomment the php_gd2 extension

extension_dir = "C:\program files\php\ext"
extension=php_gd2.dll

(zip and zlib are on by default in 5.4.7)

Install WPT server

Download the latest version (2.7.2 at time of writing) of WPT from https://sites.google.com/a/webpagetest.org/docs/private-instances/releases
Extract the contents of the www folder from webpagetest.2.7.2.zip into c:\wpt-www
Copy c:\wpt-www\settings\connectivity.ini.sample to c:\wpt-www\settings\connectivity.ini
Add test agent locations / configurations

Editing locations.ini is the probably the area I make the most errors and it can be confusing until you grasp it’s hierarchical nature.

The browser configuration labels must match the labels used in urlblast.ini and wptdriver.ini in the test agent configurations.

I name my labels as location-testdriver pairs e.g. local-urlblast (for IE) or local-wptdriver (for Chrome and Firefox)

copy c:\wpt-www\settings\locations.ini.sample to c:\wpt-www\settings\locations.ini

edit c:\wpt-www\settings\locations.ini

This is a sample config that matches the test agents below, more locations can be added:

[locations]
1=Local
default=Local

;
; These are the top-level locations that are listed in the location dropdown
; Each one points to one or more browser configurations
;

[Local]
1=Local-URLBlast
2=Local-WPTDriver
label=Local

;
; These are the browser-specific configurations that match the configurations
; defined in the top-level locations.  Each one of these MUST match the location
; name configured on the test agent (urlblast.ini or wptdriver.ini)
;

[Local-URLBlast]
browser=IE 9
latency=0
label="Local - IE 9"
;browserExe=pagetest.exe
;key=TestKey123

;
; For a wptdriver configuration (chrome, firefox), the browser labels here 
; MUST match the labels used in wptdriver.ini
;

[Local-WPTDriver]
browser=Chrome,Firefox
label="Local"

Restart Apache
Fire up a browser and navigate to 127.0.0.1 and you should see the Web Page Test Private Instance home page.

Test-Agent Configuration

Extract the contents of the www folder from webpagetest.2.7.2.zip into c:\wpt-agent
Install Chrome and Firefox (if you want them of course)
Install ffmpeg (http://ffmpeg.zeranoe.com/builds/) and add its bin folder to the path. I tend to install it in c:\program files\ffmpeg
Install AVISynth using the installer in c:\wpt-agent
Install the DUMMYNET ipfw driver

WPT 2.10 introduced a 64bit version of dummynet. Depending on your platform choose the version of dummynet you require and copy the contents of the 32bit or 64bit folder into c:\wpt-agent\dummynet.

In the properties of the Network Adapter used for Internet access
Click “Install”
Select “Service” and click “Add”
Click “Have Disk” and navigate to c:\wpt-agent\dummynet
Select the ipfw+dummynet service (and click through any warnings about the driver being unsigned)

If you’re connected via RDP you may lose the connections while installing ipfw plus dummynet and need to re-connect.

In the startup folder, create shortcuts to c:\wpt-agent\dummynet\ipfw.cmd, c:\wpt-agent\urlblast.exe and c:\wpt-agent\wptdriver.exe
Configure IE

Copy c:\wpt-agent\urlblast.ini.sample to c:\wpt-agent\urlblast.ini

Edit c:\wpt-agent\urlblast.ini

The Location entry must match the location label for the IE section of locations.ini

[Configuration]
Startup Delay=30
Log File=c:\wpt-agent\agent\urlblast
Timeout=120
use current account=1

; Where to get work from
Url Files Url=http://127.0.0.1/work/
Location=Local-URLBlast
;Location Key=TestKey123

Configure Chrome and Firefox

Copy c:\wpt-agent\wptdriver.ini.sample to c:\wpt-agent\wptdriver.ini

Edit c:\wpt-agent\wptdriver.ini

Again the location entry name must match the revelant section in the server locations.ini

I update the software paths to retrieve updates from the local machine rather than webpagetest.org but that’s a personal preference

[WebPagetest]
url=http://127.0.0.1/
location=Local-WPTDriver
browser=chrome
;key=TestKey123
;Automatically install and update support software (Flash, Silverlight, etc)
software=http://127.0.0.1/installers/software.dat

[chrome]
exe="C:\Program Files\Google\Chrome\Application\chrome.exe"
options='--load-extension="%WPTDIR%\extension" --user-data-dir="%PROFILE%" --no-proxy-server'
installer=http://127.0.0.1/installers/browsers/chrome.dat

[Firefox]
exe="C:\Program Files\Mozilla Firefox\firefox.exe"
options='-profile "%PROFILE%" -no-remote'
installer=http://127.0.0.1/installers/browsers/firefox.dat
template=firefox

Clearing prompts and warnings

Launch each browser and clear the prompts e.g. should it load extensions, should it be the default browser etc.

From the startup folder launch urlBlast, wptdriver and ipfw.cmd manually. Give each of them permission to run even though they’re unsigned, and check the ‘always do this box’

Restart the machine

Whenever you RDP into a WPT test agent, you should always restart the machine as the Window lock screen is displayed when disconnecting RDP.

You should now have a working ‘all in one’ install of Web Page Test, point your browser at the WPT machine and launch tests for IE, Firefox and Chrome just to check they work.

If you’ve got any questions or spot any mistakes feel free to leave a comment or drop me an email.

There’s also a troubleshooting guide for the most common problems at the foot of the main WPT installation instruction (https://sites.google.com/a/webpagetest.org/docs/private-instances)

If you want to discuss private instances further or need help WebPageTest Forum for Private Instances is the place to go.

Updates

30 Apr 2013 - Added 32bit / 64bit dummynet details

What if we could use CSS to alter HTML attributes?

2012-08-13T09:34:00+01:00

As the ideas behind responsive web design mature, and our skills and experience with them grow, we’re faced with a gap between what the tools and technologies can do and what we’d like them to do.

We often end up trying to fill the gap with javascript and using it not just for behaviour but to change appearance and content too.

Javascript might be the right tool for many things, but should we really be embedding attributes that control appearance in our scripts, or might a declarative approach that uses CSS to change HTML attributes be clearer?

Syntax

A syntax that allows HTML attributes to be changed via CSS might look something like this:

HTMLattribute : [operator] value

e.g.

HTMLattribute :  value        // set value
HTMLattribute : +value        // add value e.g. to lists such as class, rel
HTMLattribute : -value        // remove value from list
HTMLattribute : ^value        // toggle value - remove if there, add if not

The syntax needs improving, for example the HTML attributes probably need prefixing to prevent clashes with CSS properties that share the same name.

I’m also not sure about the list operators, but the syntax is good enough to illustrate some of the ways the idea might be useful.

Classes as hooks for javascript e.g. conditional loading

The class and rel attributes sometimes get used as hooks for javascript but what if we want to add a hook that depends on viewport size?

We could use matchMedia(), but as Jeremy Keith points out in Conditionally loading content, using matchMedia() requires maintaining breakpoints in more than one place.

Jeremy eventually chose the option of generating a hidden marker on the body element using :after (see Conditional CSS), it works but it’s a bit of a hack.

What if we could just add a class directly using CSS and use that as the hook for our javascript?

/* add class big-screen to html element */
  
@media all and (min-width: 1024px) {
  html {
      class: +"big-screen";
  }
}

Data- attributes for configuration of widgets e.g. tweet buttons etc.

jQuery mobile, social media buttons and other widgets are increasingly using HTML5 data- attributes for configuration.

The tweet button has options such as big buttons vs little buttons, counts, count position etc., all controlled via data- attributes.

In smaller viewports we might want to make the tweet button larger so it’s easier to touch, and free up some room by hiding the count.

/* set the data- attributes that makes button bigger and removes tweet count */
  
@media all and (max-width: 480px) {
  .twitter-share-button {
      data-size: "large";
      data-count: "none";
  }
}

Swapping content

The srcset / picture proposals will help resolve the responsive images problem, so we probably don’t need another solution but as a use-case it shows how updating HTML attributes could be used to swap content, while still keeping all the content in HTML.

<!-- img element that uses data- attributes to declare images, keeping content in HTML -->

<img src="small.jpg" data-src-med="medium.jpg" data-src-lge="large.jpg">

/* use media queries and attr() function to replace the small image if appropriate */

@media all and (min-width: 321px) and (max-width: 800px) {
  img {
      src: attr(data-src-med);
  }
}

@media all and (min-width: 801px) {
  img {
      src: attr(data-src-lge);
  }
}

This is just an illustration as an approach like this for responsive images wouldn’t enable the UA to use network connection throughput to pick the most appropriate image that the other proposals do.

Feedback

At the moment this is just an idea, an idea which almost certainly has plenty of issues, so what I’m really looking for is feedback.

Is it a useful idea, how would you use it, how could it be improved, what issues are there implementing it etc. or have I just been drinking too much coffee?

Cascading Attribute Sheets

Added: 24 Aug 2012

Tab Atkins and Fantasai have come out with a similar proposal for Cascading Attribute Sheets

The syntax is pretty similar but Tab and Fantasai suggested another way of adding to an existing property, so with their syntax my first example would look like this:

/* add class big-screen to html element */
  
@media all and (min-width: 1024px) {
  html {
      class: attr(class) big-screen;
  }
}

The proposal and comment are well worth a read.

Rebooting my blog

2012-08-08T14:41:00+01:00

I’ve haven’t written a post in ages, partly because I’m really bad at finishing posts, and partly because I’ve never really like typepad and so haven’t paid for it this year.

Twitter is great for short conversations but it’s not up to longer posts and I’ve got some ideas I want to share so that others can kick the tyres.

So I’ve switched domains, fired up Octopress, and with the help of pandoc ported the old blog content across. Once I’ve got the redirects in place I will be killing off the old blog.

Octopress was really pretty easy to get going, but I’m not quite sure if I’m going to stay with it as I want to experiment with some RESS type things.

I’ve still got some pages to write, some tweaks to the theme I want to make but it’s good enough for now.

Web Performance - A Whistlestop Tour

2012-01-24T00:00:00+00:00

The slides from my recent talk at DigiTalks Cheltenham:

Web Performance - A Whistlestop Tour

Increasing the TCP Initial Congestion Window on Windows 2008 Server R2

2011-11-21T00:00:00+00:00

In November 2010, Ben Strong highlighted how Google and Microsoft were cheating on TCP Slow-Start by setting the initial congestion window higher than the RFC value.

Since then there’s been quite a few articles discussing the performance benefits of increasing the TCP initial congestion window (initcwnd). Most of the articles showed how to make the changes on Linux but there was no coverage on whether and how it could be increased on a Windows Server.

In one of their recent posts, Tuning initcwnd for optimum performance, the guys at CDN planet mentioned they didn’t know how to change the settings on Windows either.

Having trawled through the various options in netsh, I was coming to the conclusion it wasn’t possible but Drit Suljoti of Catchpoint pointed me in the direction of MS Hotfix KB2472264 “You cannot customize some TCP configurations by using the netsh command in Windows Server 2008 R2”.

After installing this hotfix I was able to change the initial congestion window to 10 (it can be adjusted up to 16 * MSS).

Installing and configuring

Request the Hotfix via http://support.microsoft.com/kb/2472264
Download and install the relevant version - it’s available in x32, x64 and ia64 flavours
Check the hotfix is installed ok

Type:

c:\netsh interface tcp show supplemental

and you should see:

The TCP global default template is internet

TCP Supplemental Parameters
----------------------------------------------
Minimum RTO (msec)                  : 300
Initial Congestion Window (MSS)     : 2
Delayed Ack Timeout (msec)          : 200

If you get the following then the hotfix hasn’t been installed:

The following command was not found: interface tcp show supplemental.

Update the initial congestion window settings and switch to the custom template

First set the initial congestion window in the custom template:

c:\netsh interface tcp set supplemental template=custom icw=10

Then tell Windows to use the custom template instead of the standard internet one:

c:\netsh interface tcp set supplemental template=custom

Your server should now be running with the new initcwnd value, so pull out your favourite packet sniffer - Wireshark, tcpdump etc., and test.

Performance

So far I’ve only tested this configuration on an EC2 instance and haven’t deployed it in live production environment yet.

Cursory performance tests show a great improvement downloading the default welcome.png that comes with IIS 7.5 - from 11.7s to 2.5s with a server in Singapore and a client in the UK.

I’m not sure I quite believe the 11.7s time so over the next week I’ll try some more tests with various size files and post the results.

Before (icw=2):

$ time curl ec2-175-41-175-250.ap-southeast-1.compute.amazonaws.com/welcome.png > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  180k  100  180k    0     0  15912      0  0:00:11  0:00:11 --:--:-- 20785

real    0m11.706s
user    0m0.008s
sys     0m0.015s

After (icw=10):

$ time curl ec2-175-41-175-250.ap-southeast-1.compute.amazonaws.com/welcome.png > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  180k  100  180k    0     0  75765      0  0:00:02  0:00:02 --:--:-- 93690

real    0m2.532s
user    0m0.007s
sys     0m0.015s

Final Notes

If you’re thinking of deploying the fix to a production server it’s probably worth reading the proposal submitted to the IETF on increasing the default value to 10 as it covers both the advantages and potential drawbacks with an increased default.

Jim Gettys also points out some of the potential issues particularly in relation to the possible effects on network equipment.

Testing how slow third party components affect your page load times

2010-10-01T00:00:00+01:00

Both Steve Souders and Joshua Bixby have highlighted the effect slow third party components can have on overall page load time.

Third party components have become common e.g. ads, links for bookmarking, sharing, widgets that pull in content from other sites etc, and the impact on our page load times depends on the quality of service they offer (both how the component is implemented and the reliability/speed of their service)

Steve and Joshua have highlighted approaches that can be used to decouple the components and reduce their impact on the performance of our pages but it’s early days and only some providers have implemented them (Sergey Chernyshev has a some thoughts too).

The challenge for web developers is understanding what the effect on page load times could be, when most of the time the providers’ networks remain reliable.

Charles - a bandwidth limiting proxy

To test the impact providers can have if they perform poorly we need to make the network connections to them unreliable and my tool of choice for this sort of situation is Charles (http://www.charlesproxy.com/)

Charles allows both low bandwidth and high latency to be simulated, either on an all sites or individual site basis.

To enable throttling choose Proxy -> Throttle Settings and check the first box, the same dialog also enables configuration of bandwidth and latency constraints, and the URLs they apply to.

On the sites I tested, I throttled using high latency for each provider individually - as an example here’s one for addthis.com with the latency set to ten seconds.

I also used the normal waterfall tools (Firebug and WebKit inspector) to see when onload was being fired at, so I could experiment with asynchronous loading to mitigate the effect of the latency.

Look out for subtle failures!

One webapp I tested used the widgets from addthis.com and even with long latencies set, the pages appeared to load quickly, but further testing found the tour was broken - clicking the button just replaced the current page with an image

The tour used lightbox to display a series of images, the delay in downloading the javascript from addthis prevented the onload event from being fired and so the lightbox code wasn’t being inserted into the page.

Charles works on Windows, Mac and Linux, and has a free trial, so give it a try. Find the components that cause problems for you and raise the issue with their providers.

The Need For Speed... why slow page load times matter

2010-09-20T00:00:00+01:00

About a week ago, I did a presentation at BarCamp Stroud on why page load times matter and things that can be done to improve them.

It was fun to present and share what I’d learnt on a topic I’d spent a lot of time investigating and implementing for a WebApp I run (got quite a few things still left to do)

Here’s the presentation for anyone who’s interested, and there will probably more as I’ve already made some notes on other things to cover.

The Need For Speed from Andy Davies

.net and ISA Server - reducing bandwidth consumption

2010-07-18T00:00:00+01:00

Earlier this year I came across some problems with some .net resources not being cached by ISA Server (.net and ISA Server - a match made in hell?)

The problem in a nutshell is that by default ISA Server doesn’t cache URLs that have query parameters, and the method .net uses to serve static resources that are bundled in .dlls relies on query parameters.

When .net uses the Microsoft AJAX framework, script elements like the one below are inserted into the page

<script src="ScriptResource.axd?d=ki7GLYn2P5z-CEtE0SsvKYJhnTEkD13edwjg24cmxWe4aD03WzKyGXQD45nYCFy70&t=633178683580000000" type="text/javascript"></script>

After looking at some of the lastest developments in .net 4.0 I discovered that ScriptManager could be used to serve out static versions of the AJAX framework (even in versions of .net before 4.0).

After a few attempts I came up with this code that takes the ScriptResource.axd references in the page and replaces them with static scripts instead, which are of course generally cachable by ISA Server

<asp:ScriptManager ID="Script" runat="server" EnablePartialRendering="true">
  <Scripts>
      <asp:ScriptReference Name="MicrosoftAjax.js" Assembly="System.Web.Extensions" Path="~/js/ms/MicrosoftAjax.js" >
      <asp:ScriptReference Name="MicrosoftAjaxWebForms.js" Assembly="System.Web.Extensions" Path="~/js/ms/MicrosoftAjaxWebForms.js">
  </Scripts>
</asp:ScriptManager>

(the release i.e. minified, versions of the scripts within the MS AJAX framework have been placed in /js/ms)

By default the scripts won’t be served gzipped, or with the long expiry times that ScriptResource normally serves them with but this can be addresses by configuring IIS (which is how it should be in my view)

We’re still measuring how much this improves our page load times and how much it reduces our bandwidth consumption but it looks to make a significant difference.

If you’re interested Dave Reed has written more on what’s possible with ScriptManager in .net 4.0 - ASP.NET 4.0 ScriptManager Improvements

.net and ISA Server - a match made in hell?

2010-01-19T00:00:00+00:00

I’ve been looking at the bandwidth consumption of a .net based webapp and working with the developers to implement many of the common recommendations for improving front-end performance - minifying, gzipping, cache-control headers etc - we’ve noticed some improvements in performance but we’ve also noticed some oddities.

.net has the ability to embed static resources e.g. javascript, css etc., in a .dll and then these can be served to the browser using a special handler - ScriptResource.axd for the Microsoft AJAX framework and WebResource.axd for others.

Typically these appear in a web page looking something like this:

<script src=“/WebResource.axd?d=D_qnz75P-Xnx_WWFT0nC0A2&t=633926346394042220” type=“text/javascript”></script>

<script src=“/ScriptResource.axd?d=ki7GLYn2P5z-CEtE0SsvKYJhnTEkD13edwjg24cmxWe4aD03WzKyGXQD45nYCFy70&t=633178683580000000 type=”text/javascript"></script>

(The d in the query string uniquely identifies the resource, and t is a timestamp)

The WebForms javascript that’s served via WebResource.axd, hasn’t been minified or gzipped but at least it has some cache-control headers. The scripts served out but ScriptResource.axd are somewhat better, the release versions are minified, gzipped if the browser supports it, and served with a long expiry time, so should be ideal for caching.

Why Microsoft chose URL handlers instead of nice plain and simple static files is beyond me (but then most of the .net framework seems to be about over complicating simple things)

The traffic to our production servers is monitored via Oracle REUI which allows us to slice and dice the information in it’s cubes so we can see response times, which URLs are requested most frequently, what’s consuming the most bandwidth etc.

After making several changes to try to reduce bandwidth consumption, ScriptResource.axd was still at the top of the pile of URLs by bandwidth and it wasn’t clear why until we started looking at our users environments in a bit more details.

Most of our customers are schools and they generally have a gateway cache of some form, increasingly this appears to be ISA Server.

By default ISA Server removes the accept-encoding header so users will get the uncompressed versions of scripts and other compressible resources (Steve Souders covers this in Even Faster Websites ). There are some scripts around that will change this setting but it appears not many people have implemented them.\ \ Also ISA Server doesn’t cache URLs with query string parameters (I can see the point of this) but it means that by default all those .net resources that are served using WebResource.axd and ScriptResource.axd aren’t being cached.

There are ways of using the asp:ScriptManager control to replace the ScriptResource.axd URLs with references to static files but there appears to be no way of doing that for WebResource.axd

It appears Microsoft have managed to develop a web framework and a proxy caching product that don’t work well together… the words arse and elbow spring to mind…

(Over the next few weeks I’ll be spending some time looking further into this and also comparing the ISA Server behaviour with the Squid based proxies that some of the other schools in the UK have. I’ll report back when I know more)

From RSS to delicious via ping.fm and twitterfeed

2009-01-29T00:00:00+00:00

One of my continuing frustrations in the love/hate relationship I have with delicious is there’s no easy way of getting items from an RSS feed into delicious.

I’ve got stuff I’ve ‘liked’ in FriendFeed, starred items in Google reader, favourites on Slideshare, Twitter etc, that I want to pull directly into one place and that place is delicious, but it seems delicious aren’t interested in making my life easy.

Apart from bitching on the delicious forums, I’ve been fiddling with various ways of filling the gap for a while and finally I think I’ve found a way of partially filling it using ping.fm and twitterfeed. It’s got a few limitations but it may be good enough for now.

Here’s a list of the steps needed to get things up and working:

If you don’t have a ping.fm account already, head over and create one
Add your delicious account so that ping.fm can post to it (warning you have to give them your delicious password)
Add some form of micro-blogging/blogging account too. This is needed for the link from twitterfeed to work, I used tumblr (but again I have to give my tumblr password, has no-one heard of OAuth?)
Now head over to twitterfeed, and create an account if you need one.
Create a new feed for ping.fm

You’ll need your ping.fm API key which you can get from ping.fm/key, and you’ll also need to pick a posting method - I chose ‘microblogging’ ( but I wasn’t able to make any choice until I added the tumblr account)

Add in the details of the feed you want to push into delicious along with how often the feed should be checked - I choose to have the feeds checked as often as possible and have as many items as I can pushed across.

By default twitterfeed shortens URLs using tinyurl, you can either stick with this or choose an alternative (I used bit.ly, but that’s just a personal preference)

Finally in the field ‘Prefix each tweet with’ add ‘@de’, this tells ping.fm to forward the item to delicious

Repeat the above step for as many feeds as you want to pull into delicious

Hopefully, if everything is right you should soon have items arriving in your delicious account.

The approach does have a few drawbacks though:

twitterfeed shortens the URLs, and delicious has no way of expanding them, so the tags from other users can’t be applied to bookmark. The original URL being destroyed will also be a problem for people who use delicious to keep track of others who bookmark their blog posts.
The first time twitterfeed does it’s thing it only collects the most recent RSS item so all the other items remain uncollected.
twitterfeed only collects five items at time from each feed so it may be possible to build up a backlog if you added a lot of items to your feed
You have to give ping.fm your delicious login details and probably login details for another service too
The bookmarks bypass my delicious inbox and I’m not sure whether this is a good thing or not - I don’t always want to keep all the items I like of Friendfeed, or star in Google Reader so having them arrive in the inbox for a final last filter seems like a good idea
If a item already has tags in the feed delicious can’t make use of them.

As a word of warning I’ve also got no idea what happens if one of the items already exists in my delicious collection and ping.fm tries to add it from a feed, so perhaps I need to check that.

It’s an imperfect solution but I’m going to give it a go for a while to see how it works, ultimately this is something delicious and other social bookmarking sites should provide, indeed if one of the other sites introduces the ability to add bookmarks from RSS I’m likely to jump ship and leave delicious behind.

Finally if you do decide to go down this route think about making a donation to twitterfeed, Mario offers a great service for free.

(Oh and thanks to Scoble, whose broadcast of his plan for world domination led me to checkout ping.fm)

If anyone comes up with any improvements or alternative ways of tacking this, I’d be very interested in hearing them.

Alas, twitter cripples SMS in the UK (and elsewhere)

2008-08-14T00:00:00+01:00

Woke up this morning to an email from twitter saying they were scrapping outbound SMS messages for the portion of the world that charges for sending rather than receiving text messages i.e. anywhere that’s not the US, Canada or India.

The same message is on their blog too - Changes for Some SMS Users—Good and Bad News - and despite the title there was no good news in the post, only the vague promise that they’d introduce new local SMS numbers for unspecified countries within Europe in the coming weeks and months.

I can understand why they’ve done this - the costs of sending all the messages must burn through money - but by removing the service before a replacement is available I think they’ve screwed up.

Unlike the US, Canada and India, most of the world charges people for sending text messages but they also offer a way to charging people for receiving messages via SMS reverse billing.

In my view, twitter should have had this option up and ready to go before scrapping their existing SMS service. I guess the danger for them is that their users will continue to migrate to other service (http://identi.ca/ for example) or other ways of being notified e.g. RSS on phones, damaging their future revenue streams.

Perhaps the fact twitter are US based limits their understanding as to how people use mobile phones and how the networks work in the rest of the world?

Also as twitter is pretty open (via rss and the API), perhaps there’s also an opportunity for someone to set up a third party service to provide and charge for SMS updates?

Of course this screws up my traffic bot (uk_m5_traffic) as it relies on twitter to send out the SMS notifications. In the short term I can set up the feed reader on my phone to update from the RSS feed but longer term I guess I’ll need to see what twitter do and what other options I’ve got.

Have you ever launched a product you've been ashamed of?

2008-05-22T00:00:00+01:00

In a few weeks time the product I’ve spent the last eighteen months on will launch. I’ve built and launched products before and despite their quirks and issues I’ve been proud of them all, but this time it’s different, this time I have an almost overwhelming feeling of sadness.

I’d describe myself as a pragmatic perfectionist (I like things done right but the closer the deadlines get the more pragmatic I become), but this time I seem to have run out of pragmatism.

Part of me (I’m not sure how much) doesn’t care about this product any more and I find that really sad as I care about what I do but the effort it’s taken to get it this far and the frustrations involved in development have taken their toll. When I told my wife about how I felt she just put her arms around me as she understood the magnitude of what it meant (no chance of me ever complaining that my wife doesn’t understand me!).

Functionally the product appears to work OK, but there’s so much that’s clunky and inconsistent about the way it’s been built, behaves and looks, and in the time we’ve had for development many of them could have been addressed.

The problem as I see it is that the software development side of the product was outsourced and I don’t think the company I work for picked the right partner or have learnt from their previous experiences with this partner.

In general I believe product development is far too important to outsource, as “no-one cares about your product like you do”. I knew the development was being outsourced when I joined, but at the time I just finished a contract job and needed another so was in the position of “beggars can’t be choosers”

The product had already had a few false starts before I arrived but between us we finally got it really moving early last year some of the early prototypes were promising but the immaturity of our partner quickly started to show through.

Their web development techniques seemed to be stuck at around five years ago, all tables and no semantic markup or styles, where there were styles they tended to be in-line rather than an in a stylesheet so you could never be sure that a change in style would be reflected across the product. Their javascript was quite frankly appalling (badly structured and indecipherable) and I’m sure they wouldn’t have accepted the same quality of code in the web-services components of the product.

I’ve spent the last year teaching a development team about everything from web development techniques (semantic markup, how to use css, why they should use a library like jquery (or prototype etc.) instead of writing their own DOM manipulation code), through web infrastructure (how caching can make a huge difference to the performance of our product - we’re serving quite a lot of media rich content, the role load-balancing has in scaling and high availability) though to how we can’t make users decisions for them (if the user wants to delete something, let them just warn them of the consequences before they do).

Even now I’m not sure how much of what I’ve tried to teach them has stuck, sometimes I see glimmers that it has and sometimes I see the old ways are back, all I know is that this product has just been all take with no return and I need something back, so it’s time to find something else to move on to…

(I will try to write more about this product, what’s gone wrong and what I think I’ve learnt in the next couple of weeks, just need a bit more time to reflect first)

Feed readers and social bookmarking - a missing link?

2008-04-14T00:00:00+01:00

Why isn’t there an easy way of getting items from a feed reader into a social bookmarking service?

Like many people (or is it just the geeks?) I use Google Reader to keep up-to-date with news and opinions from around the web, and del.icio.us to bookmark the interesting things I come across. A couple of weeks ago MichaelSearcy’s analysis of Google’s Broken Bookmarking Systems popped up and got me thinking about how I could get more out of the Reader/del.icio.us combination and what it would take.

In Reader I tend to mark items I want to read later or keep for the future with a star, sometimes I get around to reading them again, and sometimes I get around to adding them to del.icio.us but sometimes despite my best intentions I don’t and slowly I’m building up two parallel sets of items I want to keep, one in Reader and one in del.icio.us.

What I need is an easy way to get items I want to keep from Reader into del.icio.us - currently I either have to open the item up in a new browser tab and then use the del.icio.us toolbar to add it, or use a Greasemonkey script to add the ability to post straight to del.icio.us from Reader and it’s not always how I want to work.

What I want to be able to do is mark something as shared in Reader and have it appear in my del.icio.us inbox and from there I can choose to save it (being able to delete items from the inbox would be helpful too).

Reader already produces a feed for shared items (and other the labels if you know how to get at them), but del.icio.us doesn’t have any way of consuming feeds and they don’t seem too keen on the idea of adding it. I’ve had a brief scan of some of the other social bookmarking sites and they don’t seem to support getting bookmarks from a feed either, so perhaps there’s an opportunity for someone?

For the second part of the puzzle I’d like a way of having any tags I attach Reader to be included in the feed, this is something del.icio.us does with its feeds so Reader could do it via something like xFolk microformat, then not only will my shared items arrive in my del.icio.us inbox but they’ll also arrive with tags already attached.

Data portability and how private data can be shared between social services is a hot topic but right now, I’d settle for services that can share my public information!

Ideas for improving my twitter traffic bot

2008-03-25T00:00:00+00:00

The bot I built for keeping me up-to-date with traffic conditions on the M5 has proved handy a few times but it also suffers from a couple of drawbacks…

It only covers the M5, and the whole of the M5 at that, if someone wants updates on the M4, or just part of the M5 there’s no way for them to choose the information that’s relevant to them.

TwitterFeed also introduces a time delay and even on it’s most frequent setting it will only check a feed every 30 minutes - as we found out on Friday morning when we ran into a jam just north of the M4/M5 interchange only to hear our phones get the text message telling us there was a hold up!

So I’ve been thinking about building a new bot that would allow someone to choose the roads (and parts of) they want to receive updates for.

The bot would allow people to send it messages about the roads they’re interested in, grab the traffic feed from the Highways Agency (and perhaps other feeds) on a regular basis, and when there’s something to tell people message them via twitter.

I’m thinking of something that’s quite similar to mytrade’s twitter bot in concept but needs to be more frequent in it’s updates. The language used to communicate with the bot also needs to allow richer concepts as in stock watching terms it needs to support not just broadcasting prices, but broadcasting when a price crosses a user set threshold.

The things I’m looking at right now are:

The syntax for communicating with the bot - something similar to the syntax of mytrade’s bot might be good enough but I need to have a hunt around to see what alternatives are about. (I’m not sure I like twitter’s format for direct messaging and wonder whether !username might be fit better with the @username format used elsewhere on twitter, but there’s not a lot I can do about that!)

Whether to use the REST API or Jabber / XMPP for communicating with twitter. The API pages suggest using Jabber / XMPP for bots that use direct messages so perhaps I’ll follow their advice to begin with.

So time to brush off my coding skills and as I (hopefully) make some progress I’ll add some updates here.

Some other online tools for filtering RSS

2008-03-07T21:35:00+00:00

UK_M5_Traffic currently uses Yahoo Pipes to filter and manipulate the RSS feed it’s based on but I came across a post on Read Write Web that lists some other online tools that might be worth exploring. ZapTXT looks to be the most interesting but won’t send text messages outside the US and Canada at the moment.

In the short term I’ll probably carry on using Pipes as I’m now using it’s regex features to cut down the message to make it a bit more punchy and readable.