Robin Minto

Software development, security and miscellany

IIS configuration file contention

I’ve been automating the configuration of IIS on Windows servers using PowerShell. I’ll tell anyone who’ll listen that PowerShell is awesome and combined with source control, it’s my way of avoiding snowflake servers.

To hone the process, I’ll repeatedly build a server with all of it’s configuration from a clean image (strictly speaking, a Hyper-V checkpoint) and I’m occasionally getting an error in a configuration script that has previously worked:

new-itemproperty : Filename: \\?\C:\Windows\system32\inetsrv\config\applicationHost.config 
Error: Cannot write configuration file

Why is this happening? The problem is intermittent so it’s difficult to say for sure but it does seem to occur more often if the IIS Management Console is open. My theory is that if the timing is right, Management Console has a lock on the config file when PowerShell attempts to write to it and this error occurs.

Disabling contactless payment cards, or preventing “card clash” with Oyster

I’ve been carrying an RFID-based Oyster card for years. It’s a fantastic way to use the transport network in London especially if you pay-as-you-go rather than using a seasonal travelcard. You just “touch in and out” and fares are deducted.

It’s no longer necessary to have to work out how many journeys you’re going to do to decide if you need a daily travelcard or not – the amount is automatically capped at the price of a daily travelcard, if you get to that point. “Auto top-up” means money is transferred from your bank to your card when you get below a certain threshold so you never have to visit a ticket office (especially given they’ve closed them all) or a machine. It’s amazing to think they’ve been around since 2003.

More recently it’s possible to pay for travel with a contactless payment card a.k.a. Visa PayWave or Mastercard Paypass. However, that leads to problems if, like most people, you carry more than one card in your wallet or purse – the Oyster system may not be able to read your Oyster card or worse, if your contactless payment card is compatible, it takes payment from the first card it can communicate with. If you have an Oyster seasonal travelcard, you don’t want to pay again for your journey by paying with credit card.

TFL have recently started promoting this phenomenon under the name “card clash”.

There’s even a pretty animation:

CardClashAnimation

I first noticed card clash in 2010 when I was issued a new credit card with contactless payment. It went straight in my wallet with my Oyster and other payment cards and the first time I tried to touch in by presenting my wallet (as I’ve always done with Oyster) – beeeep beeeep beeeep (computer says no).

It didn’t take me long to work out that the Oyster reader couldn’t distinguish between two RFID-enabled cards. I soon had a solution in the form of Barclaycard OnePulse – a combined Oyster and Visa card. Perfect!

OnePulseCard

I could now touch in and out without having to get my card out of my wallet again. But not for long…

I don’t know about you but I like to carry more than one credit card so if one is declined or faulty, I can still buy stuff. I had a card from Egg Banking for that purpose. To cut a long story, slightly shorter, Egg were bought by Barclaycard who then replaced my card with a new contactless payment enabled card. Oh dear, card clash again.

I wrote to Barclaycard to ask if they could issue a card without contactless functionality. Here’s the response I got:

All core Barclaycards now come with contactless functionality; this is part of ensuring our cardholders are ready for the cashless payment revolution. A small number of our credit cards do not have contactless yet but it is our intention these will have the functionality in due course. That means we are unable to issue you a non-contactless card.

Please be assured that contactless transactions are safe and acceptance is growing. It uses the same functionality as Chip and PIN making it safe, giving you the same level of security and as it doesn't leave your hand there is even less chance of fraud.

I trust the information is of assistance.

Should you have any further queries, do not hesitate to contact me again.

I was very frustrated by this – I was already ready for the “cashless payment revolution” with my OnePulse card! Now, I was back in the situation where I would have to take the right card out of my wallet – not the end of the world but certainly not the promised convenience either. Barclaycard’s answer is to carry two wallets – helpful!

I’m certainly not the only one who has issues with card clash. I’m not the only one to notice TFL’s campaign to warn travellers of the impending doom as contactless payment cards become more common.

CardClash

So what’s the solution? If your card company isn’t flexible about what type of cards they issue and you can’t change to one that is (I hear HSBC UK may still be issuing cards without contactless payment on request), another option is to disable the contactless payment functionality. Some people have taken to destroying the chip in the card by putting a nail through it, cutting it with a knife or zapping it. However, that also destroys the chip and PIN functionality which seems over the top. You can probably use the magnetic strip to pay with your card but presenting a card with holes in it to a shop assistant is likely to raise concerns.

RFID cards have an aerial which allows communication with the reader and supplies power to the chip via electromagnetic induction (the zapping technique above overloads the chip with a burst of energy by induction). This means we can disable RFID without disabling chip and PIN by breaking the aerial circuit, meaning no communication or induction.

The aerial layout varies. I dissected an old credit card to get an idea where the aerial was in my card (soaking in acetone causes the layers of the card to separate). Here you can see that the aerial runs across the top and centre of the card.

20140321_095853

There are some helpful x-rays on the interwebs now showing some alternate layouts.

CardXray

The aerial can be broken with a hole punch or soldering iron but I’ve found that making a small cut across the aerial is sufficient. It’s not necessary to cut all the way through the card – all that’s needed is to cut through to the layer containing the aerial. Start with a shallow cut and make it deeper if required. From the images above, a cut to the top of the card is likely to disable most cards.

20140307_095731

The central cut on this card wasn’t effective. The cut at the top was.

If you have an NFC-enabled smartphone, it’s possible to test the card to check that contactless is disabled. I’m using NFC TagInfo on a Samsung Galaxy Nexus. Here’s the card above before:

Screenshot_2014-03-07-09-32-26

and after the aerial is cut (the chip no longer registers):

Screenshot_2014-03-07-09-33-54

Job done! The card continues to work with chip and PIN but doesn’t interfere with Oyster.

Barclaycard have thrown another spanner in the works for me – OnePulse is being withdrawn. The solution seems to be to switch to contactless payments instead of Oyster. Here’s hoping that TFL can finish implementing contactless payments (promised “later in 2014”) before my OnePulse is withdrawn.

Disclaimer: if you’re using sharp knives, have a responsible adult present. If you damage yourself or your cards, don’t blame me!

New host, new look – Azure and responsive design. Part 2.

Part 2. A new responsive design.

Mobile is the new black. Mobile phones are increasingly used to access the web and websites need to take that into account.

I’m using BlogEngine.NET which can deliver pages using a different template to browsers with small screens but that’s not ideal for larger phones and tablets. However, there is a better way – responsive web design. With a responsive design, CSS media queries and flexible images provide a flexible layout that is adjusted based on the screen (or browser) size.

The BlogEngine.NET team have recognised that responsive design is the way forward so the recently released version uses the Bootstrap framework for the default template and administration UI. They’ve done a great job.

BlogEngineResponsive

As part of moving my blog to Windows Azure as described in part 1, I felt a new look was in order but I didn’t want to use the new BlogEngine.NET template so set about designing my own.

Bootstrap isn’t the only framework that can help you build a responsive design. There’s Foundation, HTML5 Boilerplate, HTML KickStart and Frameless to name a few. I wanted a framework without too much in-built styling. Although Bootstrap can be customised, there are a lot of choices if you take that route.

BootstrapChoices

I chose Skeleton as the framework for my design. It’s a straightforward grid system with minimal styling. And I’ve retained the minimal styling, only introducing a few design elements. My design is inspired by the Microsoft design language (formerly Metro) – that basically comes down to flat, coloured boxes. Can you tell I’m not a designer?

An added complexity is the introduction of high pixel density displays such as Apple’s Retina. Images that look fine on a standard display, look poor on Retina and the like. Creating images with twice as many pixels is one solution but I chose the alternative route of Scalable Vector Graphics. modernuiicons.com provides an amazing range of icons (1,207 at time of writing), intended for Windows Phone but perfect for my design. They appear top-right and at the bottom of the blog.

Another website that came in very handy was iconifier.net – icon generator for Apple and favicon icons.

So, I’m now running on Windows Azure with a new responsive design. Perhaps I’ll get round to writing some blog articles that justify the effort?

New host, new look – Azure and responsive design. Part 1.

Part 1. Getting up and running on Windows Azure.

Microsoft’s cloud computing platform has a number of interesting features but the ability to create, deploy and scale web sites is particularly interesting to .NET developers. Many ISPs offer .NET hosting but few make it as easy as Azure.

The web server setup and maintenance is all taken care of and managed by Microsoft, which I already get with my current host Arvixe. However, apps can be deployed directly from GitHub and Bitbucket (and more) and it’s possible to scale to 40 CPU cores with 70 GB of memory with a few clicks. If you don’t need that level of performance, you might even be able to run on the free tier.

Here’s how I got up and running (I’ve already gone through the process of setting up an Azure account).

Create a new website from the gallery:

NewWebsiteFromGallery

In the gallery, select BlogEngine.NET. Choose a URL (something.azurewebsites.net) and a region (I choose North or West Europe as those are closest to my location).

Wait. 90 seconds later, a default BlogEngine.NET website is up and running.

I’ve created a source control repository in Mercurial. This contains:

  • extensions (in App_Code, Scripts and Styles)
  • settings, categories, posts, pages (in App_Data)
  • the new design for the blog (in Themes) – see part 2

Next, I need to configure my web site to deploy from Bitbucket where I’ve pushed that repo. “Set up deployment from source control” is an option in the dashboard:

DeploymentFromSourceControl

Select Bitbucket and you’ll be asked to log in and allow Azure to connect to your repos:

Bitbucket

Azure will then deploy that repo to the website from the latest revision.

All that’s left is to log into blog and make a couple of changes. Firstly, I delete the default blog article created by BlogEngine.NET. I’m displaying tweets on my blog using the “Recent Tweets” widget so I install that.

I’m using a custom domain rather than something.azurewebsites.net. That means I have to scale up from the free tier to shared:

SharedScale

Windows Azure must verify that I am authorized to use my custom domain. I do this by making a DNS change (my DNS records are hosted with DNSimple), adding a CNAME records pointing from awverify.www.robinminto.com to awverify.robinminto-com.azurewebsites.net and from awverify.robinminto.com to awverify.robinminto-com.azurewebsites.net. If I wasn’t using the domain at the time, I could have added the CNAME records without the awverify prefix. Verification step complete, I can then add the domain names to Azure.

ManageDomains

Finally, having checked everything is working properly I change the DNS entries (as above without the awverify prefix) to point to my new web site and it’s job done!

I took the opportunity to update the design that I use. I’ll deal with that in part 2.

Analysing Forefront TMG logs

I’m lamenting the demise of Forefront TMG but I still think it’s a great product and continue to use it. Of course, I’m currently planning what I’m going to replace it with but I expect it’ll have to be a combination of products to get the features that I need.

Anyway, I’m currently doing some log analysis and wanted to capture some useful information about the process. TMG is using SQL Server Express for logging and I want to do some analysis in SQL Server proper.

First, export the logs from TMG. I won’t repeat the steps from that post but in summary:

  1. Run the Import and Export Data application (All Programs / Microsoft SQL Server 2008 / Import and Export Data). Run this elevated/as administrator.
  2. Choose the database (not today’s databases or you’ll cause contention).
  3. Select Flat File Destination and include Column names in the first data row.
  4. Select Tab {t} as the Column delimiter.
  5. Export.

I took the files from this export and wanted to import them into SQL Server. Rather than import everything into varchar fields, I wanted to reproduce the schema that TMG uses. I couldn’t find the schema on the interwebs so I grabbed the databases from an offline TMG and scripted the tables.

Here’s the script to create the FirewallLog table:

CREATE TABLE [dbo].[FirewallLog](
    [servername] [nvarchar](128) NULL,
    [logTime] [datetime] NULL,
    [protocol] [varchar](32) NULL,
    [SourceIP] [uniqueidentifier] NULL,
    [SourcePort] [int] NULL,
    [DestinationIP] [uniqueidentifier] NULL,
    [DestinationPort] [int] NULL,
    [OriginalClientIP] [uniqueidentifier] NULL,
    [SourceNetwork] [nvarchar](128) NULL,
    [DestinationNetwork] [nvarchar](128) NULL,
    [Action] [smallint] NULL,
    [resultcode] [int] NULL,
    [rule] [nvarchar](128) NULL,
    [ApplicationProtocol] [nvarchar](128) NULL,
    [Bidirectional] [smallint] NULL,
    [bytessent] [bigint] NULL,
    [bytessentDelta] [bigint] NULL,
    [bytesrecvd] [bigint] NULL,
    [bytesrecvdDelta] [bigint] NULL,
    [connectiontime] [int] NULL,
    [connectiontimeDelta] [int] NULL,
    [DestinationName] [varchar](255) NULL,
    [ClientUserName] [varchar](514) NULL,
    [ClientAgent] [varchar](255) NULL,
    [sessionid] [int] NULL,
    [connectionid] [int] NULL,
    [Interface] [varchar](25) NULL,
    [IPHeader] [varchar](255) NULL,
    [Payload] [varchar](255) NULL,
    [GmtLogTime] [datetime] NULL,
    [ipsScanResult] [smallint] NULL,
    [ipsSignature] [nvarchar](128) NULL,
    [NATAddress] [uniqueidentifier] NULL,
    [FwcClientFqdn] [varchar](255) NULL,
    [FwcAppPath] [varchar](260) NULL,
    [FwcAppSHA1Hash] [varchar](41) NULL,
    [FwcAppTrusState] [smallint] NULL,
    [FwcAppInternalName] [varchar](64) NULL,
    [FwcAppProductName] [varchar](64) NULL,
    [FwcAppProductVersion] [varchar](20) NULL,
    [FwcAppFileVersion] [varchar](20) NULL,
    [FwcAppOrgFileName] [varchar](64) NULL,
    [InternalServiceInfo] [int] NULL,
    [ipsApplicationProtocol] [nvarchar](128) NULL,
    [FwcVersion] [varchar](32) NULL
) ON [PRIMARY]

and here's the script to create the WebProxyLog table:

CREATE TABLE [dbo].[WebProxyLog](
    [ClientIP] [uniqueidentifier] NULL,
    [ClientUserName] [nvarchar](514) NULL,
    [ClientAgent] [varchar](128) NULL,
    [ClientAuthenticate] [smallint] NULL,
    [logTime] [datetime] NULL,
    [service] [smallint] NULL,
    [servername] [nvarchar](32) NULL,
    [referredserver] [varchar](255) NULL,
    [DestHost] [varchar](255) NULL,
    [DestHostIP] [uniqueidentifier] NULL,
    [DestHostPort] [int] NULL,
    [processingtime] [int] NULL,
    [bytesrecvd] [bigint] NULL,
    [bytessent] [bigint] NULL,
    [protocol] [varchar](13) NULL,
    [transport] [varchar](8) NULL,
    [operation] [varchar](24) NULL,
    [uri] [varchar](2048) NULL,
    [mimetype] [varchar](32) NULL,
    [objectsource] [smallint] NULL,
    [resultcode] [int] NULL,
    [CacheInfo] [int] NULL,
    [rule] [nvarchar](128) NULL,
    [FilterInfo] [nvarchar](256) NULL,
    [SrcNetwork] [nvarchar](128) NULL,
    [DstNetwork] [nvarchar](128) NULL,
    [ErrorInfo] [int] NULL,
    [Action] [varchar](32) NULL,
    [GmtLogTime] [datetime] NULL,
    [AuthenticationServer] [varchar](255) NULL,
    [ipsScanResult] [smallint] NULL,
    [ipsSignature] [nvarchar](128) NULL,
    [ThreatName] [varchar](255) NULL,
    [MalwareInspectionAction] [smallint] NULL,
    [MalwareInspectionResult] [smallint] NULL,
    [UrlCategory] [int] NULL,
    [MalwareInspectionContentDeliveryMethod] [smallint] NULL,
    [UagArrayId] [varchar](20) NULL,
    [UagVersion] [int] NULL,
    [UagModuleId] [varchar](20) NULL,
    [UagId] [int] NULL,
    [UagSeverity] [varchar](20) NULL,
    [UagType] [varchar](20) NULL,
    [UagEventName] [varchar](60) NULL,
    [UagSessionId] [varchar](50) NULL,
    [UagTrunkName] [varchar](128) NULL,
    [UagServiceName] [varchar](20) NULL,
    [UagErrorCode] [int] NULL,
    [MalwareInspectionDuration] [int] NULL,
    [MalwareInspectionThreatLevel] [smallint] NULL,
    [InternalServiceInfo] [int] NULL,
    [ipsApplicationProtocol] [nvarchar](128) NULL,
    [NATAddress] [uniqueidentifier] NULL,
    [UrlCategorizationReason] [smallint] NULL,
    [SessionType] [smallint] NULL,
    [UrlDestHost] [varchar](255) NULL,
    [SrcPort] [int] NULL,
    [SoftBlockAction] [nvarchar](128) NULL
) ON [PRIMARY]

TMG stores IPv4 and IPv6 addresses in the same field using as a UNIQUEIDENTIFIER. This means we have to parse them if we want to display a dotted quad, or at least we find a function that will do it for us.

Here’s my version:

CREATE FUNCTION [dbo].[ufn_GetIPv4Address]
    (
      @uidIP UNIQUEIDENTIFIER
    )
RETURNS NVARCHAR(20)
AS 
    BEGIN
        DECLARE @binIP VARBINARY(4)  
        DECLARE @h1 VARBINARY(1) 
        DECLARE @h2 VARBINARY(1) 
        DECLARE @h3 VARBINARY(1)
        DECLARE @h4 VARBINARY(1)
        DECLARE @strIP NVARCHAR(20)

        SELECT  @binIP = CONVERT(VARBINARY(4), @uidIP) 
        SELECT  @h1 = SUBSTRING(@binIP, 1, 1) 
        SELECT  @h2 = SUBSTRING(@binIP, 2, 1) 
        SELECT  @h3 = SUBSTRING(@binIP, 3, 1) 
        SELECT  @h4 = SUBSTRING(@binIP, 4, 1) 
        SELECT  @strIP = CONVERT(NVARCHAR(3), CONVERT(INT, @h4)) + '.'
                + CONVERT(NVARCHAR(3), CONVERT(INT, @h3)) + '.'
                + CONVERT(NVARCHAR(3), CONVERT(INT, @h2)) + '.'
                + CONVERT(NVARCHAR(3), CONVERT(INT, @h1))

        RETURN @strIP 
    END  

Action values are stored as an integer so I’ve created a table to decode them:

CREATE TABLE [dbo].[Action](
    [Id] [int] NOT NULL,
    [Value] [varchar](50) NOT NULL,
    [String] [varchar](50) NOT NULL,
    [Description] [varchar](255) NOT NULL,
 CONSTRAINT [PK_Action] PRIMARY KEY CLUSTERED ([Id] ASC)
)

INSERT INTO Action
        ( Id, Value, String, Description )
VALUES
(0, 'NotLogged', '-', 'No action was logged.'),
(1, 'Bind', 'Bind', 'The Firewall service associated a local address with a socket.'),
(2, 'Listen', 'Listen', 'The Firewall service placed a socket in a state in which it listens for an incoming connection.'),
(3, 'GHBN', 'Gethostbyname', 'Get host by name request. The Firewall service retrieved host information corresponding to a host name.'),
(4, 'GHBA', 'gethostbyaddr', 'Get host by address request. The Firewall service retrieved host information corresponding to a network address.'),
(5, 'Redirect_Bind', 'Redirect Bind', 'The Firewall service enabled a connection using a local address associated with a socket.'),
(6, 'Establish', 'Initiated Connection', 'The Firewall service established a session.'),
(7, 'Terminate', 'Closed Connection', 'The Firewall service terminated a session.'),
(8, 'Denied', 'Denied Connection', 'The action requested was denied.'),
(9, 'Allowed', 'Allowed Connection', 'The action requested was allowed.'),
(10, 'Failed', 'Failed Connection Attempt', 'The action requested failed.'),
(11, 'Intermediate', 'Connection Status', 'The action was intermediate.'),
(12, 'Successful_Connection', '- Initiated VPN Connection', 'The Firewall service was successful in establishing a connection to a socket.'),
(13, 'Unsuccessful_Connection', 'Failed VPN Connection Attempt', 'The Firewall service was unsuccessful in establishing a connection to a socket.'),
(14, 'Disconnection', 'Closed VPN Connection', 'The Firewall service closed a connection on a socket.'),
(15, 'User_Cleared_Quarantine', 'User Cleared Quarantine', 'The Firewall service cleared a quarantined virtual private network (VPN) client.'),
(16, 'Quarantine_Timeout', 'Quarantine Timeout', 'The Firewall service disqualified a quarantined VPN client after the time-out period elapsed.')

The next step is to import the exported text files into the relevant table in SQL Server. Note that the SQL Server Import and Export Wizard has a default length of 50 characters for all character fields – that won’t be sufficient for much of the data. I used the schema as a reference to choose the correct lengths.

I can now look at the log data in ways that the log filtering can’t and without stressing the TMG servers.

-- Outbound traffic ordered by connections descending
SELECT ApplicationProtocol, A.String AS Action, COUNT(*) 
FROM FirewallLog FL
INNER JOIN Action A ON A.Id = FL.Action
WHERE DestinationNetwork = 'External' AND FL.Action NOT IN (7)
GROUP BY ApplicationProtocol, A.String
ORDER BY COUNT(*) DESC


-- Successful outbound traffic ordered by connections descending
SELECT ApplicationProtocol, A.String AS Action, COUNT(*) 
FROM FirewallLog FL
INNER JOIN Action A ON A.Id = FL.Action
WHERE DestinationNetwork = 'External' AND A.String IN ('Initiated Connection')
GROUP BY ApplicationProtocol, A.String
ORDER BY COUNT(*) DESC


-- Successful outbound traffic showing source and destination IP addresses, ordered by connections descending
SELECT  ApplicationProtocol ,
        dbo.ufn_GetIPv4Address(SourceIP) AS SourceIP ,
        dbo.ufn_GetIPv4Address(DestinationIP) AS DestinationIP ,
        COUNT(*) AS Count
FROM    FirewallLog
WHERE   ApplicationProtocol IN ( 'HTTP', 'HTTPS', 'FTP' )
        AND DestinationNetwork = 'External'
        AND Action NOT IN ( 7, 8, 10, 11 )
GROUP BY ApplicationProtocol ,
        dbo.ufn_GetIPv4Address(SourceIP) ,
        dbo.ufn_GetIPv4Address(DestinationIP)
ORDER BY COUNT(*) DESC

Removing IIS headers using Powershell, ASafaWeb and server configuration

I’ve been using the ASafaWeb security analyser to scan various websites that I work on. It picks up basic configuration problems in .NET websites and, very handily, will scan periodically to make sure that misconfiguration doesn’t creep back in.

ASafaWeb was created by Troy Hunt, a software architect and Microsoft MVP for Developer Security based in Australia. Troy writes some great articles on improving application security, with a focus on .NET, and he links to those articles in ASafaWeb to illustrate why and how improvements should and can be made.

Most of the issues can be addressed in the application, generally in web.config, but the issue I’m interested in here can be solved by configuring at the server level. Here’s the output from ASafaWeb:

13082012_113203

See that “X-Powered-By: ASP.NET” header? That one’s inherited from the IIS root configuration. We could remove it in every web.config but better to remove it once.

Troy links to his blog post on removing all of those headers – Shhh… don’t let your response headers talk too loudly. As he mentions, there are many different ways of removing those headers – he goes into IIS Manager UI and removes it from there.

I’m trying to make sure that I script server configuration changes. This is a great way of documenting changes and allows new servers to be configured simply. I want servers that can rise like a phoenix from the metaphorical ashes of a new virtual machine rather than having fragile snowflake that is impossible to reproduce (thanks to Martin Fowler and his ThoughtWorks colleagues for the imagery).

So, the script to remove the “X-Powered-By” header turns out to be very straightforward once you figure out the correct incantations. This assumes you have Powershell and the Web Server (IIS) Administration Cmdlets installed.

Import-Module WebAdministration
Clear-WebConfiguration "/system.webServer/httpProtocol/customHeaders/add[@name='X-Powered-By']"

That’s a lot of words for a very little configuration change but I wanted to talk about ASafaWeb and those ThoughtWorks concepts of server configuration. I also need to mention my mate Dylan Beattie who helped me out without knowing it. How? StackOverflow of course.

Accurate time synchronisation using ntpd for Windows

Computers can be finickity about time but they’re often not very good at keeping it. My desktop computer, for instance, seems to lose time at a ridiculous rate. Computer hardware has included a variety of devices for measuring the passage of time from Programmable Interval Timers to High Precision Event Timers. Operating systems differ in the way that they use that hardware, perhaps counting ticks with the CPU or by delegating completely to a hardware device. This VMware PDF has some interesting detail.

However it’s done, it doesn’t always work. The hardware is affected by temperature and other environmental factors so we generally synchronise with a more reliable external time source, using NTP. Windows has NTP support built in, the W32Time service, but it only uses a single time server and isn’t intended for accurate time. From this post on high accuracy on the Microsoft Directory Services Team blog,

“W32time was created to assist computers to be equal to or less than five minutes (which is configurable) of each other for authentication purposes based off the Kerberos requirements”

They quote this knowledgebase article:

“We do not guarantee and we do not support the accuracy of the W32Time service between nodes on a network. The W32Time service is not a full-featured NTP solution that meets time-sensitive application needs. The W32Time service is primarily designed to do the following:

  • Make the Kerberos version 5 authentication protocol work.
  • Provide loose sync time for client computers.”

So, plus or minus five minutes is good enough for Kerberos authentication within a Windows domain but it’s not good enough for me.

NTPD

I’ve been experimenting with the NTPD implementation of the NTP protocol for more accurate time. NTPD synchronises with multiple NTP servers but only synchronises when a majority agree. I’m using the Windows binaries from Meinberg. Meinberg also have a monitoring and configuration application, called NTP Time Server monitor, which is very useful.

The implementation is straightforward, so I won’t go into it in detail. Install the service. Install Monitor. Add some local time servers from pool.ntp.org to your config and away you go.

I do, however, want to describe an issue that we had with synchronisation on our servers. Wanting more accurate time for our application, we installed NTPD and immediately started getting clock drift. Without NTPD running, the clocks were quite stable but with it running, they would drift out by a number of seconds within a few minutes.

A quick Google led me to this FixUnix post and I read this quote from David:

“Do you have the Multi-Media timer option set? It was critical to getting
good stability with my systems”

The link was broken but I eventually found this page which describes the introduction of the –M flag to NTPD for Windows. This runs the MultiMedia timer continuously, instead of allowing it to be enabled occasionally by applications like Flash.

Cutting a long story slightly shorter, it turns out that the –M flag was causing the clock drift on our Windows servers. Removing that option from the command line means that our servers are now happily synchronising with an offset of less than 300ms from the time source.

Tardis

As I mentioned, my desktop computer is very inaccurate. NTPD can’t keep it in check so I’m trying a different approach – Tardis 2000. So far, so good. I’m not sure about the animated GIFs on the website though.

Microsoft Threat Management Gateway web farm publishing issue – “The remote server has been paused or is in the process of being started”

We’ve recently uncovered an issue with the way that I had configured web farm publishing in Microsoft Threat Management Gateway (TMG). When I say “we”, I include Microsoft Support who really got to the bottom of the problem. Of course, they’re in a privileged position as they have access to the source code of the product.

Perhaps I would have resolved it eventually. I’m thankful for MS support though. I didn’t find anything on the web to help me with this problem so, on the off chance it can help someone else, I thought I’d write it up.

The Symptoms

We’ve been switching our web publishing from Windows NLB to TMG web farms for balancing load to our IIS servers and we began seeing an intermittent issue. One minute we were successfully serving pages but the next minute, clients would receive an HTTP 500 error “The remote server has been paused or is in the process of being started” and a Failed Connection Attempt with HTTP Status Code 70 would appear in the TMG logs.

08102011_211144

The issue would last for 30 to 60 seconds and then publishing would resume successfully. This would normally indicate that TMG has detected, using connectivity verifiers for the farm, that no servers are available to respond to requests. However, the servers appeared to be fine from the perspective of our monitoring system (behind the firewall) and for clients connecting in a different way (either over a VPN or via a TMG single-server publishing rule).

The (Wrong) Setup

Let’s say we have a pair of web servers, Web1 and Web2, protected from the Internet by TMG.

08102011_223957

Each web server has a number of web sites in IIS, each bound to port 80 and a different host header. All of the host headers for a single web server map to the same internal IP address like this:

Host name IP address
prod.admin.web1 172.16.0.1
prod.cms.web1 172.16.0.1
prod.static.web1 172.16.0.1
prod.admin.web2 172.16.0.2
prod.cms.web1 172.16.0.2
prod.static.web1 172.16.0.2

In reality, you should fully qualify the host name (e.g. prod.admin.web1.examplecorp.local) but I haven’t for this example.

I’ll assume that you know how to publish a web farm using TMG. We have a server farm configured for each web site with each web server configured like this (N.B. this is wrong as we’ll see later):

08102011_220001

The benefit of this approach is that because we’ve specified the host header (prod.admin.web1) rather than just the server name (web1), we don’t have to specify the host header in the connectivity verifier:

08102011_215828

This setup appears to work but under load, and as additional web sites and farm objects are added, our symptoms start to appear.

The Problem

So what was happening? TMG maintains open connections to the web servers which are part of the reverse-proxied requests from clients on the Internet. Despite the fact that all of host headers in the farm objects resolve to the same IP address, TMG compares them based on the host name and therefore they appear to be different. This means that TMG is opening and closing connections more often than it should.

The Solution

The solution is to specify the servers in the server farm object using the server host name and not the host header name. You have to do this for all farm objects that are using the same servers.

08102011_212545

You then have to specify the host header in the connectivity verifier:

08102011_212849

You could also use the IP address of the server. This is the configuration that Jason Jones recommends but I prefer the clarity of host name over IP address. I’m trusting that DNS will work as it should and won’t add much overhead. If you need support with TMG, Jason is excellent by the way.

Conclusion

Specifying the servers by host header name seemed logical to me. It was explicit and didn’t require that element of configuration to be hidden away in the connectivity verifier.

I switched from host header to IP address as part of testing but it didn’t fix our problem. It didn’t fix the problem because I only used IP addresses for a single farm object and not all of them.

Although TMG could identify open server connections based on IP address, it doesn’t. It uses host name. This has to be taken into account when configuring farm objects. In summary, if you’re using multiple server farm objects for the same servers, make sure you specify the server name consistently. Use IP address or an identical host name.

How many hops? Internet from far, far away.

IMG_20110917_123232

I was lucky enough to spend a week in the Seychelles this month. If your geography is anything like mine you won’t know that the Seychelles is “an island country spanning an archipelago of 115 islands in the Indian Ocean, some 1,500 kilometres (932 mi) east of mainland Africa, northeast of the island of Madagascar” – thank you Wikipedia.

In our connected world, the Internet even reaches into the Indian Ocean so I wasn’t deprived of email, BBC News, Facebook etc. but I did begin to wonder how such a small and remote island nation is wired up. It turns out that I was online via Cable and Wireless and a local ISP that they own called Atlas.

Being a geek I thought I’d run some tests, starting with SpeedTest. The result was download and upload speeds around 0.6Mbps and a latency of around 700ms. In the UK, the average broadband speed at the end of 2010 was 6.2Mbps and I would expect latency to be less than 50ms. Of course, I went to the Seychelles for the sunshine and not the Internet but I was interested in how things were working.

So, I ran a “trace route” to see how my packets would traverse the globe back to the BBC in Blighty. I thought the Beeb was an appropriate destination.

01102011_004019

Then I added some geographic information using MaxMind's GeoIP service and ip2location.com.

The result was this:

HopIP AddressHost NameRegion NameCountry NameISPLatitudeLongitude
1 10.10.10.1            
2 41.194.0.82   Pretoria South Africa Intelsat GlobalConnex Solutions -25.7069 28.2294
3 41.223.219.21     Seychelles Atlas Seychelles -4.5833 55.6667
4 41.223.219.13     Seychelles Atlas Seychelles -4.5833 55.6667
5 41.223.219.5     Seychelles Atlas Seychelles -4.5833 55.6667
6 203.99.139.250     Malaysia Measat Satellite Systems Sdn Bhd, Cyberjaya, Malay 2.5 112.5
7 121.123.132.1   Selangor Malaysia Maxis Communications Bhd 3.35 101.25
8 4.71.134.25 so-4-0-0.edge2.losangeles1.level3.net   United States Level 3 Communications 38.9048 -77.0354
9 4.69.144.62 vlan60.csw1.losangeles1.level3.net   United States Level 3 Communications 38.9048 -77.0354
10 4.69.137.37 ae-73-73.ebr3.losangeles1.level3.net   United States Level 3 Communications 38.9048 -77.0354
11 4.69.132.9 ae-3-3.ebr1.sanjose1.level3.net   United States Level 3 Communications 38.9048 -77.0354
12 4.69.135.186 ae-2-2.ebr2.newyork1.level3.net   United States Level 3 Communications 38.9048 -77.0354
13 4.69.148.34 ae-62-62.csw1.newyork1.level3.net   United States Level 3 Communications 38.9048 -77.0354
14 4.69.134.65 ae-61-61.ebr1.newyork1.level3.net   United States Level 3 Communications 38.9048 -77.0354
15 4.69.137.73 ae-43-43.ebr2.london1.level3.net   United States Level 3 Communications 38.9048 -77.0354
16 4.69.153.138 ae-58-223.csw2.london1.level3.net   United States Level 3 Communications 38.9048 -77.0354
17 4.69.139.100 ae-24-52.car3.london1.level3.net   United States Level 3 Communications 38.9048 -77.0354
18 195.50.90.190   London United Kingdom Level 3 Communications 51.5002 -0.1262
19 212.58.238.169   Tadworth, Surrey United Kingdom BBC 51.2833 -0.2333
20 212.58.239.58   London United Kingdom BBC 51.5002 -0.1262
21 212.58.251.44   Tadworth, Surrey United Kingdom BBC 51.2833 -0.2333
22 212.58.244.69 www.bbc.co.uk Tadworth, Surrey United Kingdom BBC 51.2833 -0.2333

What does that tell me? Well, I ignored the first hop on the local network and the second looks wrong as we jump to South Africa and back again. The first stop on our journey is Malaysia, 3958 miles away.

MSP130619habd3ff28e320700000g53di634b58d5ec

We then travel to the west coast of America, Los Angeles. Another 8288 miles.

MSP209619habc360b3abg0g0000322e7i1185717864

We wander around California. The latitude/longitude information isn’t that accurate so I’m basing this on the host name but we hop around LA and then to San Jose followed by New York. Only 2462 miles.

MSP15619habg6g8548fc9b00002a9i7gaf324f582g

We rattle around New York and then around London, 3470 miles across the pond.

MSP16819habgbc842f7750000045e0cea0hga1a972

Our final destination is Tadworth in Surrey, just outside of London.

That’s just over eighteen thousand miles (and back) in less than a second – not bad, I say.

p.s. don’t worry, I spent most of the time by the pool and not in front of a computer.

I’ve moved! (hosts and blog software)

I’ve just finished switching my blog from WordPress on TsoHost to BlogEngine.Net on Arvixe. I got an account with Arvixe following a recommendation from a colleague, who also switched from TsoHost.

The reasons for switching? Well, Arvixe allows me to use .NET 4 and SQL Server databases (which I wanted for an Umbraco CMS site that I’m now running) plus I got a promotional rate from Arvixe. Splendid.

Getting BlogEngine.NET up and running was easy but I did go back and forth trying to get the correct application settings for the server. I had a few HTTP 500 errors with nothing else to go on but experience. The Arvixe control panel is a little bit slow, which was frustrating.

The migration was fairly straightforward – I only had a couple of posts to transfer so I did them manually and then tweaked a little. I’d like to add some pictures to the blog theme – it’s a bit too clean at the moment.

Of course, hosts and blog engines and themes are a means to an end – I’m going to try and get some posts up too!