Update:22 April 2009  19.53

It appears a problem with a server power supply unit set off the datacentre fire system. Also the rack lost power. Services are gradually being restored.

Update: Thu Apr 23 13:05:25 BST 2009

At approximately 15:18 BST we were made aware of our VESDA (Very Early Smoke Detection Apparatus) system detecting a pre-fire condition in our data-centre which armed our fire suppression Inergen system - One of the facility engineers visited the data-centre to investigate but could not see any obvious culprit.

The system moved from armed to active to protect the data-centre environment by lowering the oxygen content of the room to a point sufficient to sustain human life, but insufficient to support combustion.

Soon after the alarm, all staff from the entire facility were evacuated until the fire department gave the go ahead to re-enter the facility - extra staff members were drafted in to help.

Rest assured we have been working around the clock to help customers who may have been affected by the incident.

Location: STA.LON2

The Coreix engineering team will be conducting network maintenance beginning at 00:00AM GMT on Sunday 30th November 2008, lasting until 06:00AM GMT of the same morning.

Summary of downtime and customers affected:

Part 1 – None anticipated

Part 2 – Customers who take communal load balancing solutions will be affected by 30 seconds to 1 minute of downtime.

Part 3 – All customers, there will be an increase in latency and some packet loss. There will be a customer edge outage for approximately 2 to 5 minutes per rack whilst we migrate customers from one chassis to another.

Part 4 – None anticipated – Minor route changes are expected.

———————-

Part 1 – The operating system upgrade of core1.sta.lon2.coreix.net

Scope of work: The upgrade of the JunOS on core1.sta.lon2.coreix. The hardware will be taken offline cleanly and should have no impact on networking operations.

Outage: We do not anticipate any downtime during this maintenance section due to the Coreix network resiliency.

Rationale: Coreix performs periodic Operating System upgrades on its equipment to improve security and features on its equipment.

Part 2 – Upgrade the operating system on lb1.sta.lon2.coreix.net

Scope of work: The upgrade of the WebOS on lb1.sta.lon2.coreix. The hardware will be taken offline cleanly, upgraded and rebooted.

Outage: All customers who take communal load balancing services may see 3 or 4 packets lost during this maintenance (approximately 30 seconds to 1 minute of downtime), while the load balancing equipment fails over. Customers with dedicated load balancer solutions will not be affected.

Rationale: Coreix performs periodic Operating System upgrades on its equipment to improve security and features on its equipment.

Part 3 – The upgrade of dist7b.sta.lon2.coreix.net

Scope of work: As part of our expansion at the Stratford site (sta.lon2), we will be migrating customers from one distribution 7a and replacing the chassis of distribution 7b. The new chassis will still maintain the redundancy you expect from Coreix (dual supervisor, dual power supplies, multiple copper and optical cards for dual edge switching ports and core uplinks).

Outage: As we swap over to the new chassis customers will be migrated rack by rack. Since all Coreix edge switches have 2 uplinks to the distribution the downtime will be minimal since we will run the two in parallel. I have estimated 2 to 5 minutes per rack, however in certain instances the swap over will be significantly lower.

Rationale: We are increasing the port density of our distribution layer in response to an unanticipated rapid expansion of colocation, enterprise services and dedicated servers.

Part 4 – The addition of core3.sta.lon2.coreix.net

Scope of work: The Coreix network expands its peering arrangements on a daily basis and as we grow our routing requirements increase. As part of our expansion we are installing a third core router at the Stratford site. The main task of the router will be to mix our primary peering traffic (private peers plus public exchanges), and will act as another resilient node on our network.

Outage: This is a node addition therefore we do not expect any downtime during this period. Customers may see minor route changes for direct peers during this period as the traffic migrates from one core to another (however no traffic should be lost).

Rationale: This upgrade is in response to an increase in traffic and peers on our core routers.

Issue start: 11:50:23
Final Issue resolution: 12:19:13
Outage Duration: ~ 30 Minutes
Cause of issue: Main breaker trip on PDU 4B/2-A feed affecting local power-strips Weycrest apologise to customers for the delay in restoring the services. This was partly because it was neccessary to `fsck` the large disc array on one server which took some time after the power was restored. Though this was however the first power outage in this datacentre in four years but are investigating the possibility of connecting our equipment via “higher availability” power drops in future.

This affects customers on mercury and pluto.

We will update as soon as we have more information.

Update: Just back from the datacentre: Service restored about 30 minutes ago. Problem caused by failed gigalink connection to the switch. - Ellis

Update: SMTP service was resumed 20 February.

Update: Sat Jan 19 14:23:16 GMT 2008

The network core was restored, all services functioning normally.

This is a slightly tricky one to sort, as according to our network monitoring the httpd server appears “up” all the time. If its a “proper” outage we get SMS’s of course, if the server doesn’t automatically restart.

Last time we had this issue its was caused by an over aggressive content grabber, that was slowly “DoS’ing” the server. Please bear with us while we track down the issue.