C2 NOC

Issues at Manchester TCW

2014-05-13T10:16:00.002+01:00

10:15 Our monitoring systems are currently alerting multiple failures at Manchester TCW. Engineers are investigating and we will have more information shortly. The nature of the failures do not indicate a single item failure. We are waiting feedback from TCW remote hands before deciding our next course of action however engineers are prepping to go onsite.

10:27 Feedback from remote hands is our racks appear to be drenched in water. Our Technical Manager is on his way to asses the damage and formulate workarounds where possible.

12:00 On-Site

12:50 Damaged equipment has been replaced. A misconfiguration on a vendor side network has also been detected which will need rectifying - if the misconfiguration had not been present the impact on our clients would not have been so great. Except for when we correct the configuration issue (which will not be today) we do not expect any further downtime. Our sincere apologies for the downtime caused.

Access router in London HEX6&7

2014-03-19T07:40:00.002+00:00

07:00 - We've lost contact with one of our routers in Harbour Exchange 6&7, while all other equipment is operational any directly connected services to this router will currently be down. A power cycle through remote hands has been requested.

08:30 - I've put in some additional VLANs to enable some additional routing. Most clients should now be seeing connectivity.

09:05 - Telecity engineers have rebooted the affected router, the workarounds will stay in place for the remainder of the day or will be engineered further into an additional fallback option later on. In the meantime routing will be sub-optimal.

Monitoring systems

2014-02-10T10:48:00.000+00:00

A firewall that sits in front of our primary monitoring system is currently experiencing some troubles. This has caused that system to raise false down alerts. Due to the number and frequency of the down alerts we have decided to disable the primary monitoring system until a solution can be put in place, the alerts have been causing considerable confusion. Key infrastructure is also monitored via a second system, this system however does not raise customer facing alerts and is generally used to gain confirmation of the primary system during outages so our core continues to be monitored.

Engineers are working to restore the primary system to operation as soon as possible.

Best Regards
Atlas Technical

DSL Network Issues

2014-01-17T17:21:00.000+00:00

17:00 - We appear to be having issues in our DSL network, the problem itself is within our providers wholesale network. Connections will appear to be running extremely slowly, you may find your router will drop the connection and try to re-establish the link, it is likely the reconnect will fail until the issue is resolved. Unfortunately it is affecting connections coming into all our client facing DSL terminating routers in both Manchester and London so it is not possible for us to work around the issue at this time. We're waiting for feedback from our provider as to a fix time/update. Our fixed line and metro wireless services are not affected.

At risk notification Manchester 30th July 2013

2013-07-16T10:03:00.000+01:00

One of our upstream transit providers have informed us they are upgrading their infrastructure in Manchester.

The work is taking place on 30th July 2013 between the times of 11pm and 6am the following day. We are to expect one hour of downtime during this window.

In itself this should not be service affecting to any of our clients as traffic will flow through other pathways and our other transit partners, though as the service switches to the alternate providers there may be some routing instabilities.

Kind Regards
Atlas Technical

Fibre issues in Nantwich/Aston areas.

2013-03-12T22:50:00.000+00:00

10:30pm 12/3/2013

Our monitoring system has picked up failures on both the North and South sets of fibre bundles feeding our Bunker Datacentre.

Traffic is however flowing via another pathway so the datacentre remains connected to the rest of the network, due to the increased load however some contention is apparent.

Customers in Nantwich taking direct Internet Access connections from us over EAD, these circuits connect via the datacentre and are unfortunately currently down, our apologies for the inconvenience.

--
10:45pm

Fibre provider is carrying out repairs to multiple sections of fibre, repairs are being carried out by multiple teams both to the North and South of the datacentre.

Completion of work is estimated by 6am, however our circuits are being given priority for reconnection.

Router issue in Manchester

2013-02-12T02:02:00.001+00:00

12/Feb/2013 - 1:26am - Monitoring has detected a failure of a router in Manchester - Engineers are currently investigating the problem. Services mostly affected are DSL and Leased Lines terminating in Manchester as well as some cross core activity.

12/Feb/2013 - 1:45am - Waiting response from datacentre.

12/Feb/2013 - 2:20am - Confirmation from datacentre of router reboot.

12/Feb/2013 - 2:30am - Router back online and exchanging traffic.

Network outage at London HEX6&7

2012-11-10T16:42:00.002+00:00

10/11/2012 16:30 - We've just had notification of a router going offline at London HEX6&7, we're currently investigating as to what may be happening before taking any action to redirect connections etc. Currently any connections going via HEX6&7 will be offline.

--

16:45 Actions underway to redirect all DSL connections to alternate node as well as activate backup DSL connections for leased line connections terminating at HEX.

--

17:00 Reload of the router has been completed - router is back online and exchanging data with it's peers. DSL connections for the moment will be left pointing at Manchester routers.

Notification of engineering works affecting Bunker datacentre 13/11/2012

2012-10-02T14:17:00.002+01:00

We've been notified by our electricity provider that on the 13th November 2012 they will be carrying out engineering works that will affect our mains feeds into the Bunker. Mains power will be switched off from 9am until 4pm.

We anticipate no impact on any services as throughout this period the Bunker will be operating on it's own internal generator power.

Regards
Atlas Technical

Bunker Partial Power Failure

2012-08-19T20:39:00.002+01:00

19/8/12 20:20 - We're currently experiencing a partial power failure on the mains at the Bunker datacentre. It appears we have lost one of the phases. Unfortunately under these circumstances the generators did not kick in, a manual override has been applied. Equipment has been running on UPS but some of them have exhausted their battery supplies - as soon as these UPS's have enough charge they will start passing current again and the equipment will power up.

20:40 - SP Confirm single phase failure.

20/8/12 08:30 - All work complete and Bunker verified as back on mains power.

DSL Network Issues 25/7/2012

2012-07-25T15:21:00.003+01:00

15:22

Our wholesale DSL interconnects have just gone down - we're investigating the issue now.
15:26
A router in the wholesale network experienced an unexpected restart - connections are coming back online now. If your DSL router does not connect try a 1 minute power-cycle, if that does not work try a full 15 minute power cycle. Any issues after that please give us a call.

Issues with C2 DSL Network

2012-07-06T16:56:00.001+01:00

July 6th 2012 16:56
From approx 15:55 we have lost connectivity to our equipment in Telecity HEX6&7. Remote hands cannot identify any issues so we are therefore sending an engineer to site with spare equipment to be able to investigate and replace as required. We estimate being onsite 9pm weather permitting. An update will follow shortly after that. This is affecting DSL customers and fixed line services provided through that Datacentre. Services provided out of our other datacentres are not affected and our own facility at Hack Green Bunker is also unaffected.

Regards C2 Technical

10:00pm Engineer onsite.

10:54pm Equipment replaced and configuration loaded - DSL connections and fixed line services restored. DSL connections may require a reboot in order to reconnect.

Apologies for the inconvenience.

Regards C2 Technical

Emergency Maintenance Notification

2012-04-05T09:11:00.001+01:00

We have received notification from our primary transit provider that on 12th April 2012 between 1am and 4am GMT there is a possibility of an upto 20 minute outage.

Regards
Atlas Technical

Future works on Manchester ring 19/08/2011

2011-08-09T10:50:00.003+01:00

We've received notification from one of our network providers that as part of expanding their network, a number of fibre routes between TeleData and IFL2 require diverting.

This work will take place during the period 19/08/2011 23:00 to 20/08/2011 11:00 (BST).

This will result in a loss of protection on the Manchester ring until the work is complete.

Regards
C2

Issues on Manchester Ring

2011-07-20T08:54:00.005+01:00

The causes of the issues on the Manchester ring last night are still not yet clear, we know part of the ring went down, what we cant do however is replicate the issue. Each time we manually shut down part of the ring traffic simply flows in the opposite direction. We are aware of issues with our network provider, they've been working on the TCW to IFL2 links, we will be chasing today for an update as to if they were working on anything at the time. As we cannot replicate the issue it would appear that some outside influence may be affecting the network, however, we will today be taking some emergency investigative works onsite. From 1pm onwards you may see some network blips, we will try and keep these to a minimum.

Apologies for the short notice, however if we can we need to find what the cause was.

Once we have completed our tests we will post a further update.

---

Following on from the tests a switch at TCW, which is part of a pair, has been identified as the most probable cause for the issues and instabilities. As the issue is somewhat intermittent I'm reluctant to pull the switch while it's under load from directly connected clients - the quietest time across all the ports is around 7am, so, we will look to swap the unit out then on Thursday 21st.. Directly connected clients and those using services connected to this switch will drop for a few minutes while the switch is replaced.

---

The work is now complete - the suspect switch will be taken back for further tests in the lab, further updates will be posted here.

DSL issues in London

2010-12-21T17:43:00.000+00:00

Some C2 customers in the London area may currently be experiencing DSL issues due to an incident at the West End BT exchange. BT engineers are trying to restore services after a flood prompted a fire in the exchange. BT expect to have services restored shortly.

Scheduled maintenance 6th December 10:00 and 16:00

2010-12-02T17:18:00.000+00:00

Hi,

We have been advised by our network provider that as part of their capacity planning they are having to move the current line between IFL and Telecity Williams to another line. Though the window for the required work is between 10:00 and 16:00, they expect that the line will only be down for 1 hour. However the remaining two legs of the ring will remain untouched, so customers should not notice any disruption to service, but the Manchester ring will obviously be more at risk of interruption while testing takes place.

We have insisted that this work be completed while we are in contact with their engineers, so that if we notice anything unexpected on the network, we can get any work reversed immediately.

Kind Regards

Stuart McKindley

Removal of secondary mail server mail-relay20.c2internet.net

2010-11-22T15:15:00.001+00:00

Due to hardware failure the server mail-relay20.c2internet.net is being removed from service.

This server's only role was to operate as a secondary/backup mx to customers requesting this functionality where those customers operated their own primary mail servers.

While initially this type of setup was the norm as the war against spam continues these type of backup servers have been targetted as easy routes in. This brings rise to a few problems;

It's not uncommon for these backup servers to be whitelisted/trusted by the primary server, thus totally defeating any anti-spam techniques they are utilising. The backup servers will accept all mail for the domains where it is told to be the secondary, if when forwarding that email to the primary server the primary server rejects a mailbox as unknown the backup server will want to send a non-delivery report. If the originating email was from a forged email address then these NDR's clog the system further which just puts extra load on the server for no real good reason. Worst case is the NDR's are sent to a valid email address but one which had nothing to do with the original email, at which point the server is generating backscatter which is every bit as bad as spam.

If the primary mail server was to fail most sending servers will now quite happily queue email, notify the sender of any sending delays and generally look after sending the email again after a few minutes when the server comes back up.

With all this in mind will we shortly be removing all entries from DNS for mail-relay20.c2internet.net. The unusual thing here is customers who have been using the service may well see a drop in the amount of incoming spam to that of which they had been used to.

This does not affect customers that have their own secondary mail servers

DSL Connections via BT

2010-11-22T14:15:00.004+00:00

-- 15:15

Fault is now cleared - we will continue to monitor.

Apologies for any inconvenience.

-- 14:26

We're seeing a number of lines coming back up, though as yet have had no notification of this fault being cleared. We will continue to monitor.

-- 13:49

There is an issue affecting a number of tail circuits that are provided over the BT wholesale network. This is affecting a number of ISPs and is not related to anything within our network or anything under our direct control. The issue is being investigated and more information will be posted as soon as is available.

Service Outage report for 10th November 2010

2010-11-10T13:42:00.004+00:00

17:32-

We've just had confirmation that the outage was from two separate faults happening in two separate geographic locations, one fault was on the providers Leeds to Sheffield connection, the other on their Warrington to Birmingham connection.

13:42-

At 10:50am this morning we lost both our west and east-bound connections from Manchester to London, this had the outcome of partitioning our core network into two. This partitioning would have caused routing issues and due to the location of name and radius servers within the network name lookup and xDSL authentication would also have failed.

Our transit feed out of Manchester was also experiencing problems which as this issue cleared at the same time our connections came back up was no doubt down to the same core root problem.

With the issue affecting multiple providers it was clear the problem was itself not within any equipment within our direct control or the outcome of any of our actions within the network.

Our main telephone system is also based out of Manchester however when the server went offline it failed over onto the backup analogue PSTN system, the number of incoming calls obviously proving a challenge.

We're currently in discussion with our network provider for the Manchester to London connections as these routes should be separate and diverse, initially they also went via separate providers however due to consolidation within the market one provider has ended up owning both networks. If it transpires that our provider has without our knowledge or authorisation joined these pathways then of course action will be taken.

At 12:40pm both connections came back up, with the exception of transit our of Manchester once the network had re-converged connections and traffic flows returned to normal. Approximately ten minutes after our connections re-established transit via our transit provider also re-established.

Our apologies for this outage and the inconvenience.

Upstream transit provider

2010-07-26T09:10:00.003+01:00

-- 9:00AM

We're currently experiencing packet loss on one of our upstream transit providers. The connection needs to remain active for a short while to allow us to run diagnostics before passing the call to our provider.

-- 9:25am

Sessions to this transit provider have now been shutdown and traffic is flowing via alternative pathways, another upstrean provider however is also now showing packet loss so this provider has also been disabled.

General Service Issues

2010-06-03T04:47:00.005+01:00

4:00AM - Customers may be experiencing some general connectivity issues ranging from some Internet hosts being unavailable through to DSL lines not reconnecting after a reboot.

There appears to be a service outage in London which is affecting some of our suppliers, this outage is affecting multiple providers.

If you are experiencing any issues please do not reboot your router at this stage.

More information will be posted as it becomes available

5:40AM - Most of the affected circuits came back up approx 15 mins ago, work is continuing to restore the remainder as soon as possible.

6:50AM - The remaining circuits have now re-established - all connectivity has returned to normal.

Browsing / Access delays

2010-03-26T10:56:00.002+00:00

Customers are experiences delays in reaching some parts of the Internet.

---

11:04am Sites giving problems are those we would normally access through Telehouse North in London, we have therefore closed down transit and peering at THN which is forcing traffic to take alternative routes through differant transit partners. this has impoved things to some sites however some others remain problematic. The issues causing these delays are located outside of our network and thus outside of our control, we are waiting on updates as to when these problems will be corrected, we will then enable once more peering and transit at THN. Apologies to those customers affected by this issue.

C2 21CN ADSL

2010-03-15T16:53:00.003+00:00

5pm - We're experiencing some issues with our 21CN connections failing to reconnect after a reboot, BT are investigating, if your connection is 21CN please do not do a manual reboot until the problem is resolved. Thanks.

8:37pm - Problems appear to be resolved, most of the disconnected sessions have come back online - incident should be considered closed, we will however continue to monitor for a while.

C2 DSL Network

2010-01-22T13:11:00.002+00:00

We're experiencing some issues on the DSL network, engineers are investigating, the problem is affecting multiple interconnects at multiple locations but does not appear to be core network related.