The excessive memory usage, weird quirks and incessant netsplits caused by various bugs in InspIRCd prompted this move, hence the lack of prior notice.

This should be the last of any major network maintenance for the foreseeable future

The network help pages will be updated shortly to reflect the changes to various commands.

All the other servers on the network will operate as normal.

Update @ 03:00: A very smooth, successful update

The Network Services page has been updated to reflect Atheme's commands and syntaxes.

At the same time, we upgraded to InspIRCd 2.0.0.

You can use irc6.J3di.org to connect using IPv6, however you will need an ISP that supports IPv6, or a 6-to-4 proxy. If you don't know what these are, it's likely you don't have them

If/when the other servers gain IPv6 addresses, they will be added to the irc6.J3di.org round-robin, just like the current IPv4 irc.J3di.org round-robin.

RapidSwitch have not yet provided an RFO (Reason For Outage) report, but they did say this:

At approximately 00:20 a network issue was detected affecting connectivity to the North cluster in RapidSwitch Spectrum House.

Our on call engineers were contacted and the problem has been resolved. Service was restored at approximately 01:35. Unfortunately we do not currently have any further details available at this time but will issue them as and when they become available.

An update from RapidSwitch [emphasis mine]:

At approximately 00:20 a network issue was detected affecting connectivity to the North cluster in RapidSwitch Spectrum House. The cause of this appears to have been a very large amount of malicious traffic directed at the RapidSwitch network. This traffic has unfortunately caused a routing issue within the RSH.North cluster.

Our on call engineers were contacted and the traffic has been removed from the network. Normal service was restored at approximately 01:35. This problem will have affected different clients in different ways. It will have ranged from no effect, to an increase in latency and some packet loss, to a more substantial loss of connectivity.

After establishing that it was the server that had crashed, and not RapidSwitch's network, the server was power cycled and came back up as expected

We were provided a solution by Cisco TAC at approximately 01:40 UK time. This was the third solution provided and we are pleased to say that it seems to have been effective.Network capacity and performance is running at normal levels, and over 99% of clients are back online.

And just after midday today:

Please be advised that we are now able to officially provide an all-clear notification to all clients regarding the network disturbances, outages, interruptions to service and emergency maintenance experienced by clients in RSH-North since Midday, Thursday 24th September. To clarify, normal service has been resumed for the large majority of clients since before 3am however we are now able to issue a full all clear.

Hopefully this means that the connectivity problems are over for the UK server

We will add the UK server back to the DNS round-robin, and cease the redirecting of the DNS records for the UK later this evening (assuming there is no more downtime).

Update @ 27/09 01:00: We have reverted the changes to the DNS records; they are now back to how they were before the downtime started.

We have set up a secondary/backup DNS service which will store a copy of the main DNS server's records, so that you are still able to resolve the US and DE server's IP addresses during any future downtime.

In addition, we have removed the UK server from the DNS round-robin for irc.J3di.org, and are also redirecting the records for the UK server to point at the US server until this crisis has abated.

For more frequent (and always available) updates, you might like to follow @J3di_IRC, @Fr3d_org or even @RapidSwitch on Twitter.

We'd also like to pass on RapidSwitch's profuse apologies for the disruption caused by this downtime.

This is due to RapidSwitch having to perform emergency network maintenance:

The maintenance is to perform an emergency upgrade of Cisco software. [...] The Cisco TAC team have diagnosed a fault with the software on the router in the form of a memory leak. Cisco has supplied us with a new version of the software for the router which will fix the memory leak and slow performance.

- RapidSwitch Support

They say that this maintenance should take no longer than 45 minutes, and that they will do all they can to speed everything up and reduce the maintenance time.

To maintain network cohesion (as much as possible), the DE server will automatically failover its link to the US server in the event it loses its connection, and cannot reconnect, to the UK server.

Update @ 01:50 25/09: RapidSwitch seem to be back online now. No word from them yet as to why the maintenance took so long; we will update this post when we know more.