Some customers have reported latent and/or dropped packets along various routes from CariNet today. We have examined the health of our network in all data centers and determined that the latency is not originating within our network. We have escalated the issue to our upstream providers but do not have anything definitive as of yet.

If you are experiencing problems, please open a support case in your iBizPanel account and provide us with a traceroute to and from your location to assist with narrowing down exactly where the problem is occurring. Thank you for your patience as we focus all of our resources on resolving this.
———————————— UPDATE —————————————
Our upstream providers have reported no problems, and the traceroutes we are running today are all clean. If you are seeing otherwise with your own connection, please send us a traceroute and we will investigate further.

In an ongoing effort to provide the highest level of network and IP services to our customers, CariNet will hold a network maintenance window this Friday, March 27th between 11:59pm and Saturday 1:00am, PDT. The purpose of this maintenance window will be to complete a project which started three months ago during a previous maintenance window.

During this time, CariNet will be upgrading one of the core routers.

IMPACT – Routing adjustments will occur as connections are moved to the new router. Customers with active sessions may experience latency as these changes are made.

================
UPDATE:
In an ongoing effort to provide the highest level of network and IP services to our customers, CariNet will postpone the network maintenance window this Friday, March 27th until a future date. While planning this maintenance, CariNet received notification that our upstream provider, Level3, will be holding a maintenance window Saturday morning between 3 and 6am. To prevent unnecessary network latency to our customers, we have decided to not hold our window at this time.

During this time, Level3 will be upgrading the OS on their San Diego core routers.

IMPACT – Routing adjustments will occur as the router CariNet is connected to is rebooted to load the new OS. Customers with active sessions may experience latency as routes change.

In an ongoing effort to provide the highest level of network and IP services to our customers, CariNet will hold a network maintenance window this Friday, March 13th between 11:59pm and Saturday 1:00am, PST. The purpose of this maintenance window will be to continue with a project which started two months ago during a previous maintenance window.

During this time, CariNet will be installing a set of new distribution switches to service the C5 data center.

IMPACT – Routing and Spanning Tree adjustments will occur as the switch connections and VLAN gateways are moved within the C5 datacenter. Customers with active sessions may experience periods of latency as these changes are made.

In an ongoing effort to provide the highest level network and IP services
to our customers, CariNet will hold a network maintenance window this Friday, March 6th between 11:59pm and Saturday 1:00am, PST. The purpose of this maintenance window will be to continue with a project which started two months ago during a previous maintenance window.

Level-3 10Gig Bandwidth Integration ? CariNet recently brought a new 10-Gig connection to the Level-3 network online. The connection has been “live” and in test-mode for the past 2 weeks. We are now ready to integrate this connectivity into our BGP mix. This work will happen during the Friday, March 6th window.

IMPACT – Routing adjustment will occur as the new connection is brought
online. Customers with active sessions to other carrier networks may experience
latency as the new routes are populated.

In an ongoing effort to provide the highest level of services to our customers, CariNet will hold a maintenance window on Friday, February 27th starting at 11:59pm, and lasting no more than 20 minutes.

The CariNet Systems team will be performing routine maintenance and redistribution of all Caching and Authoritative DNS servers, in order to enhance server performance and improve redundancy. During this time customers may experience slow DNS lookups if using the IP addresses below as part of any server’s network settings.

Caching:
216.98.128.204
216.98.128.70
216.98.128.160
216.98.128.132
209.126.190.70

Authoritative:
209.126.190.71
216.98.128.74
Branded DNS for ASPadmin

At 12:15am the C3 router experienced a line card failure. Network engineers were on site within 15 minutes. Once the issue was identified the affected card was rebooted to bring services back online. The decision was made to replace the suspect card resulting in a second reboot of the affected card at 12:50am. All systems were running normal by 12:55am.

This issue primarily affected CariNet corporate systems like the iBizPanel and www.cari.net. We are not aware of any customer servers being adversely affected in any way. If you are having any difficulties currently, please call our Support Center or open a support case inside your iBizPanel account.

In an ongoing effort to provide the highest level of network services to our customers, CariNet will hold a network maintenance window this Friday, December 26th starting at 11:59pm, lasting no more than 20 minutes.

The CariNet engineering team will be installing a new transit router in the C5 data center. During this upgrade, the Level3/Williams transit connection will be temporarily shut down and network traffic will be re-routed while the distribution links are moved from the existing transit router to the new equipment. Once this new equipment is in place and operational, the Level3/Williams connection will be restored.

Customers in the C5 data center will experience an interruption of service (up to 20 minutes) while the links are moved from the existing equipment to the new gear and testing is performed. All other data center customers may experience brief periods of latency as network traffic is routed away from the Level3 and back after the installation is completed.

We are currently experiencing routing issues in our C5 data center. Our full network team is working on the problem. More details will be provided as they become available.

Update
All customers should be back up now. Please give us a call or open a support case if you are having any continued connectivity issues.

We had a problem with routes on one of our primary upstream providers, and had to shut down the connection and perform BGP Recalculations. Downtime varies with the customer, but it should have been no more than 20 minutes. Further details are pending.

CariNet will be conducting a network maintenance window to provide additional protection and redundancy for the C5 Data Center.

On Wednesday, Nov 26th, the CariNet Network Service Team will make changes to the spanning tree topology in our C5 data center. This will improve redundancy and provide additional protection to our distribution equipment in case of a failure. The maintenance will begin promptly at midnight and is expected to take approximately 3 hours. Service impact should be minimal, though customers located in C5 may notice brief periods of latency and intermittent loss of connectivity during this short window as STP recalculations are completed. Senior Networking Team members will be on-site managing this event.

At approximately 11:00am today, the CariNet BizMail cluster started having connectivity problems. It was investigated immediately and the cluster was back online by approximately 11:30am. This is normally a very stable system so the cause of this anomaly is still under investigation.

Please note that the DisH mail network was not affected during this outage in any way. External mail clients were still able to connect to individual mail servers without any problem. Only the BizMail interface was affected.

We apologize for any inconvenience.