Network Status

Here you’ll find the latest updates on any Tagadab service incidents, including major service outages and planned engineering works.  In the unlikely event that you experience a service interruption, please check here for an update before you call us.

Service
Maintenance
Upgrades

Current Network Status

04/07/2015 14:35 - An update on SSD cluster performance issue: Following the power loss this morning one of the five nodes in the cluster failed to boot. The cluster is tolerant of the failure of a node, however for reasons we do not yet understand the nature of the power resuming all at once meant that the other nodes in the cluster did not recognise that the node was down. They did correctly note that the disks in the node are unavailable and offlined them as intended. The cluster is therefore still operating on the basis that all nodes are available (but not all disks) and we believe this is causing the performance issue. A new node is now being installed to replace the failed one. We do not yet have a time for a fix.

04/07/2015 13:36 - Our SSD cluster has suffered a failed node (one node of five) as a result of the power issues. It is operating but its performance is suffering. We are working with the vendor to resolve the issue. All Cloud services and SSD VPS are affected. Some services are actually unresponsive because increased I/O times cause memory issues.

04/07/2015 10:44 - The majority of services are back to normal, however there are still some systems that are offline. We are working very hard to get everything back to normal, so please accept our apologies for not replying to emails so far. Once we are satisfied that all monitored services are back to normal we will post a further update asking you to let us know if your service is still unavailable.

04/07/2015 07:59 - Mains power has now been restored to the data centre. It will take some for our core systems to come online, and only once that has been done can we determine which services have failed to start and need manual intervention.

04/07/2015 07:54 - We're starting to piece together the situation in our Romford data centre, although the following information is unconfirmed and therefore subject to change. Overnight there have been widespread power outages in the Romford area which appear to have been caused by a lightning strike.  In this situation equipment should continue to be supplied with power from the UPS system whilst a backup generator is started. For an as yet undetermined reason the backup generator failed to start, and power was lost once the UPS batteries were exhausted. We are working to get power restored as quickly as possible.

04/07/2015 07:02 - Data centre engineers are currently looking at a power supply problem at the Romford data centre.

04/07/2015 05:50 -  There is currently no network connectivity at our Romford data centre. Data centre engineers are currently investigating the cause. All customer services are affected as well as the customer control panel, and we are currently unable to reply to support tickets. This page will be updated with the latest information. 

04/07/2015 05:15 - Engineers are currently investigating a network outage at our Romford data centre. More details shortly. 

 

Planned Works

There are currently no planned engineering works.