The reason the network had to be relocated was due to a change in ownership of RedBus Meridian Gate. RedBus was recently acquired by TeleCity and renamed to TelecityRedbus [TeleBus]. After the exchange TeleBus then proceeded to cancel all existing contracts and put new contracts in place.

In some cases these contracts had price rises of over 400%. We have been told that one of the reasons for these increases was due to power. Because the Meridian Gate data centre is within the London 2012 catchment area a large portion of power has been reserved for the Olympic Games. Unfortunately adding servers in a datacentre is not just a case of installing additional sockets or extension leads. In some cases power needs to be brought in from additional suppliers and even new sub-stations are required. This has caused the cost of additional power to increase massively to TeleBus who are then attempting to reclaim this back from their clients. In addition there is also speculation that TeleBus are moving more towards major corporate clients like Banks and major corporations as a way of quickly increasing revenue in preparation for a sale or flotation.

We are not a direct customer of TeleBus simply because we do not have the volume to warrant purchasing direct. However our rackspace supplier is a direct customer who has over 60 racks with TeleBus (almost an entire floor). This equates to hundreds of servers. They were unsuccessful in being able to negotiate a staged increase leaving pretty much the option of pay up or leave.

The decision was taken to relocate to IPHouse which is outside of the London 2012 catchment area and they are largely unaffected by the problems this created although still susceptible to general power increases. IPHouse is a privately owned datacentre well connected to the internet backbone and only a short drive from TeleBus.

Originally the work was scheduled to take place over the weekend of the 9th June. However we were told that TeleBus became very obstructive and the move had to be abandoned.

On the evening of the 13th June we were informed that some of the routing equipment was relocated to the new data centre which caused a brief period of outage. This was in advance of the main servers in an attempt to keep the main downtime to a minimum.

During the 14th June we received word that TeleBus was going to turn off power to the floor at 18:00 hours and the equipment had to basically be out by then. We then made an announcement to this effect. Our rackspace supplier then managed to negotiate an extension with TeleBus so the servers could be moved during more off-peak hours.

Our servers were taken offline at approximately 04:00 BST (GMT+1) 15th June 2006.

Then some severe bad luck as part of the Blackwall Tunnel was shut due to flooding causing huge traffic headaches and a massive increase in traveling time between the two locations.

I personally was not involved on the actual move since our rackspace supplier was managing the whole process and I was confident in their abilities given they run a suite of over 60 racks and the apparent plans they had put in place. I was told throughout the day that there were some issues with the routing. I am not fully confident that this was the main reason for the delay. I am unprepared at this time to reveal who the rackspace supplier is for a number of reasons. However I feel that I have been badly let down and given half truths throughout the day as to the reasons for the extremely lengthy downtime. I do accept that this is my fault for placing such trust in one of our suppliers.

Service was fully restored at approximately 12:30 BST (GMT+1) 16th June 2006.

It is very unlikely I will be changing rackspace supplier given the downtime and inconvenience this has already caused, changing suppliers now would just make this worse. Instead I intend to have a full and frank debrief with them to ensure a clear understanding is in place of what is expected in the future. Furthermore any moves or changes of this nature I will ensure that either myself or an AGUK representative is present, on site, at all times.

There are also a couple of other points that have come out of this move. Firstly we do operate a backup mail server system which has worked throughout the downtime. Mail delivered while the main server was unavailable has been queued by the offsite backup server. Then when the main server became available again this mail has been delivered. However as a further level of redundancy we will set-up an additional offsite mail server. This server will maintain a recent copy of the main mail server’s configuration. Then if there is an extended outage such as this again we will be able to route mail to this additional server allowing collection of mail to be restored fairly quickly.

One of the other points that have come out of this move is the way we operate our support system. Currently the support system is hosted on the same network as our main servers. This means at times like these we are unable to respond to support requests and queries. While we do continually update www.viewnetworkstatus.net we also need to be able to continue to communicate with individual customers who have additional queries. Therefore we will be moving our support system to a separate network in the same way we have with the Network Status Page.

I apologise to all customers for the continual changes in the schedule and the unacceptably long downtime experienced. The move was unavoidable and had to take place. However the actual handling of the move was unacceptable and will be thoroughly reviewed.

If anybody has any queries or problems now that the network is restored please feel free to contact us.

Regards

Andy Gambles
AGUK Solutions Ltd