Weycrest.Com Support Status

We appear to have suffered a Router failure in IP House. This is currently affecting customers on pluto.weycrest.com and a small number of VPS customers. Engineers are currently investigating and we hope to update on progress shortly.

Update: Sun Nov 25 09:52:11 GMT 2007: We gather there was some kind of technical incident at IP House overight, which has led to a number of outages. At the present time we are still awaiting an access code in order to gain access and fix things.

Update: Sun Nov 25 17:58:17 GMT 2007: Power and networking were  restored by 3.00pm this afternoon. The failure of a power rail in IP House did quite some damage requiring a rebuild of the router. Our apologies for the inconvenience caused.

This is the September monitoring report provided. We hope to automate this with a dynamic status page hosted on an independent web site, as we intend to be completely transparent about our hosting performance:

Please note if you see less than 100% service it doesn’t mean a total outage, but perhaps only the loss of partial service. The uptimes quoted are an aggregate of all services, ie: httpd, httpsd, pop3, smtp etc,, so a small outage, say 5 minutes loss of smtpd one server will show towards total downtime on that server, when in fact websites, pop etc will still be functioning:

This afternoon we have suffered a major network outage which took our some shared hosting on the mercury server and some reseller and VPS hosting currently housed in IP House.

This was caused by a failure of a bgp4 router which has now been replaced. Delays were exacerbated by London traffic which delayed getting a network tech on site. We apologise for the downtime and any inconvenience caused.

Hosting in the Coreix datacentre was unaffected.

There are mail issues at the current time due to a customer running a very large mailing list through the server. We have suspended the account in question and have temporarily suspend mail service until we can clear the mailq. Please note none of our shared hosting packages, email or web hosting are intended for large mail list hosting (which requires a dedicated server).

UPDATE: Wed May  9 20:42:45 BST 2007

Problems resolved, all services and load average back to normal.

We carried out a number of hardware node reboots to load the latest Virtuozzo kernel. Some of the machines took longer to bounce back than expected as they had been up a long time, the machines forced a filesystem check, which due to the large arrays can take some time.

On the shared hosting stack, eris (Plesk 8.1) was affected as was xena (cpanel/WHM) on the reseller hosting stack.

We have received this response from TelecityRedbus

Update from Redbus

Dear Customer,

We experienced a short power outage today which affected the North side
(right hand side as you come onto the floor) of Sovereign House. Power
has now been restored to your systems on the data floor. The entire
electrical system is currently fed through ‘raw mains’. We will update
you as and when new information is received.

All available staff have been diverted to Sovereign House to resolve any
issues that you may still be experiencing. Please log your tickets
through the portal or phone in for our staff to deal with. Please
contact Sovereign House NOC to log any support requests on +44 (0) 207
001 0050.

Obviously we apologise for the inconvenience.

There was another major power outage at Telecity Redbus’s increasingly accident prone Sovereign House Data Centre today (12th March 2006). This will have affected a number of hosting providers. We are still awaiting an official explanation.

Fortuneatly we managed to get a tech onsite very quickly, but for a while he was locked outside while Telecity Redbus Techs, laboriously checked everyones security credentials, just to ensure there were no terrorists with WMD’s posing as server technicians. At one point tempers were getting a little fraid.

We don’t have as much as we used to in Redbus, as we are in the process of migrating to other DC’s (in London and in Europe). After all that all our stuff bounced back up quite quickly (it has to being in a Redbus rack).

However there were residual network outages caused by the fact that some of our upstream transit providers were still trying to get their routing acts togethor (presumably their techs still hammering on the door, trying to get into the building).

This caused the server load on our mercury server to hit the roof for a while, but matters have now stabilized.

For the non techie minded one of the main benefits of colocating in a Data Centre is stable power supplies and redundancy - we live in hope!
The Register has now picked up on this:

http://www.theregister.co.uk/2007/03/12/telecity_power_failure/

Incidentally our brave security services have “foiled” another plot to blow up nearby Telehouse. “Foiled” as in, `no one has actually been arrested` but some groups may have carried out “reconnaissance” according to a Senior Whitehall Spin Doctor (oops sorry, we mean “official” ;o))

http://www.timesonline.co.uk/tol/news/uk/crime/article1496831.ece

SWsoft have just announced the release of Service Pack 1 for Virtuozzo for Linux. This includes improved tools and features including:

• Red Hat Enterprice Linux 4.0 Support
• Ethernet Layer Network Adapter Support
• VLAN Support
• Virtual CPU Management

Our VPS Hardware Nodes have been upgraded with the new tools. We have also implemented secure /tmp (using the bindmount function) in all Virtuozzo VPS’s.

Outage - Reseller and VPS customers will notice a brief outage lasting 1-3 minutes whilst cables are switched between old and new distribution switches

Reason - The new Cisco 3750G distribution stack will increase both performance and redundancy of the distribution core.

Another fairly convincing email that appears to come from Barclays Bank but doesn’t. It actually directs to a scam site using subdomains of thirdt.info.

.info domains are now getting an underserved “reputation” largely because of special promotions making them extremely cheap to register, so favoured by spammers, and sploggers.