Weycrest.Com Support Status

RE: Coreix :: Scheduled Network Maintenance – Sunday 30th November 2008

Location: STA.LON2

The Coreix engineering team will be conducting network maintenance beginning at 00:00AM GMT on Sunday 30th November 2008, lasting until 06:00AM GMT of the same morning.

Summary of downtime and customers affected:

Part 1 – None anticipated

Part 2 – Customers who take communal load balancing solutions will be affected by 30 seconds to 1 minute of downtime.

Part 3 – All customers, there will be an increase in latency and some packet loss. There will be a customer edge outage for approximately 2 to 5 minutes per rack whilst we migrate customers from one chassis to another.

Part 4 – None anticipated – Minor route changes are expected.

———————-

Part 1 – The operating system upgrade of core1.sta.lon2.coreix.net

Scope of work: The upgrade of the JunOS on core1.sta.lon2.coreix. The hardware will be taken offline cleanly and should have no impact on networking operations.

Outage: We do not anticipate any downtime during this maintenance section due to the Coreix network resiliency.

Rationale: Coreix performs periodic Operating System upgrades on its equipment to improve security and features on its equipment.

Part 2 – Upgrade the operating system on lb1.sta.lon2.coreix.net

Scope of work: The upgrade of the WebOS on lb1.sta.lon2.coreix. The hardware will be taken offline cleanly, upgraded and rebooted.

Outage: All customers who take communal load balancing services may see 3 or 4 packets lost during this maintenance (approximately 30 seconds to 1 minute of downtime), while the load balancing equipment fails over. Customers with dedicated load balancer solutions will not be affected.

Rationale: Coreix performs periodic Operating System upgrades on its equipment to improve security and features on its equipment.

Part 3 – The upgrade of dist7b.sta.lon2.coreix.net

Scope of work: As part of our expansion at the Stratford site (sta.lon2), we will be migrating customers from one distribution 7a and replacing the chassis of distribution 7b. The new chassis will still maintain the redundancy you expect from Coreix (dual supervisor, dual power supplies, multiple copper and optical cards for dual edge switching ports and core uplinks).

Outage: As we swap over to the new chassis customers will be migrated rack by rack. Since all Coreix edge switches have 2 uplinks to the distribution the downtime will be minimal since we will run the two in parallel. I have estimated 2 to 5 minutes per rack, however in certain instances the swap over will be significantly lower.

Rationale: We are increasing the port density of our distribution layer in response to an unanticipated rapid expansion of colocation, enterprise services and dedicated servers.

Part 4 – The addition of core3.sta.lon2.coreix.net

Scope of work: The Coreix network expands its peering arrangements on a daily basis and as we grow our routing requirements increase. As part of our expansion we are installing a third core router at the Stratford site. The main task of the router will be to mix our primary peering traffic (private peers plus public exchanges), and will act as another resilient node on our network.

Outage: This is a node addition therefore we do not expect any downtime during this period. Customers may see minor route changes for direct peers during this period as the traffic migrates from one core to another (however no traffic should be lost).

Rationale: This upgrade is in response to an increase in traffic and peers on our core routers.

This e-mail is to notify you of a power issue with one of the breakers in power distribution unit 4B/2 in the Coreix sta.lon2 data-centre that affected local power-strips.

Issue start: 11:50:23
Final Issue resolution: 12:19:13
Outage Duration: ~ 30 Minutes
Cause of issue: Main breaker trip on PDU 4B/2-A feed affecting local power-strips Weycrest apologise to customers for the delay in restoring the services. This was partly because it was neccessary to `fsck` the large disc array on one server which took some time after the power was restored. Though this was however the first power outage in this datacentre in four years but are investigating the possibility of connecting our equipment via “higher availability” power drops in future.

We have experienced a brief network outage at Coreix. Must equipment has come back on line but eris and xena together with a few customer VPS are currently down. Engineers are investigating and we’ll update you shortly.

There is currently a network outage affecting our servers in Redbus and IP House. A technician has been despatched to investigate.

This affects customers on mercury and pluto.

We will update as soon as we have more information.

Update: Just back from the datacentre: Service restored about 30 minutes ago. Problem caused by failed gigalink connection to the switch. - Ellis

We have been forced to shut down SMTP relaying on eris, for the time being due to serious abuse of the relaying service. We hope to resume this service as soon as possible.

Update: SMTP service was resumed 20 February.

Currently carrying out a php and apache rebuild on saturn.weycrest.com, which is one of our cpanel/whm reseller servers. Customers are requesting an number of additional features building in PHP including cURL support.

We are experiencing some difficulties with our Redbus and IP House network. Affected servers are mercury and pluto. An engineer is en route to restore the service.

Update: Sat Jan 19 14:23:16 GMT 2008

The network core was restored, all services functioning normally.

One or two customers hosted on WWW4 are reporting outages or slow loading of pages.

This is a slightly tricky one to sort, as according to our network monitoring the httpd server appears “up” all the time. If its a “proper” outage we get SMS’s of course, if the server doesn’t automatically restart.

Last time we had this issue its was caused by an over aggressive content grabber, that was slowly “DoS’ing” the server. Please bear with us while we track down the issue.

More server uptime stats…

Please note if you see less than 100% service it doesn’t mean a total outage, but perhaps only the loss of partial service. The uptimes quoted are an aggregate of all services, ie: httpd, httpsd, pop3, smtp etc,, so a small outage, say 5 minutes loss of smtpd one server will show towards total downtime on that server, when in fact websites, pop etc will still be functioning:

Basically 100% network performance over the past week.

We are pleased to announce that our Windows hosting server is now back on line, with all websites and email functioning normally.

We believe the issue may have been caused by a Windows update.