Network problems with 78.xxx & 87.xxx subnets..

[Rock]

Dear Customers,

All our shared servers, VPS Hosting nodes, Semi-dedicated host machines & Dedicated servers in the sub-net range of 95.*.*.*, 78.*.*.* & 87.*.*.* are facing problems at the moment.. The servers will be back to normal functionality shortly.

We still do not know the exact cause of this outage & will update you all with the exact reasons & the measures taken to avoid them. We still await an official statement from the DC. We appreciate the patience & understanding during this while..

[Andy P]

Rather frustrated I must say. Whats going to happen with regards to:

8. What is your uptime Guarantee for Dedicated servers?
Ans :- We offer 100% network and power uptime guarantee on all our dedicated servers.

[Rock]

Hi,

Here's an update received from our upstream network engineers :

"We needed to perform some emergency network maintenance as soon as possible. This has been scheduled for tonight, which we appreciate is very short notice. We need to reboot a router to install new software, & this reboot will take up to 45 minutes. We will do everything we can to speed up the process as much as we can & reduce the maintenance time.

The maintenance is to perform an emergency upgrade of Cisco software. We are using a Cisco VSS-1440 as part of our network core, & we have been experiencing some reduced performance with it today. There is no cause within our network configuration & set up of this, & it started to have a detrimental effect on some clients today. We escalated this to the Cisco TAC team, who have diagnosed a fault with the software on the router in the form of a memory leak. Cisco has supplied us with a new version of the software for the router which will fix the memory leak & slow performance.

The nature of this problem is that it will escalate as time goes on, which is why we want to apply the fix as soon as core business hours finish today. Please accept our apologies for the short notice, we hope our clients appreciate this problem was out of our control caused by Cisco software, & we are working as best we can to resolve it quickly. "

We apologise for any inconvenience may have caused, but we received this emergency notice on a very short period. Please do not hesitate to contact us through live chat or email support if you have any queries or questions regarding this maintenance.

[natehall]

Rather frustrated I must say. Whats going to happen with regards to:

exactly, i took out a second server less than a week ago - it was down on tuesday needing a network card, and now down again, add this to the downtime from about 5-6 weeks ago on my dedicated server and it means im not happy

Do you not have redundancy in the network equipment? If so there is no way you can realistically offer a 100% guarantee as there is a single point of failure

[Welshy]

We need to reboot a router to install new software, & this reboot will take up to 45 minutes.


90 mins so far and counting

[natehall]

We need to reboot a router to install new software, & this reboot will take up to 45 minutes.


90 mins so far and counting

I will await official response before i comment any more, other to say im not surprised

[Rock]

Rather frustrated I must say. Whats going to happen with regards to:

exactly, i took out a second server less than a week ago - it was down on tuesday needing a network card, and now down again, add this to the downtime from about 5-6 weeks ago on my dedicated server and it means im not happy

Do you not have redundancy in the network equipment? If so there is no way you can realistically offer a 100% guarantee as there is a single point of failure

We need to reboot a router to install new software, & this reboot will take up to 45 minutes.


90 mins so far and counting

I will await official response before i comment any more, other to say im not surprised

Hello Andy/Nathan/Lance,

We completely understand your concerns towards the downtime you're facing as I type. I can reassure you that the current situation does concern us too & we're unhappy about it as well.

We do have a second core router as a part of a VSS-1440 router cluster for redundancy, but the problem at the moment is that of the VSS-1440 technology, which was released as stable by Cisco a while ago now, has exhibited problems under live load. We do understand why that doesn't fill you with confidence, however, what has actually happened here is that two discrete problems have combined to leave us in a position where a reboot of a core router will cause a break in the entire network connectivity.. These problems are very rare & the ones in the past were caused by either the hardware or firmware of our Cisco equipment.

They have already provided us with an updated firmware apparently addressing the memory leak & hopefully that will end that particular problem. I'm sure they will be able to address the remaining problem just as effectively & once they do, the VSS technology should provide a very resilient point of failure. I'm sure you can understand that points of failure have to exist, its protecting against them that matters & I do wish to try & assure you that we do take this seriously. We are working as hard as we can to resolve it & get back to our normal levels of network redundancy..

I hope this helps to explain the situation a little further.. If you do have any further questions, please post them here or have a live chat initiated from our main website. Thank you for your continued patience & understanding.

[Andy P]

I'm sure you can understand that points of failure have to exist,

If we all agree on that, then we all agree that boasting 100% connectivity is naive to say the least

[Jester]

Typical! The time I decide to work throughout the night on some SQL work.. and the server goes down! I haven't even got any whiskey to drink! boooooo! lol

I presume its still playing up? I've got zero access to all my sites.. Not particulary happy about it - but, going wrong is something that technology never fails to do.

Any update on the status?

[Alex C]

It was back working for about 20mins now its broken again!!! Super!!!

[Gallier]

Smile, it's irritates people

[Jester]

Definately think that Eukhost should provide complementary Whisky for moments like these. Mine's a double.

[Rock]

If we all agree on that, then we all agree that boasting 100% connectivity is naive to say the least

This entire sequence of network failure came under the emergency network & security maintenance, where we had very little or no time to convey the message to our clients. Once again we do genuinely regret for the inconvenience for all our affected clients..

Typical! The time I decide to work throughout the night on some SQL work.. and the server goes down! I haven't even got any whiskey to drink! boooooo! lol

I presume its still playing up? I've got zero access to all my sites.. Not particulary happy about it - but, going wrong is something that technology never fails to do.

Any update on the status?

We do completely agree with you, but as mentioned earlier, it was something which really needed to be done in an emergency situation. It's nothing that we've to hide from you, I truly hope you do understand our concern as well.

It was back working for about 20mins now its broken again!!! Super!!!

There was a network hiccup which caused the servers to come online over the network for a short period & then become unreachable again. They'd be back accessible over the network within the next few mins.

[texterted]

A couple of my sites are down too... they aren't in the quoted IP addresses though!

Hurry up please?

[Rock]

A couple of my sites are down too... they aren't in the quoted IP addresses though!

Hurry up please?

Hi Eddy,

Your VPS is accessible to me, I think it's the firewall on it. I'll check & get back to you quickly..

[Rock]

Hi Eddy,

I checked your VPS & the firewall, everything appears to be in place. The name servers which are used by your domains/sites are hosted on the servers which are down at the moment, hence causing the sites to be temporary down due to failure in name resolution. Once these servers are back online, your sites would be back accessible again. Thank you for your patience.

[texterted]

No worries Jack, things break now and then I suppose even on the best of hosts, thanks for checking for me.

[Rock]

No worries Jack, things break now and then I suppose even on the best of hosts, thanks for checking for me.

You're welcome, Eddy. I sincerely appreciate yours & others' patience & understanding during such unforeseen problems. We'll definitely take all possible measures to ensure that such problems are avoided altogether in the future..

[Rock]

No worries Jack, things break now and then I suppose even on the best of hosts, thanks for checking for me.

An update, few of the servers are already back into network, which include the name servers. Eddy, your sites should be back online by now.

[Rock]

***UPDATE***

All of the servers are back online/accessible over the network.

In case your servers are still inaccessible, please get in touch with our Support Dept, either through live chat or by opening a ticket.

Please accept our sincerest apologies as the outage was already longer than intended. As updated from the DC network engineers, there was a complication with the new router firmware which was requiring additional troubleshooting, causing the delay.. We tried our best to get the services restored back to normal as quickly as possible.

Thank you all for bearing with us all the while we're trying to restore your services.