MK Problem

[John]

Hello,

We are currently investigating a problem in MK which seems to be impacting a number of servers. We will update this thread when we have more news.

[John]

Hello,

The issue is relating to a power issue on the B side of MK. Although the UPS and generators have kicked in, and power is still coming from the A side, there are a few racks impacted by this.

Cloud VMs are unaffected by this outage.

[John]

Hello,

The B side lost power for approximately 20 seconds. All racks receive power from the B side and A side together, which is why most things are up. However, 3 (so far) of our racks appear to have fallen over with the switchover and we are investigating these as we speak.

[nickbits]

Any update. I have been waiting on LiveChat for 5+ minutes for an update on my servers, which are still down, presumably because of this.

[FrankT]

Hello,

The power issue is now resolved and we are now checking each server manually to ensure that they came cleanly.

This should not take too long.

Frank Tighe

Senior Technician

[nickbits]

Frank,

Thank you for the update. More useful than Live Chat, has taken 20 minutes to get just 4 lines from them, none of which was of any use.

Can I raise a few issues/points. I have been an EUKHost member for 3 years or so starting with shared and working up to VPS. I was looking at dedicated but am now holding off on that. Anyway over the last 6 months I have noticed an increasing number of issues, several times the MK centre has been an issue, a DDNS attack which I appreciate is totally beyond your control, but other power outages and broken hardware as well. The two questions I would like answered are:

1. What is being done to prevent this from happening again?

2. Why are we not being kept up to date? I get very annoyed with issues such as this cropping up and would be a little less annoyed if I was kept in the loop. I have raised this before and it was my impression that the forums would be kept more upto date on such issues and that twitter and facebook would/may be used as well.

Nick

[chieflorenzo]

Frank,

Thank you for the update. More useful than Live Chat, has taken 20 minutes to get just 4 lines from them, none of which was of any use.

Can I raise a few issues/points. I have been an EUKHost member for 3 years or so starting with shared and working up to VPS. I was looking at dedicated but am now holding off on that. Anyway over the last 6 months I have noticed an increasing number of issues, several times the MK centre has been an issue, a DDNS attack which I appreciate is totally beyond your control, but other power outages and broken hardware as well. The two questions I would like answered are:

1. What is being done to prevent this from happening again?

2. Why are we not being kept up to date? I get very annoyed with issues such as this cropping up and would be a little less annoyed if I was kept in the loop. I have raised this before and it was my impression that the forums would be kept more upto date on such issues and that twitter and facebook would/may be used as well.

Nick

I agree with you Nick. It's got too much. We're meeting with another hosting provider and likely moving everything away next week.

[Cristiano]

Dear Customers,

We apologize for all the inconveniences caused due to this downtime.

All the servers are powered on & should be up and running now. If you still face any issues, PM me with the server IP OR contact us via Live Chat

@nickbits :: You faced a delay in responses as at the same time, there were many chats being attended by the technicians. Also, at an initial stage we didn't had any update about the exact reason of the downtime.

We are waiting for a detailed failure from the UPS manufacturers as to what exactly went wrong. A full report will be posted once we have further updates.

Your patience and understanding is much appreciated.

[nickbits]

Cristiano,

Thanks for the info. As for Live Chat, it is not just today, I have been having the same for the last 3 months that a simple chat, 4 lines can take 30mins easily with me waiting 10mins between feedback from your side. I always swore by EUKHost, but I do feel that the Live Chat is really letting you down, especially over the last few months. Forums and tickets give much better feedback, but are obviously slower.

As for the problems and reports, initially all that was required is to say what was affected and give us progress reports, even if it is "still working to resolve the problem" or "waiting for an engineer." You could DM (via twitter?) those affected if you don't want it to go on the forums. Just something to keep us in the loop, especially when it is a long outage as this was, which I doubt we will be compensated for.

As for the report, I look forward to seeing it, I can point you to a decent UPS manufacturer in Aldermaston if you need one!

Nick

[Cristiano]

Thanks for the info. As for Live Chat, it is not just today, I have been having the same for the last 3 months that a simple chat, 4 lines can take 30mins easily with me waiting 10mins between feedback from your side. I always swore by EUKHost, but I do feel that the Live Chat is really letting you down, especially over the last few months. Forums and tickets give much better feedback, but are obviously slower.

Could you forward me the chat transcripts [cristiano@eukhost.com] ? I will have a word with our technicians & will address them accordingly.

As for the problems and reports, initially all that was required is to say what was affected and give us progress reports, even if it is "still working to resolve the problem" or "waiting for an engineer." You could DM (via twitter?) those affected if you don't want it to go on the forums. Just something to keep us in the loop, especially wh
en it is a long outage as this was, which I doubt we will be compensated for.

I will have a word with our management & will see as to what can be done. Still, won't be able to commit about the same.

As for the report, I look forward to seeing it, I can point you to a decent UPS manufacturer in Aldermaston if you need one!

Much appreciated ! Again, will have this forwarded to our management.

[nickbits]

Thanks for that, will dig out the transcripts at somepoint over the weekend.

Don't mean to complain, I and others I am sure, would feel less frustrated if information about critical issues flowed out better and more regularly...

Thanks,
Nick

[John]

Hello All,

I'd like to explain a few things about the setup in MK and it will hopefully help explain how we do things a little better.

Firstly, MK is an independently rated Tier 4 Data Center. In Data Center quality terms, you can't actually go higher, but of course there are certain things you can do per rack to make a robust or not so robust setup. Onto the racks..

Our racks are fed power from Side A and Side B. This is either done through a feed from Side A and Side B going into two different server PSUs, or through the two feeds going into a changeover power bar which can operate with a single feed in case of failure. In terms of rack totals, we are talking hundreds with these setups in MK.

For whatever reason today, Side B lost power for 20 seconds. In a good 98% of racks (yes, really, no exaggeration) no issue was noticed as they were also getting power from Side A. However, for a select few racks, this wasn't enough and it caused them to lose power. Quite strange since it worked so perfectly on the others.

I know a few of you have pointed to the power loss as the overall problem, and I don't blame you for reaching that assumption, but from my point of view I see the fact that the dual feeds weren't enough for a few racks as the service disruption cause.

I'm sure you have noticed, just as we have, the same racks impacted first by the AC problems on Tuesday were the ones which had problems today. I'm not a big believer in coincidence and this is something we will be looking into it. If the AC problems on Tuesday have made these racks more susceptible to failure than the others or if the fact there is a reason these racks failed on Tuesday above all the others is a question we will be investigating very deeply over the coming days.

I'm aware, rather unfortunately, that servers hosted in these racks have not had a great time recently in regards to VPS performance, which I'm sure makes the whole experience even more painful for you guys.

In terms of what we would do differently, the honest answer is nothing. There isn't actually a way to make this setup more robust than it is, hence why so few racks failed, so we are instead looking at either the AC problems causing damage on Tuesday or some underline manufacturer problem in these few cases.

As soon as we have an official word of the cause of the power issue today, we will post it. We will also announce our findings in relation to why these racks were so servery impacted when others didn't have a problem.

In terms of downtime announcements, we were on this one very quickly and forum post was made near instantly. I also followed up as much as possible. Live Chat is a comforting thing and I know all too well why so many of you like to use it when there is a problem. You essentially want to feel someone is there and working on your issue. Unless you are very fortunate in who accepts your chat, as it would need to be someone very senior to have the latest up to the minute info, the forum is going to contain the latest news.

Our forum is the first place we will post a service disruption and it is the first place we will update. Going forward our new billing system will give us more creative options.

Thanks for your patience today.

[mdlp]

My VPS was afftected in this outage but did not come back automatically. Live chat got it restored but there was a considerable delay utill I could get to a computer and contact them. I appreciate that the outage was hardware issue but the customer should not have to confirm that his own VPS is working. It is lucky it happened today as if tommorow I would not have found out till Tuesday.

The only explanation from live chat was
Diego: your server was in mounted stated
Diego: state*
Diego: I have restarted it manually

Which means nothing to me as an explanation. I've been with EUKHOST for 3 years and at times it's been a rocky road but this last 3 months have been frustrating. My package is due for renewal in June and I have been seriously looking at other options.

This is hobby for me and is supposed to be fun but lately fun it ain't. I find myself getting angry and frustrated.

Martyn

[John]

My VPS was afftected in this outage but did not come back automatically. Live chat got it restored but there was a considerable delay utill I could get to a computer and contact them. I appreciate that the outage was hardware issue but the customer should not have to confirm that his own VPS is working. It is lucky it happened today as if tommorow I would not have found out till Tuesday.

The only explanation from live chat was
Diego: your server was in mounted stated
Diego: state*
Diego: I have restarted it manually

Which means nothing to me as an explanation. I've been with EUKHOST for 3 years and at times it's been a rocky road but this last 3 months have been frustrating. My package is due for renewal in June and I have been seriously looking at other options.

This is hobby for me and is supposed to be fun but lately fun it ain't. I find myself getting angry and frustrated.

Martyn

Martyn, I'll forward this on, as it is incredibly easy for our techs to see a mounted VPS (it actually says mounted!).

Apologies for your issues.

[WelshTom]

It seems the power failed again at 7:22PM - 8th of May.

This seems very odd for a Tier 4 datacenter, what keeps causing this?

[Cristiano]

It seems the power failed again at 7:22PM - 8th of May.

This seems very odd for a Tier 4 datacenter, what keeps causing this?

Hi.. No issues with power failure in MK y'day. Have had confirmed with the network engineers too.

[WelshTom]

Hi Christiano,

I spoke to John who confirmed that some of the servers were facing some sort of DDoS attack and were rebooted in the process.

All seems to be working okay now.