Cluster 3

[John]

Hello,

The e-mail has been prepared and will be sent to customers tonight/tomorrow. Apologies the the delay, but I wanted to include as much information as I could.

[Jester]

Cheers John

[ejsolutions]

Once again, thanks to John and The Team.
This time it is for the detailed and frank description (email) of the events and planned resolution. It appears that lessons have been learned.
Additionally, the recognition of the disruption caused, is well appreciated.

I'm sure some (many?) of us will agree, this is why we remain faithful to EUKHost.

Regards,
EJ

[patrick198]

Once again, thanks to John and The Team.
This time it is for the detailed and COLOR="Black"]free online movies[/COLOR] frank description (email) of the events and planned resolution. It appears that lessons have been learned.
Additionally, the recognition of the disruption caused, is well appreciated.

I'm sure some (many?) of us will agree, this is why we remain faithful to EUKHost.

Regards,
EJ

Great post! It's very nice. Thank you so much for your post.

[Jester]

Currently on Live support. All my sites down. Doesn't appear to be my IP banned on my firewall (normally my number 1 cause of not being able to access my server..)


So I ask, is this cluster down, again? I'm posting here because the forum seems to always get the best response from senior management.


I'm waiting for my phone to ring off the hook any second..

[John]

Hello,

NFS is currently disconnected on node ESX15 of Cluster 3. This is causing problems for a few people at the moment. We are currently investigating.

[Jester]

Thanks John for the update.

Is this related to the previous outage? If so, were changes implemented from last time to stop this occuring. Is there no redundancy the node and NFS?. I'm not an expert how these things work - but I was under the impression that clouds worked in a way that one broken connection could be picked up by another "route" so to speak - hence the 100% SLA

You get awards for comedy outage-timing however. I has just emailed a client, showing them their new website about 5 minutes before this fell over. Joy.

Anyhow. Can you provide an ETA once its available please..

[John]

Hi Jester,

It has been resolved. Restarting the VMs.

The problem came about as we were trying to switch over to what we put in place last time. There were a few conflicts with the old storage not wanting to go down without a fight.

VMs impacted (12) are being restarted now and we will check them all for FSCK.

[Jester]

Cheers for the reply John,

Would have been nice to have had scheduled upgrades done out of hours.. but hey-ho.

I'll be straight and say i'm not too pleased. Last time there was an outage, there was compensation. However, i'm more concerned about customers being happy than money in my pocket (although I never say no lol).

My biggest concern when anything goes down is email. Most my clients probably wouldn't be aware that their site is down unless they were told.. however, the second they try to send an email and can't (or worse, send out payroll emails etc) then they are straight on the phone to me. Is there a way that a mirrored email system can be established so that in the event of downtime, the email system can still run? That'd probably make my life perfect if i'm honest.

of course, now this is fixed, it'll never happened again, right?

Anyhow, thanks for being straight and updating immediately on the issue. That was very much appreciated.

James

[John]

Hi James,

Clusters by their very nature are worked on each and every day. Night time or day time. It can't be a case of doing certain tasks at only certain house of the day. Some times of the day are good for some but bot good for others. There isn't a good time of day for downtime at all.

The problem was something which we should have seen coming in this case. When we make mistakes I have no issue saying it. We messed up on this one and overlooked something very obvious.

I apologise for the trouble once again.

[Jester]

You mean the world doesn't revolve around my timezone? damn, I need to be more important - or rich.. or both. lol.

All good now though. Thank you again for resolving it so fast, and thanks again for coming on the forum to explain


James

[Smithy]

I have a client who said his VPS was down for 2 1/2 hours

[AlexP]

Dear Chris,
There might be other issue with his CloudVPS please send an email or come on live chat so we will investigate it further.

[John]

Hello,

All VMs from cluster3 are now operating from our iSCSI storage. The NFS storage will be kept for backups only now.

The problem resulting from the NFS storage wasn't because the storage wasn't up to the job, it was swapped because of the time needed to fix if a problem ever did come up. iSCSI storage takes less time to restore in the event of a major problem than NFS.

We have made a six figure investment for our latest iSCSI storage via BlueArc, who are a leading storage solutions provider. Although I can't promise that there will never be a problem ever again, I can tell you that we have spared no expense to ensure that our infrastructure is as robust as possible and we consider our uptime guarantee as very much achievable.

[ejsolutions]

... Last time there was an outage, there was compensation.

I (and my clients) haven't received that, yet. Assumed it will be in next billing cycle. Like yourself, reliability is paramount over pounds but it always helps when you can give a little cash back to clients. Kinda 'sweetens' the disruption and shows that you care.

[John]

I (and my clients) haven't received that, yet. Assumed it will be in next billing cycle. Like yourself, reliability is paramount over pounds but it always helps when you can give a little cash back to clients. Kinda 'sweetens' the disruption and shows that you care.

Hello,

This was added to your account on 08/02/2011.

[ejsolutions]

Thanks. Hadn't received a notification but will pass on to my clients at the end of the month.

[susiehaynes]

thanks for the fast action and responding the problem immediately.