LINX - Rapidswitch network issue

[DrydenM]

As the other thread is closed can someone from Euk please keep me updated on the LINX - Rapidswitch network issue that is affecting various Euk servers.

I notice the route for traffic keeps changing and I am still getting time outs.




I would also like to know, can you go multihomed at the datacentre in question and are you going to imminently (I do not know the details of your arrangements there and do not want to bug my mate for any more information if I can help it)? I am told you rely totally on rapidswitch there and they in turn have insufficient peering for resilience which is why we have had connectivity problems since 6pm.

I also have to wonder what would happen if rapidswitch went bust, no one there turned up for work and power was cut to rapidswitch equipment.
What would be the Eukhost contingency plan for this scenario?

Thank you.

[eUKShane]

Hello,

Other peering is in place but some routes were more reliant than others. It can also depend which individual ISP you use for your own connection, as some ISPs are heavily reliant on LINX.

A delay comes from stopping traffic going over LINX and then passing traffic over it again. At the moment it is working as it should be, but still some providers haven't enabled it again. This can create certain problems for a few users using certain ISPs, but it is something outside of our network.

In terms of Rapidswitch, you ask an interesting question. We have offsite backups of all data (another DC) but I don't recall a DC going bankrupt for a long time, if ever in the UK. That shouldn't be something you should be worrying about, as it is extremely unlikely to ever happen and you could say the same for any DC. Playing the "what if" game is not a good idea.

[serverdown]

There is still a problem with connectivity via various isp's to my vps.

I can use proxy servers and it is very fast and on my isp connection using google DNS, open DNS and my ISP DNS I experience a flaky connection.

Same with my clients (notably ones based in London, Manchester and Lincoln - I am in south Wales).

When will the connectivity issues be fully resolved? Are EUK still monitoring this? Do they even care?

I appreciate that this might not be something under your control, but it is obviously affecting a number of clients.

[Welshy]

problems here still.... also in s wales....

using bt internet, that runs through telehouse which is a linx partner...

why is it though I can get all of our other servers... its just the ones at rapidswitch that are the problem...

I know there are linx issues as thats being reported on other forums also...

getting a bit tedious now though

cant even get a ping response from 2 servers... yet can get the pages ok.. well sometimes unless they 404 due to the packet loss.. mail is also timing out...

[serverdown]

cant even get a ping response from 2 servers... yet can get the pages ok.. well sometimes unless they 404 due to the packet loss.. mail is also timing out...

same here, pages hang.... intermittently

I've optimized databases and apache and done allsorts to limit the impact but to no avail - even upgraded vps to top spec (thanks euk for suggesting that as a solution)

Couple of clients are really doing their nut. Euk not really that bothered. Thanks.

[serverdown]

Additionally frustrating is seeing how fast the sites are loading through proxy servers and at the times when they are loading here.... boo!

[serverdown]

What i have also noticed is that pages hang, but if you refresh they load ok.

Obviously fine if you know that, but not great if you are selling something to a new customer....

[Welshy]

thats due to hitting them when theres a gap in the data response/timeout refreshing hits them when they are responding....

[serverdown]

yup, like i say, fat lot of use and no concellation to clients.....

It's soul destroying, I have a 1001 other things I need to be doing

[Rock]

There is still a problem with connectivity via various isp's to my vps.

I can use proxy servers and it is very fast and on my isp connection using google DNS, open DNS and my ISP DNS I experience a flaky connection.

Same with my clients (notably ones based in London, Manchester and Lincoln - I am in south Wales).

When will the connectivity issues be fully resolved? Are EUK still monitoring this? Do they even care?

I appreciate that this might not be something under your control, but it is obviously affecting a number of clients.

problems here still.... also in s wales....

using bt internet, that runs through telehouse which is a linx partner...

why is it though I can get all of our other servers... its just the ones at rapidswitch that are the problem...

I know there are linx issues as thats being reported on other forums also...

getting a bit tedious now though

cant even get a ping response from 2 servers... yet can get the pages ok.. well sometimes unless they 404 due to the packet loss.. mail is also timing out...

same here, pages hang.... intermittently

I've optimized databases and apache and done allsorts to limit the impact but to no avail - even upgraded vps to top spec (thanks euk for suggesting that as a solution)

Couple of clients are really doing their nut. Euk not really that bothered. Thanks.

Additionally frustrating is seeing how fast the sites are loading through proxy servers and at the times when they are loading here.... boo!

What i have also noticed is that pages hang, but if you refresh they load ok.

Obviously fine if you know that, but not great if you are selling something to a new customer....

thats due to hitting them when theres a gap in the data response/timeout refreshing hits them when they are responding....

yup, like i say, fat lot of use and no concellation to clients.....

It's soul destroying, I have a 1001 other things I need to be doing

Hi all,

Thank you for your continued patience during this entire incident which unfortunately was beyond our call of control. I'd like to update you further on this case.

Today at 13:10 we received an update from LINX to say that their emergency maintenance steps had restored stability & they were advising all peers to re-establish sessions.

We followed that advice & began restoring our LINX sessions in the datacenter.. This process was completed at 13:48, however a small number of our peers have kept their sessions administratively down. This is their choice but may impact traffic if you would normally cross their network to reach us..

LINX are planning further 'tidy up' maintenance tonight, but according to their descriptions of the works, they do not affect the Eukhost or any other particular datacenter which falls in our IP space. We are hopeful that this is a conclusion to this incident & that LINX will maintain the resumed normal service as earlier.

If any one is still experiencing difficulties please do not hesitate to contact us either through Live Chat or Email/Ticket support.. We will do what we can to investigate any issues & resolve them for you if we can. However, please bear in mind that the cause of all the problems over the last 24 hours has been outside the Eukhost network & we may not be able to act for you.

We do understand that this is affecting you, but it's also affecting us & most of the ISPs in the United Kingdom, as LINX being the premium source upstream provider.

Thank you once again for your understanding..

[Welshy]

Hi Rock,

Have stated in the closed post by John that it was external to EUK, hence not alot can be done... its a pain that its only affecting servers at rapidswitch... as all other servers are accessible on a different IP range.... worst thing you guys did was move some servers from BS to RS... first the cisco catastrophe, now routing... not blaming RS here but its only that DC we cant get too properly...

I think in the new year we will need some server migrations out of that place

[Rock]

Hi Rock,

Have stated in the closed post by John that it was external to EUK, hence not alot can be done... its a pain that its only affecting servers at rapidswitch... as all other servers are accessible on a different IP range.... worst thing you guys did was move some servers from BS to RS... first the cisco catastrophe, now routing... not blaming RS here but its only that DC we cant get too properly...

I think in the new year we will need some server migrations out of that place

Yes, we'll be moving out few or most of our servers out of that DC very soon & it has been stated here short while ago:
http://www.eukhost.com/forums/f10/cl...html#post62968

[sihost]

are the dedicated servers in this datacentre?

[Rock]

are the dedicated servers in this datacentre?

Yes, this involved moving dedicated servers as well.

[sihost]

Yes, this involved moving dedicated servers as well.

will you be moving them to milton keynes where its tier 5? so no power outages or anything like that. If so thats good because I'm signing up to dedicated at end of december

[Welshy]

well i just lost my ISP connection, called their support and they advised they were rebooting stuff due to a routing outage... I asked if it way linx related and they advised yes...

just come back online and I can get to all servers with no drops...

hopefully thats the end of it... hopefully its back for others affected too

thought would keep you updated

[Rock]

will you be moving them to milton keynes where its tier 5? so no power outages or anything like that. If so thats good because I'm signing up to dedicated at end of december

Absolutely, at present we are hosting all our new servers in the Milton Keynes facility of Bluesquare which is a Tier IV [4] datacentre.

A Tier IV datacenter is composed of multiple active power & cooling distribution paths, has redundant components, & is fault tolerant, providing 99.995% availability. It's at the most stringent level, which is designed to host mission critical computer systems, with fully redundant subsystems & compartmentalized security zones controlled by biometric access controls methods.

[Rock]

well i just lost my ISP connection, called their support and they advised they were rebooting stuff due to a routing outage... I asked if it way linx related and they advised yes...

just come back online and I can get to all servers with no drops...

hopefully thats the end of it... hopefully its back for others affected too

thought would keep you updated

Hi Lance,

Thank you very much for your updates. Yes, indeed this seems to be the conclusion to the routing issue.. Good luck..

[John]

same here, pages hang.... intermittently

I've optimized databases and apache and done allsorts to limit the impact but to no avail - even upgraded vps to top spec (thanks euk for suggesting that as a solution)

Couple of clients are really doing their nut. Euk not really that bothered. Thanks.

Hi,

It's not a case that we aren't bothered; it's more a case that it is out of our control and the network we use. ISPs pick which route they take to the RS DC. For whatever reason, they decided not to change those routes from LINX to another. There were other routes available.

We only have the option of changing the routes we send traffic over, not which routes we receive it over.

[John]

Hi Rock,

Have stated in the closed post by John that it was external to EUK, hence not alot can be done... its a pain that its only affecting servers at rapidswitch... as all other servers are accessible on a different IP range.... worst thing you guys did was move some servers from BS to RS... first the cisco catastrophe, now routing... not blaming RS here but its only that DC we cant get too properly...

I think in the new year we will need some server migrations out of that place

We didn't move any servers from BS to RS. RS hired a building from BS previously. They still had their separate network, which could have potentially had the same problems as today.