All My Domains Hacked!

[daledavies]

I noticed yesterday that some strange things were happening with the sites on my domains, so I logged in via FTP to find the index files had disappeared from all the sites I have set up.

I submitted a support ticket, finally went to bed at 2am with still no resolution, however 12 hours later I get an incomprehensible response from Stephen S...

In performing mandatory maintenance on the server, there are few domains who's data got quarantine. We will like to offer our apologies, but server maintenance was also required.

Do revert if you face any issues or doubts with the same, we will be glad to assist you.

So anyway, wake up this morning, see the email, check my sites and what do I see?

~::HaCkeD By Chief::~

WTF? Looks like the same set of files uploaded by the hackers in every directory. Including those outside the www root.

Looking at the modified time of these files I can see they were uploaded about 1 hour after Stephen S had supposedly resolved the issue.

So, someone in support has re-opened my ticked and promised me I'd hear something in 30 minutes, keep in mind all my sites are now displaying "Hacked By Chief" messages, but I've heard nothing so far.

Sorry guys but where is the urgency here? I've just reported the possibility of this shared server being compromised, I only had static files in those domains so SQL injection is not a possibility, are you not even concerned?

[daledavies]

OK, so still no response, not even an email to confirm my support ticked has been re-opened.

Some things that would make me feel more comfortable in this situation are...

• When I initially contact you regarding an issue like this I would like to get the impression are at least concerned!
• I would appreciate being given a run through of the steps you will take to resolve this issue and an ETA for having my sites back to normal.
• I would like my sites back online before you investigate how the breach happened.
• Once you have had time to investigate I would like a full report on the extent of the breach and what services have been affected. E.g. At the moment I'm quite concerned about the fact that my email accounts may also have been compromised.

At the moment I feel as if I have been left high and dry.

I'd like to reiterate that this is a reseller account, if I was actually reselling the space my business would be at risk and I would be getting seriously embarrassed right now.

[daledavies]

Looks like the same problem over at Web Hosting UK...

Server hacked for more than 24 hours

[eUK - Chris]

HI Dale,

I am working on your issue, I am in the process of restoring the website.
The site Dale Davies, The Online Home of... is running fine, Please allow me some time more to restore the pages of other websites.
I will update your ticket "IGA-144-27641" shortly

[daledavies]

Chris, daledavies.co.uk is not running fine, it is currently showing a "Server Application Unavailable" message.

When I FTP to the domain I can see that the FTP root is full of files added by the hackers, plus the web root and all other folders under it still contain those files too.

[daledavies]

Just received your email Chris...

Hi Dale,

I was able to restore the Data for some of your websites, but for the rest of the website, the Data has been overwritten with hacked files.

I would request you to upload the index pages for those websites .


Thanks for your patience.

Do get back to us, in case of any doubts or for further assistance,
We will be glad to help you.

So far Chris it seems you have done absolutely nothing to resolve my issue. In fact some might say it is now worse, the websites at all of my domains now time out.

First of all, are you telling me you do not have a backup of my accounts? I was told over the phone that you would restore my account from a backup.

Second, what exactly have you done to prevent this from happening again?

I fail to see how you can pass this off as acceptable customer service, seems like I've spent hours waiting for you to do pretty much nothing.

[eUK - Chris]

Hi Dale,

We have rebooted the server after applying security fixes.

Hence I cannot access the website at this moment, I will check this once the server is back online.

Mean while, I would request you to update the ticket.

[daledavies]

Updated with the same questions as above.

[daledavies]

Hello is anyone at eUKhost actually doing anything about this? I've not heard anything and it appears you have not actually done anything either.

Actually forget about it, I will not be paying this moths invoice and I will not be using your services again. Please just cancel my account, I don't see how I can continue to use eUKhost as I have absolutely no trust in your services.

I have had nothing but trouble over the last 4 years or so that I've been hosting with you, to be honest I don't know why I didn't move to another web host sooner. This most recent cock up has given me the perfect excuse to do it.

[Richie]

It has taken us 2-3 days to get a migration then when they did migrate it they only copied the files across and not the databases.

I have spoken to my solicitor and my solicitoor has suggested a file for compensation. Apparently in this case as the hosts have claimed responsibility, we have a chat log transcript with them stating clearly it is a problem with their server security not our site, we can issue a claim for compensation for each user put at risk on our website. we have over 3000 users so I can see that being costly. £1 per user compensation is £3000.

I have a meeeting with my solicitor on Tuesday I will let you know how it goes

[daledavies]

To be honest I couldn't be bothered with the hassle, although if you are running a business I can see why you might want to take that step.

At this point I would just be happy with a response, so far there has been too much silence and very little activity from what I can see.

[Richie]

Totally agree with you.

If you read the improve your security thread http://www.eukhost.com/forums/f14/im...917/#post83882 - link to thread

It outlines all the responses I have had with them through live chat.

[daledavies]

Well, I have just received another unacceptable response again from support...

We really apologize for the issues you and our clients are facing.
We always take the backup of all our websites data but unfortunately in your case the backup data has been overwritten and all the infected files were saved in the backup repository. We have removed those infected files.
And also we have removed around 95 % infections from the server and still we are working on it and in next couple of hours other 5% infection will be removed.

We have found some accounts on the server which were responsible for this issue. we have complete terminated those accounts from the server and also installed new security and monitoring service. And also we have enabled security logging on the Web server.

Now we assure that you and our other clients will not face such issues again.

Well, I was always taught that if you cannot recover fully from your backups then you might as well not have any backups to begin with.

Do not try to tell me that you only have one backup, if this is the case then you are incompetent.

I am insulted that after the communication we have had over the last few days you would still try to fob me off with a response like this.

You have not removed the files added during this hack and you have also lost most of the original files that were stored on my sites. To make things worse you seem unable to resolve the issue completely in a timely manner.

I simply cannot continue to host with you as I have no trust in the quality of your services, your security practices, or the competence of your support.

Please cancel my account immediately.

[Richie]

yeah we go t this reply after the 2nd or 3rd time the site was hacked.

Then after the 4th or 5th time the agent said the security had only been updated that day. a full 2 weeks after the initial attack.

[daledavies]

Finding it hard to believe it could take 2 weeks, with customer accounts being compromised across the server on multiple occasions and this issue is still not resolved.

I was Googling the text found on my site and the results showed 6 or 7 sites with the same defacement, these sites were cached by Google on the 24th. All these sites use ns1.ukdnp.com and ns2.ukdnp.com (ukdnp.com is reistered to eUKhost).

My main fear was that Google would index my site during the defacement and slap a Hacked Site Warning on it. Thankfully this didn't happen because I was able to migrate my domains to another hosting provider as soon as I realised the issue was not going to be resolved quickly. However looks like some of eUKhost's customers have not been so lucky.

[Richie]

christ I never thought of that. there will be hell to pay if its happened on my main site. How do you check ? just google the site ?

[Richie]

just checked it and the hacker page is cached on my site. how do you get that changed ? how often do google cache pages and can we request a google cache ?

[daledavies]

Well Google usually decides how frequently it visits your site itself, based on how often you update the content (and probably several other things). You can specify certain things using Webmaster Tools, but I'm not sure how effective this is.

Best advice I can give is to get your site back to normal as soon as posible so when Google comes back it re-indexes your normal content without any defacement.

If your site has been slapped with a Hacked Site Warning then read the following Google Webmaster Central help FAQ...

https://sites.google.com/site/webmas...d-hacked-sites

[daledavies]

Can somebody please respond?

John the MD, surely you are able to provide a proper legible response?

Ive given up on actually getting this issue resolved, I would just like to cancel my account but Im just being ignored.

[Rock]

Hi Dale & other customers,

John won't be able to provide you with any legible response as he isn't a technical person & knows little about this incident. I'll answer all your queries now.

Please accept my deepest apologies for not keeping you all updated on this hacking dilemma & the delay on responding to your queries. I gave more priority on securing the servers & removing any traces of back-doors/worms from it to get your services working to normal, than posting here. I may be wrong this time, but will ensure that this doesn't happen anymore in the future. This exceptional server/website hack incident is not a regular incident; its a completely unexpected & complicated co-incidence where multiple reasons cause us to fail in what we planned or designed. We never intended to & will never in the future too, want to cause grievances & trouble to our customers.

This entire DNP cluster2 comprises of 8 servers in total of which only one was affected/hacked. Webserver1 hosts more than 2 thousand websites, of which only a max 100 were affected. All these websites & their permissions have been corrected now & the hacked pages removed. This attack was done on only those particular websites which had weak permissions granted on them. Hence we had to scan the entire web server to check if any other are found to be affected. Scanning 2000 websites with an anti-virus, each & every file & leaving all the services online in the meantime, took more than a day. After the antivirus scan completed, we ran an anti root-kit scan, which took another 6 to 8 hours to complete. Few unknown services & the root-kit were detected, which needed several reboots for their removal. Later we noticed that the server failed to boot as the hacker had deleted the NTLDR & the NTDETECT.COM files from C:\ which are necessary for the server's booting process. We tried to copy those files from a fresh installation DVD, which failed. We had to boot the server from a Linux Live CD & hence these all resulted into further delays.

We don't deny of the fact that this Webserver1 on our DNP Cluster2 was hacked, as a root-kit was found installed on it, which caused the hacker to upload various scripts to duplicate content to several live websites, which displayed the hacked page on certain websites which had weak permissions.

Here's a brief description on what exactly a root-kit is: A rootkit is software that enables continued privileged access to a computer while actively hiding its presence from administrators by subverting standard operating system functionality or other applications. The term rootkit is a concatenation of "root" (the traditional name of the privileged account on Unix operating systems) & the word "kit" (which refers to the software components that implement the tool). The term "rootkit" has negative connotations through its association with malware.

Typically, an attacker installs a rootkit on a computer after first obtaining root-level access, either by exploiting a known vulnerability or by obtaining a password (either by cracking the encryption, or through social engineering). Once a rootkit is installed, it allows an attacker to mask the ongoing intrusion & maintain privileged access to the computer by circumventing normal authentication & authorization mechanisms. Although rootkits can serve a variety of ends, they have gained notoriety primarily as malware, hiding applications that appropriate computing resources or steal passwords without the knowledge of administrators & users of affected systems. Rootkits can target firmware, a hypervisor, the kernel, or—most commonly—user-mode applications.

Rootkit detection is difficult because a rootkit may be able to subvert the software that is intended to find it. Detection methods include using an alternate, trusted operating system; behavioral-based methods; signature scanning; difference scanning; & memory dump analysis. Removal can be complicated or practically impossible, especially in cases where the rootkit resides in the kernel; reinstallation of the operating system may be the only alternative.

We've had setup several security restrictions initially on all of our Windows shared servers, but new vulnerabilities are discovered everyday now & then. We weren't able to (or rather any anti-virus or anti-root-kit wasn't able to) detect the exact problem behind the hack in our daily server security audits, as it's quite difficult to find a root-kit with a naked eye unless you're a security expert with loads of experience on your back. We toiled & finally found out what exactly was behind this root-kit installation. Alarms went off when we noticed an IUSR added to the administrators group which resulted into the hacker gaining administrator access to the entire server, either from the websites itself. Several shell scripts (c99, r57 for PHP & other ASP/ASP.Net ones) were uploaded under one of the domains, the real-time scanning software detected instantly & to which we deleted them at that very moment. But to our surprise, same scripts were uploaded to several other websites that very day, which were apart from the pages which displayed the hacked message.

Today, we've tightened the security to the maximum level, to which few legitimate websites may malfunction as several vulnerable but necessary functions have been disabled globally & permanently.

It wasn't the fact the we did not have a plan, but sometimes when too many things go down at the same time, there is only so much we can do. But, what we can do after that is to work for the better with hindsight & that's exactly what we have embarked upon. We know & understand for a fact that you as a customer depend & rely on us to be able to provide uninterrupted service. We strive hard to ensure that you get the best of services & quality. However, on occasions such as this, a few things go wrong together which makes the situation impossible.

As for improving communications with the customers, the forums would be monitored with twice the vigor & efforts by me & other members. We understand the fact that its a window of connecting to our customers. Again, not that it was not being done now; we will only be pushing it more as to answering each & every query/post/thread. We have learnt from this episode of how many things can go wrong together & so would have having multiple security systems in place to avoid such situations and/or counter them.

We being a service provider take complete responsibility of this incident & ensure that such incidents won't recur again. I'd request you all to PM me with the Ticket IDs or websites which still aren't working & I'll get that sorted ASAP.

Once again I apologise for any inconvenience this may of have caused & hope I have been able to explain things to a level of satisfaction where you do repose your faith in us. We would be very glad to be of further help in case you need it. Do let us know if we can help more.