VPS Down again!

[John]

Hi Nick,

I was looking at it before but unfortunately had to move onto some other work.

I looked for the 15th when you last had an issue and I noticed one of your sites receives move visits than other days. However, I didn't have time this morning to confirm that with other dates in the logs.

I also noted you get quite a few brute force attempts on SSH. You should consider changing the SSH port.

In regards to what has changed, potentially many things. cPanel changes often and the resource amounts used can change along with it. There is also the possibility we turned off burstable RAM on that node at some point (I'll look into this).

[nickbits]

Forgot to say, that still doesn't answer the question about the core files that keep appearing!

[John]

Forgot to say, that still doesn't answer the question about the core files that keep appearing!

Hi,

I believe the core files come up when cPanel moves files during an update, but I'll have one of the techs confirm this.

[nickbits]

Hi Nick,

I was looking at it before but unfortunately had to move onto some other work.

I looked for the 15th when you last had an issue and I noticed one of your sites receives move visits than other days. However, I didn't have time this morning to confirm that with other dates in the logs.

I also noted you get quite a few brute force attempts on SSH. You should consider changing the SSH port.

In regards to what has changed, potentially many things. cPanel changes often and the resource amounts used can change along with it. There is also the possibility we turned off burstable RAM on that node at some point (I'll look into this).

Thanks for the feedback. I did attempt to change the SSH port a while back and it completly locked me out. I could not get back in on the new or the old port. I had to go to livechat and get one of your guys to reset it. Since then I have left it. I am happy to try to change it again, but would like some advice as to the best way to do it so I do not lock myself out again. Just point me to a tutorial!

Can you also PM me when you get time and tell me what logs you are looking at, jsut so I am looking at the same ones as you (just to double check).

I do appreciate the help, but it just doesn't seem logical to me as to what is going on. I know one of the sites had a major spam attack a while back and wonder if the issues have arrisen as a result of that.

Regards,
Nick

[nickbits]

Hi,

I believe the core files come up when cPanel moves files during an update, but I'll have one of the techs confirm this.

Thanks. The core files come up randomly in the public_html directories of my sites. Sometime in just 1 site sometimes in 3 or 4. That is why I didn't think it was cPanel initially.

And again is only something that has been appearing since January/February this year, since all these issues started. I ahve never noticed core files before. Hence the assumtion the core files are related to the problems I am having.

If they are to do with cPanel, then I would bet money on cPanel being the cause of my issues.

Thanks again,
Nick

[Rsync]

Thanks. The core files come up randomly in the public_html directories of my sites. Sometime in just 1 site sometimes in 3 or 4. That is why I didn't think it was cPanel initially.

And again is only something that has been appearing since January/February this year, since all these issues started. I ahve never noticed core files before. Hence the assumtion the core files are related to the problems I am having.

If they are to do with cPanel, then I would bet money on cPanel being the cause of my issues.

Thanks again,
Nick

Hello Nick,

Core files are getting created under user "thatcham" and the core files does not belongs to the cPanel.
The core files create because of the application crashed for the user "thatcham" and you need to investigate on the scripts of user "thatcham"

Also, I have checked the logs for your VPS from "Apr 10, 2010" and logs clearly shows that your VM is facing memory issue from "Apr 10, 2010" to current date "June 20, 2011".

You can also check the logs from your end via login into the Parallels Power Panel of your VPS.
You can access Parallels Power Panel as http://your_VPS_IP:4643 use your VPS root login details for login and go to Logs >> Resource Alerts option.

You clearly see the Red Zone for "slmmemorylimit" parameter for your VPS.

Parameter Description slmmemorylimit

An approximation of the size of the physical memory allocated to the Container. In other words, the Container performance is similar to the performance of a physical computer with as much physical memory installed as is indicated in this parameter.

[John]

Hi Nick,

Rsync would have been my first choice to ask on this as he knows a lot about cPanel and VPS. I've seen the core dump files before when I've had alerts setup during cPanel updates, but he has investigated and found it is due to processes crashing/being killed.

In this situation it becomes difficult for us to help you so much. We can see the memory problem and to be honest it wouldn't be overly unusual for a project of your size to hit that memory limit. All you really need is for a few things to combine at the same time for that limit to come.

He has given you a heads up on what he perceives as the problem account and I'd encourage you to look into it.

[nickbits]

Also, I have checked the logs for your VPS from "Apr 10, 2010" and logs clearly shows that your VM is facing memory issue from "Apr 10, 2010" to current date "June 20, 2011".

Firstly thank you for looking into this. I know in 2010 I had issues, but it was rare. As for the "Parallels Power Panel" that is the first I have ever heard of it. Is this something I should have known about?

You have, and are the first to do so, given me a good answer to what the problem is, at least I now know which set of sites is causing the issue and can investigate further. Although as I say I still truly don;t get it, I never had issues like this last year and the stats do not show any major difference in visitors to most of the sites.

Thanks,
Nick

[nickbits]

Hi Nick,

Rsync would have been my first choice to ask on this as he knows a lot about cPanel and VPS. I've seen the core dump files before when I've had alerts setup during cPanel updates, but he has investigated and found it is due to processes crashing/being killed.

In this situation it becomes difficult for us to help you so much. We can see the memory problem and to be honest it wouldn't be overly unusual for a project of your size to hit that memory limit. All you really need is for a few things to combine at the same time for that limit to come.

He has given you a heads up on what he perceives as the problem account and I'd encourage you to look into it.

John,

Thanks for the help, this is the first real support I feel I have been given for this problem, all others it has just felt like I have been fobbed off. I am still puzzled, as I said in my previous post although there were issues last year, nothing like what I am getting this year. I am going to go through everything again and see if I can get to the root of it, and maybe migrate some bits to a new server and spread the load out.

Thanks,
Nick

[Perce2]

Err not quite..!

John R and Cristiano ran some virus scans on the server and attempted to install some security measures. I say "attempted" as in fact all it did was to shutdown one of my sites and lock me out of the controling admin panel of another ! They were asked to revert these measures, I was then able to access my sites as normal and figure out myself what the problem was and how to rectify it.

I traced the problem to a corrupt database, even though I had already (several times) replaced it with a back up after the SQL Attack.

Although they didn't actually help me with the reported problem they did make a thorough job of checking my sever for virusses and malicous files. Nothing was found but it did let me sleep a little easier! Thanks for thier help with this.
G.

Right, I am back again with issues relating to the work carried out with the previous problem earlier in this post. Thought it might be better to keep it together...

Since I last spoke I have found out today by pure luck (whilst installing another site) that for some reason my cron jobs have not been updating the database files on 8 of my sites. In fact all sites that have a database cron attached to them.

Looking at all previous database files they all show last modified dates prior to 13th June, the day before your guy's tried to install some security measures that I had to tell them to remove as they caused the failure of a couple of my sites. Supposedly everything was reverted back to a previous state, but since then they have never worked again !

I have spoken with John R. earlier who had a look for me but reckons it is due to permissions not being set correctly ie. 777 as opposed to 644. He won't accept the fact that they have been working fine, most of them, for more than 6 months prior to eUKHosts intervention on the 13th ! He has changed the permissions on some of my sites but they still don't update from the cron.
I have also found that one of my site's has already got it's permissions set to 777 and it as also not worked since then, so it's nothing to do with permissions as I tried to tell him.

I can't remember the exact conversation at the time (on the 13th), but I can remember John mentioning something to me about access permissions to cPanel and they should be changed, but I told him not to change anything right now beacuse I had programs setup that needed access to resources in cPanel. He say's settings where reverted.

So, this is where I am now;

1. These 8 sites have worked perfectly, updating databases weekly for many months without error. Now I have out of date databases.
2. I have a sql attack on another site that is on the same server but completely seperate account that didn't effect any files other than it's own seperate database. None of the sites in question where effected by this.
3. eUKHost tried to install security measures on the server but stopped access to 2 of my sites and caused lots of funtionality problems so they reverted back the settings.
4. None of my cron database updates have worked since.
5. Nothing has been changed (by me) in any of the files within this time frame, including file permissions.

According to John R., it's my problem and I should sort it myself !!! But I didn't have this type of problem before their intervention on the 13th and I don't know what settings, files they've changed, where do I start ?

I'm biting my tongue right now because all I want are these sites working again....

G.

[Rock]

Right, I am back again with issues relating to the work carried out with the previous problem earlier in this post. Thought it might be better to keep it together...

Since I last spoke I have found out today by pure luck (whilst installing another site) that for some reason my cron jobs have not been updating the database files on 8 of my sites. In fact all sites that have a database cron attached to them.

Looking at all previous database files they all show last modified dates prior to 13th June, the day before your guy's tried to install some security measures that I had to tell them to remove as they caused the failure of a couple of my sites. Supposedly everything was reverted back to a previous state, but since then they have never worked again !

I have spoken with John R. earlier who had a look for me but reckons it is due to permissions not being set correctly ie. 777 as opposed to 644. He won't accept the fact that they have been working fine, most of them, for more than 6 months prior to eUKHosts intervention on the 13th ! He has changed the permissions on some of my sites but they still don't update from the cron.
I have also found that one of my site's has already got it's permissions set to 777 and it as also not worked since then, so it's nothing to do with permissions as I tried to tell him.

I can't remember the exact conversation at the time (on the 13th), but I can remember John mentioning something to me about access permissions to cPanel and they should be changed, but I told him not to change anything right now beacuse I had programs setup that needed access to resources in cPanel. He say's settings where reverted.

So, this is where I am now;

1. These 8 sites have worked perfectly, updating databases weekly for many months without error. Now I have out of date databases.
2. I have a sql attack on another site that is on the same server but completely seperate account that didn't effect any files other than it's own seperate database. None of the sites in question where effected by this.
3. eUKHost tried to install security measures on the server but stopped access to 2 of my sites and caused lots of funtionality problems so they reverted back the settings.
4. None of my cron database updates have worked since.
5. Nothing has been changed (by me) in any of the files within this time frame, including file permissions.

According to John R., it's my problem and I should sort it myself !!! But I didn't have this type of problem before their intervention on the 13th and I don't know what settings, files they've changed, where do I start ?

I'm biting my tongue right now because all I want are these sites working again....

G.

Hi Graham, these issues are being looked into & we'll get back to you once they are sorted out..

[John]

Right, I am back again with issues relating to the work carried out with the previous problem earlier in this post. Thought it might be better to keep it together...

Since I last spoke I have found out today by pure luck (whilst installing another site) that for some reason my cron jobs have not been updating the database files on 8 of my sites. In fact all sites that have a database cron attached to them.

Looking at all previous database files they all show last modified dates prior to 13th June, the day before your guy's tried to install some security measures that I had to tell them to remove as they caused the failure of a couple of my sites. Supposedly everything was reverted back to a previous state, but since then they have never worked again !

I have spoken with John R. earlier who had a look for me but reckons it is due to permissions not being set correctly ie. 777 as opposed to 644. He won't accept the fact that they have been working fine, most of them, for more than 6 months prior to eUKHosts intervention on the 13th ! He has changed the permissions on some of my sites but they still don't update from the cron.
I have also found that one of my site's has already got it's permissions set to 777 and it as also not worked since then, so it's nothing to do with permissions as I tried to tell him.

I can't remember the exact conversation at the time (on the 13th), but I can remember John mentioning something to me about access permissions to cPanel and they should be changed, but I told him not to change anything right now beacuse I had programs setup that needed access to resources in cPanel. He say's settings where reverted.

So, this is where I am now;

1. These 8 sites have worked perfectly, updating databases weekly for many months without error. Now I have out of date databases.
2. I have a sql attack on another site that is on the same server but completely seperate account that didn't effect any files other than it's own seperate database. None of the sites in question where effected by this.
3. eUKHost tried to install security measures on the server but stopped access to 2 of my sites and caused lots of funtionality problems so they reverted back the settings.
4. None of my cron database updates have worked since.
5. Nothing has been changed (by me) in any of the files within this time frame, including file permissions.

According to John R., it's my problem and I should sort it myself !!! But I didn't have this type of problem before their intervention on the 13th and I don't know what settings, files they've changed, where do I start ?

I'm biting my tongue right now because all I want are these sites working again....

G.

Hi,

I've just read your chat with John R. Might I suggest that you spend a lot less time accusing someone who is trying to help you and more time helping resolve the actual issue.

We will help you solve any issues you have with your server, but only if you answer our questions during the course of our investigation and not divert the conversation down a path which isn't going to help anyone.

[Perce2]

Might I suggest you re-look at the chat and get your facts right before talking in a defensive condesending manner !

I fail to see what help anyone can give when the one who is said to be helping won't listen to anything or answer questions given to them relating to the problem in hand. If you look at the chat you will see where I have had to repeat myself many times but have failed to get the message across or get an answer.
Dealing with your support staff is always a problem in it's self, you only have to look through your forum to see things are exactly right when it comes to user issues.

I'm not sure what you are insinuating with your last derogatory comment but advise you take a different approach when communicating with your customer.

What questions do you need answering ?

G.

[John]

Hello,

You don't have to look too far back at my posts to see I will pretty much slam our technicians whenever I think they make a mistake. I've done it several times today over the phone and in tickets. I don't think I've even been too nice about them in this actual thread. However, that said, I will defend them when I think they are being unfairly treated.

John R currently has a stress fracture on his arm and he is in pain constantly, even more so when he types. I'm not telling you that for any form of sympathy, I'm merely trying to get across his character. He could sit at home on full sick pay but instead opts to come to work. Does that mean he can do no wrong? Not at all. Do you honestly think he wants to waste your time on chat, ask pointless questions or try to lie about what happened 10 days ago?

Now, during your chat you actually spent more time assigning blame for something which happened 10 days ago than you did trying to resolve the actual problem. The first half of it was John R trying to explain the crons were running and you saying they weren't. I believe you conceded near the end that the crons were indeed running, but they weren't having the desired effect. I ask you honestly, how long did you spend arguing regarding the crons weren't running?

After that you then disputed the fact that changing the permissions corrected the problem and spent further time refusing to accept this as the overall cause. These are your crons. These are your sites. These are your connections. We didn't make or set these up for you, so we can only tell you what we see.

I'm not going to come on these forums, criticize a technician and try to gain your favour. I could quite easily have done that, but it wouldn't be an accurate reflection of the reality of a situation and you also wouldn't be any further on in solving your problem.

I don't mind saying publicly for all to see, I expect customers to cooperate with our technicians when they are trying to be helped. If you feel they made mistakes, crossed a line or were generally unhelpful, you may raise it through the appropriate channels at a later stage. If this is something you don't feel you can abide by, let me know and I will refund the past 6 months of your payments and allow you 14 days to move to a provider you will be happier with. I have no interest in keeping a customer who is obviously unhappy, especially when I can see nothing from our side we could do to change their opinion.

As Rock stated, your problem is still being looked into and a ticket will be updated with the results.

[Perce2]

I really haven't the time for this, it's 3.00 am in my part of the world !
You miss so many parts of the conversation you are trying to quote and judge, some of it you are out of context as it becomes ambiguous at this stage.

At one point when John R. was tring to help HE managed to get the crons to run from whatever HE did. I also had one cron run once, then after he'd gone they stopped working again....I was running them from where I was having the problem, in cPanel. He was running them (I presume) from command line.
So hardly the way to complete judment that things where working as they should.

Yes, I know they are my cron's, my sites and my connections.
But the same as John R. neither of you actually make reference to or answer my initial question, which I kept coming back to during our chat. That is - if as you are implying, the problem lies in my cron instructions (commands), file permisssions, or whatever else you would like to blame, then why have they worked perferctly well for more than 6 months up until they touched the server with their script that took out a couple of my sites ?
Why do the files with John's correct file permissions also fail to work now ?
File permissions don't just suddenly change do they ? Command lines don't just suddenly become non-operative do they ?

Passing blame isn't helping me right now, co-operation is what I need. As I stated earlier, all I want is these sites working correctly.

I will consider your offer and let you know in due course, meanwhile hours have passed and I still haven't heard anything or seen any progress with the sites.

G.

[John]

Hello,

I believe you sought our help when you noticed an attack against your site. This was because you correctly identified an issue after you were hitting resource limits. That type of complaint I can very much understand.

Truth is, right now, I have no idea if anything which our techs did on that date stops your databases updating via the cron. I also have no idea if anything you did that day (you said yourself you resolved it yourself) was responsible. I'm not in the business of making assumptions. I've seen many a sites break with little to no human interaction over my time, which is why I don't jump to conclusions.

A technician named Patrick is looking into your issue. You can reach him on LiveChat for real time updates.

[John]

Just to add, he has just asked me to ask you to come onto LiveChat

[John]

Hello,

Patrick is now reporting this issue to be fixed. Please confirm.

The issue was with the wget binary not having permission to be run from users other than root. Patrick changed it to allow it to be run by other users.

I can't think off the top of my head which security feature would do that as part of it's settings. Perhaps John R, Cristiano or Patrick can have a think about it and let us all know.

[Perce2]

Yes, the problem now appears to be fixed and the crons run again as they did prior to the help I received on 13th June!

So, like I said it has nothing to do with the permissions of my files, folders etc. does it ?
Nothing to do with my crons, commands or sites.
In fact, not anything that I have touched or been near. But as I have found, without too much effort, it's quite normal in security fixing issues to amend the permisssions of that file.

Here is a mail taken from another host's site with exactly the same problem; (names removed)


Re: Cron Job not working - /usr/bin/wget: Permission Denied
by Xxx X on Fri Jul 02, 2010 8:53 pm

Hi Xxx,

There were some security updates done last night that might be causing this problem.


Looks like it's a common fix, thought you would have known that John ?
So who's to blame ?
We know the guy's changed the security settings, but they said they'd reverted them back, didn't they ? (check the chat John)
Coincidence ?
Really........!

Here's another coincidence, I haven't had one sale on my sites also since that date, maybe something to do with the out of date data maybe.

G.

[John]

Hi,

Ah, are we still talking about blame? To be honest, It isn't an overly obvious one and I wouldn't think to "undo" it. Yes, though, it is quite likely it was done when our technicians spent hours trying to help you with your attack problem.

I'm going to add a note to your account ensuring that our technicians only help you with control panel and OS updates in future. I can't help thinking that would be best for everyone concerned.