Cloud Performance Observations

[ejsolutions]

Some things that I've noticed may be of interest.

Cloud 1: vmware - euKHost
Since the recent outage of Cluster #3, I've at long last noticed a return to the original great levels of performance. Following the major disruption a couple of months back, with storage array(s), the performance dropped considerably and I mentioned this at the time. A few tweaks were done by support but it was never the same as before. I can happily report that things are back to normal.
I/O wait is now averaging about 5% again, with a resultant drop in CPU usage and overall load.

Cloud 2: zenenterprisepv - customer resizing (like eNlight) competitor supplier
Small Cloud with light load - displays weird rise and fall in applications memory usage. Apart from lousy slow response to support issues, generally it suits the purpose. It is a responsive system.
I/O wait averages about 5%, CPU low, Load low.

Cloud 3: zenpv - eNlight - eukHost
Small Cloud (same resources as Cloud 2) with a few domains migrated from Cloud 2. Lightly loaded in terms of traffic. Takes what an appears to be an age (3 to 8 secs. approx.) to "wake up" when a site request is made, then runs OK.
I/O wait averages about 10% and never drops below approx. 8%, CPU is relatively high and Load climbs at little provocation (just like Cloud 1 was).

Cloud 4: virtuozzo - USA provider
Big traffic, so unfair to make any comparisons, other than I prefer vmware and its logical use of vm/swap space.

Has eNlight been crippled by a slower storage medium compared to vmware Cloud, is the obvious question? Wouldn't be surprising but the knock on effect of possibly increasing the monthly running cost, due to Load is of concern, at this stage.

EJ

[DPS Computing]

Thank you for your detailed feedback . I am sure that this will be as useful to other customers as it is for me .

I'm sure someone from eUK will be along shortly to answer your questions .

[John]

Some things that I've noticed may be of interest.

Cloud 1: vmware - euKHost
Since the recent outage of Cluster #3, I've at long last noticed a return to the original great levels of performance. Following the major disruption a couple of months back, with storage array(s), the performance dropped considerably and I mentioned this at the time. A few tweaks were done by support but it was never the same as before. I can happily report that things are back to normal.
I/O wait is now averaging about 5% again, with a resultant drop in CPU usage and overall load.

Cloud 2: zenenterprisepv - customer resizing (like eNlight) competitor supplier
Small Cloud with light load - displays weird rise and fall in applications memory usage. Apart from lousy slow response to support issues, generally it suits the purpose. It is a responsive system.
I/O wait averages about 5%, CPU low, Load low.

Cloud 3: zenpv - eNlight - eukHost
Small Cloud (same resources as Cloud 2) with a few domains migrated from Cloud 2. Lightly loaded in terms of traffic. Takes what an appears to be an age (3 to 8 secs. approx.) to "wake up" when a site request is made, then runs OK.
I/O wait averages about 10% and never drops below approx. 8%, CPU is relatively high and Load climbs at little provocation (just like Cloud 1 was).

Cloud 4: virtuozzo - USA provider
Big traffic, so unfair to make any comparisons, other than I prefer vmware and its logical use of vm/swap space.

Has eNlight been crippled by a slower storage medium compared to vmware Cloud, is the obvious question? Wouldn't be surprising but the knock on effect of possibly increasing the monthly running cost, due to Load is of concern, at this stage.

EJ

Hello,

After the slow down last week, we moved quite a few VMs to a new storage platform to restore full service while faulty components were replaced. The problem last week was caused by faulty cables going to a controller, which meant the other controller was taking all the load. After we moved VMs to get the load "normal", the faulty cables were identified and replaced as well. This brought the other controller back online, which meant the VMs still hosted on it had the use of two controllers and had considerably less VMs on. Your VM wasn't moved, so it is benefiting from a considerably less busy storage device.

In regards to eNlight, it is on our latest storage platform. Looking at the stats, it isn't remotely close to hitting any limit. On your VMware Cloud you have 1024 MB RAM. On your eNlight it you have limited it to 512 MB, and it is also set in economy mode. These things combined and perhaps differences in technologies would likely explain your performance difference. On the eNlight platform, as with our latest VMware and HyperV clusters, faster switches and more connections are used compared to our older Clusters.

FYI, the storage system your VMware VM is on will soon be decomissioned, as the hardware is going end of life. You will be migrated (no downtime) to a new system over the weekend or early next week. After which you can compare performance again. I do like these types of threads

[ejsolutions]

Fantastic reply, John.

Coming from both a hardware and software background, I can appreciate the difficulty in diagnosing, let alone isolating, a faulty cable! I hope someone got at least a 'pat on the back'.

I have realised that I can't compare "my" two eUKhost Clouds as like-for-like, due to a number of differences, hence the inclusion of 'Cloud #2'. For me, it's very rare to be able to compare four different flavours of Cloud (as a consumer). The zenenterprisepv VM, as mentioned, is nigh on directly comparable to eNlight, running within 512Mb on the same release of Centos/cPanel. Its resources are capped, with no burst, and so I assume is similar to running eNlight in 'economy mode'. The back-end control panel is different, so there's likely to be hidden differences though.

It leads me to wonder if there is a fundamental difference between zenenterprisepv and zenpv, or whether there is much more to this than meets the eye. I doubt that the zenenterprisepv VM has nearly as good (poor word) an infrastructure to run from as eNlight: it is after all a cheap unmanaged service. This is what makes the 5% difference in I/O wait surprising - maybe I'm just fortunate to be on a quiet cluster on 'Cloud #2'.

If further interested/appropriate, I can attach a couple of Munin charts to illustrate the discussion.

EJ

[ejsolutions]

FYI, the storage system your VMware VM is on will soon be decomissioned, as the hardware is going end of life. You will be migrated (no downtime) to a new system over the weekend or early next week.

Thanks for the 'heads up' - much appreciated. One hopes that the cPanel backup window doesn't produce Loads over 8 again.

[John]

Hello,

Your performance will only get better, of that I'm sure

In regards to eNlight, you might want to reboot it into Performance Mode if you are limiting it to 512 MB RAM anyway. That way there is no cache manipulation, which could impact I/O in certain situations. Also might be worth comparing OS versions (CentOS Version) and if one is 64 bit or 32 bit.

[eUKhost.com]

Hi Jon

I would like to get some of our R&D team members involved to see if we can make some changes on our side to give similar performance in economy mode and high performance mode.

We can make some changes in the specification of your Cloud VM and see if you experience improved performance.

[John]

Hello,

Rishi is following the thread Changing it would defeat the purpose, but in this case as scaling isn't being used anyway, it makes no sense to have it in Economy Mode. Economy Mode keeps scaling of RAM under control with a small sacrifice in performance in certain situations, where as Performance mode lets the OS do as it pleases, giving an obvious boost. In the current setup there is no benefit and only downsides to keeping a limit of 512 MB RAM and then also setting it in economy.

[ejsolutions]

As suggested, I've switched to "performance mode".

Below are the differences between the two Cloud VMs, noting the allocation of buffers to be substantially different. Interesting artefact in the way memory is report, though note the actual allocation is the same. eNlight should have the edge in raw performance due to better spec. CPU but the loading on both VMs is so low as to be of negligible significance.

[Domain names obscured but you guys will likely guess/know ]

Total processors: 1
    Intel(R) Xeon(R) CPU X3440 @ 2.53GHz
    2533.330 MHz
    Cache 8192 KB

Memory: 501248k/524288k available (2530k kernel code, 22404k reserved, 1736k data, 196k init)
Linux secure-1.cloudxxxx.com 2.6.18-238.19.1.el5xen #1 SMP Fri Jul 15 08:16:59 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

Current Memory Usage
             total       used       free     shared    buffers     cached
Mem:        524288     521180       3108          0       2496     108108
-/+ buffers/cache:     410576     113712
Swap:       522104      75928     446176
Total:     1046392     597108     449284

Total processors: 1
   Intel(R) Xeon(R) CPU X5660 @ 2.80GHz
   2792.998 MHz
   Cache 12288 KB

Memory: 187896k/16777216k available (2535k kernel code, 335844k reserved, 1748k data, 196k init)
Linux secure-1.xxxxcloud.net 2.6.18-274.12.1.el5xen #1 SMP Tue Nov 29 14:18:21 EST 2011 x86_64 x86_64 x86_64 GNU/Linux

Current Memory Usage
             total       used       free     shared    buffers     cached
Mem:        524288     468972      55316          0        240      18336
-/+ buffers/cache:     450396      73892
Swap:      2129912     277912    1852000
Total:     2654200     746884    1907316

(In my experience, in non-vm environments, a swap of more than twice RAM can be detrimental. In modern times, I usually restrict to match RAM.)

[ejsolutions]

A couple of pics added, 'cos only words can get boring.

[ejsolutions]

For other (potential) customers of eUKhost, this is the level of interest shown by the staff, that sets them apart from the competition. Yes, I have my "moments" with them but overall you're going to be hard pressed to find a better hosting provider. As for USA-based ones: they don't even come close!

[eUKhost.com]

For other (potential) customers of eUKhost, this is the level of interest shown by the staff, that sets them apart from the competition. Yes, I have my "moments" with them but overall you're going to be hard pressed to find a better hosting provider. As for USA-based ones: they don't even come close!

Thanks Jon

Our eNlight team is reading this thread. I am waiting for them to respond here and make the thread more interesting.

[ejsolutions]

A bit slow off the mark with this one..

Rishi is following the thread ...

Is this the Guru that lives under the suspended floor of the machine room, wearing an antistatic wrist strap and tin foil hat?


Meanwhile, back to comparing PHP code. :'(

[John]

A bit slow off the mark with this one..


Is this the Guru that lives under the suspended floor of the machine room, wearing an antistatic wrist strap and tin foil hat?

He is one of the main developers of eNlight, so is best placed to advise on performance differences

[eUK-Rishi]

Hi ejsolutions,

Thanks for starting a wonderful thread and we expect more to come.

Theoretically there are multiple factors on performance scale, in sorted order
* SAN vs Local Disk
* Kernel
* CPU family
* Size of RAM and cache pressure
* LVM vs non LVM
* FileSystem ext3 vs ext4
* OS type 64 vs 32

eNlight Cloud runs same storage platform as vmware so the difference is on the Hypervisors and VM OS.

From your reply the factors which I see is

* Both have 64bit OS
* Kernel 2.6.18
* But a big Buffer and Cache difference, buffer and cache are yet to fill in enlight VM which will reduce your iowait.

You can do

sync ; echo 3 > /proc/sys/vm/drop_caches

on both the server and wait for 1 day to see the graphs.
Above command will drop cache and buffer to disk and will put both VMs on standard disk load.

I would like to have results of this. If you can PM me your enlight server details, I can take a deeper look into it. Also it depends on the app of both server

[ejsolutions]

I'm not willing to drop the cache on the vmware Cloud: it's too busy with paying clients/customers to jeopardise anything. I'm on thin ice with a big client just now due to sporadic "white page" issues. Grr.

Have dropped the caches on both xen VMs (they are the ones that are closest in spec.) and below are the subsequent figures, immediately afterwards - eNlight being the second one:

             total       used       free     shared    buffers     cached
Mem:        524288     502076      22212          0        860      74452
-/+ buffers/cache:     426764      97524
Swap:       522104      77588     444516
Total:     1046392     579664     466728

--------------------//-----------------------
             total       used       free     shared    buffers     cached
Mem:        524288     468668      55620          0        316      21908
-/+ buffers/cache:     446444      77844
Swap:      2129912     307208    1822704
Total:     2654200     775876    1878324

Now we have a point of reference.

Now for disc mappings, which is rather interesting to note the differences:

/dev/xvda3 on / type ext3 (rw,noatime,usrquota)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/xvda1 on /boot type ext2 (rw,noatime)
tmpfs on /dev/shm type tmpfs (rw,noexec,nosuid)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/usr/tmpDSK on /tmp type ext3 (rw,noexec,nosuid,loop=/dev/loop0)
/tmp on /var/tmp type none (rw,noexec,nosuid,bind)

----------------------//------------------------
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw,usrquota)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/xvda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw,noexec,nosuid)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/usr/tmpDSK on /tmp type ext3 (rw,noexec,nosuid,loop=/dev/loop0)
/tmp on /var/tmp type none (rw,noexec,nosuid,bind)

I haven't been told what the underlying disc structure is on 'Cloud #2' but it is apparent that eNlight uses LVM of some sort. Not logging access time should theoretically speed disc writes, if nothing else but has other implications. Having /boot as ext3 is a waste of time, IMO, though once the server is underway, should have little bearing.

[Will PM the eNlight login details (and 'Cloud #2' if wished) - both VMs are used for live sites, albeit not "crucial". By their nature they serve up different traffic at present and one is 'forced' to use FCGI (yuk!) until its software is migrated/updated.]

The saga continues...

EJ

[eUK-Rishi]

Hello,

Swap: 522104 77588 444516
Total: 1046392 579664 466728

--------------------------------------- vs -----------------------------------------

Swap: 2129912 307208 1822704
Total: 2654200 775876 1878324

As you can see the overall load pressure is more on eNlight for total used RAM size ( 579M vs 775M )
You can notice the swap difference as well ( 77M vs 307M) which again proves your RAM pressure.

total used free shared buffers cached
Mem: 524288 502076 22212 0 860 74452

--------------------------------------- vs -----------------------------------------

total used free shared buffers cached
Mem: 524288 468668 55620 0 316 21908

- Low cache ( 74M vs 21M) suggests that you have more dynamic content on eNlight which is not useful for cache.

Usage of swap indicates that system does not have enough RAM for required data processing and swapping a lot and hence generating %iowait.
It could be that OS is swapping more because of available swap space ( eNlight need chanegs ) but figs are still cloudy as swap pressure is 77M/512M vs 307M/2048M which is 15% in both cases.

Lets wait for 24 hours to have the Munin graphs and then if you agree we will make changes to MAX RAM (768M) and MAX CPU (2) and will observe for next 24 hours.
I think I should not mention but you wont face any downtime in this auto upgrades because of eNlight's design

Thanks for the cooperation and its really helpful for everyone !!!

[ejsolutions]

Yes, dynamic content is the name of the game; I'm a specialist ( & part of the Dev. Team) for an osCommerce-derived package. However the 'low order' Cloud is for running informational content, a demo and a relatively small Joomla site. The bread & butter UK Cloud is the vmware one.

The intention is for eNlight to replace 'Cloud #2', which costs a flat rate of £15/month (at current resource levels), including cPanel. This is for an "unmanaged service", however. At this moment of time the 'pressure' on both servers is split, due to this testing phase. Previously, 'Cloud #2' coped perfectly fine serving up what is now on eNlight, in addition to its current Joomla site.

Hopefully from this, you can see that raising the resources to what you assess as being appropriate is just not going to happen.

Unlike in Windoze, swapping is a natural occurrence for a *nix environment and is actually healthy. Were the OS not to associate some RAM to disc buffers, slab etc., then there would be more free RAM for applications, reducing the need to swap. That is not how a *nix OS operates under normal circumstances.
This can be readily seen on the MUCH busier vmware Cloud that has 1Gb RAM - very little above 512Mb RAM is utilised, even though mySQL has been allocated large cache etc. The disc buffers are proportionally higher, along with other memory management allocations.
The dual processor, 2Gb RAM USA-based Cloud, which has a 57k product ecommerce site and mediocre levels of traffic (apart from search bots), barely touches the upper 1Gb and I had considered reducing it back to 1Gb but decided to retain it for 'bursty moments'.

EJ

[ejsolutions]

To respond directly to your swap comments:
I actually think a reduction to the allocated swap space is a worthwhile 'experiment', even as far as 512Mb, matching currently allocated RAM and that of 'Cloud #2'. If that appears to be too extreme, then I could live with 1Gb/768Mb.

[ejsolutions]

Over 24 hours since the cleared cache operations, so here's the latest "Scores on the boards": eNlight being shown as the second set again.

             total       used       free     shared    buffers     cached
Mem:        524288     511728      12560          0       4960     107960
-/+ buffers/cache:     398808     125480
Swap:       522104     155632     366472
Total:     1046392     667360     379032
-------------------//-------------------------
             total       used       free     shared    buffers     cached
Mem:        524288     469620      54668          0        356      20712
-/+ buffers/cache:     448552      75736
Swap:      2129912     356624    1773288
Total:     2654200     826244    1827956

To illustrate the effect of this, a small Softaculous update made eNlight "throw a wobbly", with a Load exceeding 9 (Munin gave up on it).

Any scope to reduce this excessive swap space? I'm sure a reduction will cause the OS algorithms to better manage memory usage (buffers in particular).

/dev/VolGroup00/LogVol01 swap                    swap    defaults        0 0

A remount on LogVol00 is likely to be counter-productive
Regards,
EJ