We are in the process of migrating this forum. A new space will be available soon. We are sorry for the inconvenience.

RPS First Impressions


ned14
09-03-2009, 10:54 AM

oles@ovh.net;487 Wrote:
> ned14 a écrit:
> >
> > I successfully moved my RPS to use its USB stick as its primary root
> > filing system rather than the iSCSI SAN. Here's how:

>
> it's not a good idea. if you don't have any swap, if you reach 0 RAM
> linux will kill the process and you will lost the control of your RPS.
> in the first version of the RPS (2 years ago) we didn't propose any
> USB swap. then we added it since the RPS was instable without any
> swap.
> Since, no problem.


What I would suggest there is that you partition 512Mb for swap and use
the remaining 1.5Gb as the read-only BTRFS layer in order to greatly
improve system responsiveness.

I have been testing and evaluating the RPS for a while now - and the
disc i/o has too much latency for anything useful. If you guys could get
the latency greatly improved - even at the cost of bandwidth - I'd say
you'd have a winner on your hands.

Failing that, a very small SSD drive or a 1.8" USB based hard drive
would also work. Those 1.8" USB hard drives are like US$10. They're not
fast, but they are *much* better than iSCSI on RPS.

With the current RPS configuration, I am afraid I shall be letting my
month long trial of RPS expire at the end of this month. I'll pay the
extra for a Kimsufi.

Cheers,
Niall


--
ned14
------------------------------------------------------------------------
ned14's Profile: http://forum.ovh.ie/member.php?userid=6
View this thread: http://forum.ovh.ie/showthread.php?t=16


oles@ovh.net
09-03-2009, 10:54 AM
ned14 a écrit:
>
> I successfully moved my RPS to use its USB stick as its primary root
> filing system rather than the iSCSI SAN. Here's how:


it's not a good idea. if you don't have any swap, if you reach 0 RAM
linux will kill the process and you will lost the control of your RPS.
in the first version of the RPS (2 years ago) we didn't propose any
USB swap. then we added it since the RPS was instable without any swap.
Since, no problem.


ned14
09-03-2009, 10:54 AM

I successfully moved my RPS to use its USB stick as its primary root
filing system rather than the iSCSI SAN. Here's how:

1. Do a swapoff /dev/uba and remove the swap entry from /etc/fstab
2. In the OVH Manager change kernel to Rescue boot. After you get the
email, log into rescue mode.
3. Run mkfs.ext2 /dev/uba and mount /dev/sda1 and /dev/uba1 into /mnt
4. Do a cp -Rv from the SAN to the USB
5. Change etc/fstab ON THE SAN to mount / as /dev/uba1 (ext2) and
comment out the SAN entry
6. Change the kernel back to normal boot in the OVH manger and reboot.

The bad news is that writing speeds to the USB disk are as slow as
iSCSI - you see no improvement here. However the good news is that
system is *much* more responsive and doesn't suffer from those random
hangs because now the file system is accessed quickly. Simple things
like tab completion are now fast rather than taking a few seconds.

So this is all good. However, I'd like to do much better: ext2 is not
fast on USB flash whereas NILFS and BTRFS are MUCH better especially
with writeback turned on. However we need kernel 2.6.29 for BTRFS
support and the rescue image kernel is using 2.6.28

One HUGE advantage of gaining BTRFS support is that it comes with free
unionfs so you can place a read-only BTRFS partition on the USB disk and
overlay a read-write BTRFS partition from the SAN. Therefore one gets
all the system responsiveness of having the root filing system on USB
but also all the storage capacity on the iSCSI SAN - while of course not
exhausting the USB stick's write cycles. You also still get the big RPS
advantage of being able to change hardware config and migration of RPS
images between machines because the USB stick image can be
standardised.

Furthermore NILFS and BTRFS can be configured to run with writeback so
now the iSCSI hard drive isn't so bad for writes anymore. Ok, reads for
data appended since installation will lag, but if ALL RPS's are
implemented using this USB + iSCSI BTRFS combo then total SAN usage
decreases exponentially and lag virtually disappears for everybody

So OVH, any change of changing your rescue mode Linux image to use
2.6.29 so I can test overlaying BTRFS like above?

Thanks,
Niall


--
ned14
------------------------------------------------------------------------
ned14's Profile: http://forum.ovh.ie/member.php?userid=6
View this thread: http://forum.ovh.ie/showthread.php?t=16


ned14
09-03-2009, 10:54 AM

I was then curious about how well the RPS iSCSI scales according to
parallel loads. The below shows the results for two, four, eight and
sixteen parallel random i/o reads:

root@slave1:~# ./seeker_mt /dev/sda 2
Seeker v3.0, 2009-06-17,
http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sda [41943040 blocks, 21474836480 bytes, 20 GB, 20480
MB, 21 GiB, 21474 MiB]
[512 logical sector size, 512 physical sector size]
[2 threads]
Wait 30 seconds..............................
Results: 22 seeks/second, 44.379 ms random access time (4358910 <
offsets < 21459113390)

root@slave1:~# ./seeker_mt /dev/sda 4
Seeker v3.0, 2009-06-17,
http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sda [41943040 blocks, 21474836480 bytes, 20 GB, 20480
MB, 21 GiB, 21474 MiB]
[512 logical sector size, 512 physical sector size]
[4 threads]
Wait 30 seconds..............................
Results: 43 seeks/second, 22.762 ms random access time (42440400 <
offsets < 21472742330)

root@slave1:~# ./seeker_mt /dev/sda 8
Seeker v3.0, 2009-06-17,
http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sda [41943040 blocks, 21474836480 bytes, 20 GB, 20480
MB, 21 GiB, 21474 MiB]
[512 logical sector size, 512 physical sector size]
[8 threads]
Wait 30 seconds..............................
Results: 91 seeks/second, 10.889 ms random access time (4388120 <
offsets < 21473865920)

root@slave1:~# ./seeker_mt /dev/sda 16
Seeker v3.0, 2009-06-17,
http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sda [41943040 blocks, 21474836480 bytes, 20 GB, 20480
MB, 21 GiB, 21474 MiB]
[512 logical sector size, 512 physical sector size]
[16 threads]
Wait 30 seconds..............................
Results: 85 seeks/second, 11.696 ms random access time (8502260 <
offsets < 21461931360)

So there we have it: there is some good to this situation as threading
the i/o scales *excellently* up until eight threads or so after which I
guess the hard drives in the iSCSI SAN max out with an approx. 10ms
random access latency.

If you're planning to deploy anything real on RPS, make sure you
parallelise hard drive accesses as much as possible!!!

If only Linux were written throughout to use async i/o ...

Cheers,
Niall


--
ned14
------------------------------------------------------------------------
ned14's Profile: http://forum.ovh.ie/member.php?userid=6
View this thread: http://forum.ovh.ie/showthread.php?t=16


ned14
09-03-2009, 10:54 AM

Just to add to the above ...

root@slave1:~# ./seeker /dev/sda
Seeker v2.0, 2007-01-15,
http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/sda [20480MB], wait 30
seconds.............................
Results: 10 seeks/second, 92.88 ms random access time

Jeeeeeeeesus .... *Ten* seeks a second with nearly a 0.1 second
latency! No wonder a RPS feels like treacle ...

root@slave1:~# ./seeker /dev/uba
Seeker v2.0, 2007-01-15,
http://www.linuxinsight.com/how_fast_is_your_disk.html
Benchmarking /dev/uba [1909MB], wait 30
seconds..............................
Results: 859 seeks/second, 1.16 ms random access time


Now *that* is more like it: a 1ms random access time is fantastic in
anyone's book!

Cheers,
Niall


--
ned14
------------------------------------------------------------------------
ned14's Profile: http://forum.ovh.ie/member.php?userid=6
View this thread: http://forum.ovh.ie/showthread.php?t=16


ned14
09-03-2009, 10:54 AM

I got my RPS provisioned today which took about a week since I ordered
it. I got the RPS III with a 1.9Ghz Athlon X2 and 2Gb of RAM.

This RPS -should- be far superior to my 1.2Ghz 512Mb Kimsufi L. Yet
just in general interaction and use solely through SSH it *feels* vastly
slower. A hdparm -t shows partially why:

root@r25825:~# hdparm -t /dev/sda

/dev/sda:
Timing buffered disk reads: 6 MB in 3.33 seconds = 1.80 MB/sec

RPSs use iSCSI for their HD and 1Mb/sec is all that is guaranteed.
However to be honest I don't think the bandwidth is the problem: it's
the *latency* is the problem. The iSCSI latency can vary anything up to
a second even just during SSH use. I haven't tested it with anything
meatier than doing an aptitude safe-upgrade, but I can already tell you
that that ran like treacle - it was literally ten times slower than my
Kimsufi L. I shudder to think if one tried anything more - I tried a
find / -name at one point and I actually had to ctrl-C it.

I have to wonder if RPS in its current form is viable. It's a great
idea - many of the benefits of a VPS combined with dedicated hardware -
but the iSCSI latency just sucks too badly.

So I have a better idea: RPSs have a perfectly reasonable USB flash
disk as their swap device (2Gb in my case):

root@r25825:~# hdparm -t /dev/uba

/dev/uba:
Timing buffered disk reads: 24 MB in 3.02 seconds = 7.95 MB/sec

So why not instead place the root filing system on the USB flash drive
and then mount /home onto the iSCSI SAN device?

It might not win any performance awards but general system latency
would be vastly improved. And that would make a RPS viable for real
world usage.

Just a thought - you might also consider using one of those 1 inch usb
hard drives which cost like US$10 (e.g. http://tinyurl.com/kt8fwp). Or
perhaps a boot option which copies the root filing system into a ramdisk
and copies it back again on shutdown.

BTW you should really be using ext2 not ext3 by default for your iSCSI
partitions - ext3 likes to sync its log every five seconds which doesn't
let iSCSI scale well due to all those write commits from all those
clients. At the very least you should be using data=writeback.

Cheers,
Niall


--
ned14
------------------------------------------------------------------------
ned14's Profile: http://forum.ovh.ie/member.php?userid=6
View this thread: http://forum.ovh.ie/showthread.php?t=16