Обсуждение: [OT] RAID controllers blocking one another?

We have a machine that serves as a fileserver and a database server.  Our server hosts a raid array of 40 disk drives, attached to two3-ware cards, one 9640SE-24 and one 9640SE-16. We have noticed that activity on one controller blocks access on the second controller, not only for disk-IO but also the command line tools which become unresponsive for the inactive controller.   The controllers are sitting in adjacent PCI-express slots on a machine with dual-dual AMD and 16GB of RAM.  Has anyone else noticed issues like this?  Throughput for either controller is a pretty respectable 150-200MB/s writing and somewhat faster for reading, but the "blocking" is problematic, as the machine is serving multiple purposes. 

I know this is off-topic, but I know lots of folks here deal with very large disk arrays; it is hard to get real-world input on machines such as these. 


Thanks,
Sean

Sean Davis wrote:
> 150-200MB/s writing and somewhat faster for reading

That actually seems dead slow. Whatever RAID level you configured, there
is no sane way for it to be that slow. Is this a RAID 5/6 array? Did you
forgot to align your file system to stripe boundaries?


--
Best regards,
Hannes Dorbath

Sean Davis wrote:
> I know this is off-topic, but I know lots of folks here deal with very large
> disk arrays; it is hard to get real-world input on machines such as these.

In my experience the 3ware support staff is competent and replies
within one business day. Just sign up at:

https://www.3ware.com/userregistration.asp?uType=eUser&boolCompany=1

and place a ticket. Though let us know when you got that solved please.


--
Best regards,
Hannes Dorbath

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 18 Jan 2008 16:54:17 +0100
Hannes Dorbath <light@theendofthetunnel.de> wrote:

> Sean Davis wrote:
> > 150-200MB/s writing and somewhat faster for reading
> 
> That actually seems dead slow. Whatever RAID level you configured,
> there is no sane way for it to be that slow. Is this a RAID 5/6
> array? Did you forgot to align your file system to stripe boundaries?

That seem that slow for SATA on a write. Assuming 20 (on each
controller) disks random write I would guess the best you could pull off
would be about 200 Megs a second. That's on a good day.

So 200MBs per controllers isn't that out of bounds. I would agree that
seems slow for reads though.

And to answer the question, no I have not seen the behavior he is
experiencing but to be honest I would never run that many sata drives.
I would have long pushed to SAS on that scale.

Sincerely,

Joshua D. Drake




- -- 
The PostgreSQL Company: Since 1997, http://www.commandprompt.com/ 
Sales/Support: +1.503.667.4564   24x7/Emergency: +1.800.492.2240
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
SELECT 'Training', 'Consulting' FROM vendor WHERE name = 'CMD'


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHkOFvATb/zqfZUUQRAoPLAKCGLze19/Z5VZ/d4TVEQQUC18dTcgCeLVTD
Abjvqf686r0eX2K1sxo8giY=
=G5t+
-----END PGP SIGNATURE-----

Sean Davis wrote:
> We are PURE amateurs, so that is possible.  We are running 12-disk RAID6
> (750GB drives--this might be a difference from what you are thinking)
> partitions under LVM.  We have write-cache enabled and are using ext3
> filesystems.  Any suggestions?

We use the 9650 controllers in RAID 6 a lot. With 12 discs you should be
able to get about 550MB/sec sequential write and 680MB/sec read easily.
But it's already pretty clear to me why you can't get anywhere near that:

The first thing is that ext3 is a file system designed for small block
devices that have a single physical head to seek -- not for large arrays
that consist of multiple physical devices that can seek independently.
Especially this is true for the anticipatory I/O scheduler that is used
in conjunction with ext3. Additionally ext3 does not know any form of
stripe alignment which is an absolute requirement for arrays that use a
parity based RAID level (RAID 5/6). If you don't stripe align your file
system to the stripe width configured in your controller, you will
suffer massive I/O trashing. Your controller and discs will be busy with
Load-XOR-Store cycles. That means that your discs will do a ton of micro
seeks on every write instead of just writing the chunk of data out. Both
performance and lifespan of your discs will suffer for no reason.

My recommendation is either Linux XFS or Solaris ZFS. Both are designed
form the ground up for such setups. If this box is not yet in production
and you like to test something I can give you a fitting mkfs.xfs line
for your setup -- the defaults won't work well.

The last thing to note is that LVM will mess with any kind of stripe
alignment, so it's quite useless for parity based arrays. If you need
serious volume management look into ZFS.

Finally you might need to do tweak some scheduler params, depending if
you chose to go with deadline or CFQ. Deadline is recommend by 3ware and
usually the best choice for PostgreSQL as well. Speaking of PostgreSQL,
put $PGDATA on a RAID 10 array, it will suffer on RAID 5/6.


--
Best regards,
Hannes Dorbath

2 more things:

Get BBUs for the controllers (If you have not already).

Upgrade to the latest 3ware firmware (code set 9.4.2). There was data
corruption bug with RAID 6.


--
Best regards,
Hannes Dorbath

Joshua D. Drake wrote:
> That seem that slow for SATA on a write. Assuming 20 (on each
> controller) disks random write I would guess the best you could pull off
> would be about 200 Megs a second. That's on a good day.

For random writes/reads that sure is OK. I interpreted those numbers as
pure sequential benchmark load.


--
Best regards,
Hannes Dorbath

Joshua D. Drake wrote:
> That seem that slow for SATA on a write. Assuming 20 (on each
> controller) disks random write I would guess the best you could pull off
> would be about 200 Megs a second. That's on a good day.

For random writes/reads that sure is OK. I interpreted those numbers as
pure sequential benchmark load.


--
Best regards,
Hannes Dorbath

Am 2008-01-18 16:54:17, schrieb Hannes Dorbath:
> Sean Davis wrote:
> >150-200MB/s writing and somewhat faster for reading
>
> That actually seems dead slow. Whatever RAID level you configured, there
> is no sane way for it to be that slow. Is this a RAID 5/6 array? Did you
> forgot to align your file system to stripe boundaries?

I have had such rates on a 3w85xx with 4+1 WD Raptor WD740 in Raid-5
(the OS was on a Raid-1 with 2+1 WD360)

The 3w95xx shoud do more, but with the new Hitachi 1 TByte I do not
get more then 140 MBit.  It seems, the drives are not so performant.

Thanks, Greetings and nice Day
    Michelle Konzack
    Systemadministrator
    Tamay Dogan Network
    Debian GNU/Linux Consultant


--
Linux-User #280138 with the Linux Counter, http://counter.li.org/
##################### Debian GNU/Linux Consultant #####################
Michelle Konzack   Apt. 917                  ICQ #328449886
                   50, rue de Soultz         MSN LinuxMichi
0033/6/61925193    67100 Strasbourg/France   IRC #Debian (irc.icq.com)