This raid array is pretty smokin. I needed new storage the other day, so I bought me a stack of 1 TB disks on sale from NCIX, stuck em into my box, and put ‘em into a raid 5 configuration. It took a bit of fucking around to move the windo$e installation onto ‘em (thankfully robotcopy is on the vista installer disk), but the NIx stuff, as always was pretty easy.
Getting it to play nice with the OS’s however… that was a bugger. I wanted to make sure that my block size boundary and my stripe boundarys were the same (or rather 2x) so that any write would line up perfectly and reduce the number of operations required to keep parity data up to date. That way my write speed wouldn’t be crippled, as is all to common under windows when using raid 5.
Well, much to my chagrin, Windo$e doesn’t wanna boot from an NTFS volume with a 64k allocation unit size, so using a 32 k stripe was outta the question. So it took a bit of fucking around to get a good combination and multiplier, and tuning the onboard controller, and one hell of a lot of time rebuilding the array.
Well, it finally all worked out. I’ve been checking things out with diskbench, and I’m getting 180mb write a second burst, 228-330 mb / second sustained (depending on fragmentation), and a whopping 1180mb/second read on a 3 TB filesystem. Fucking cool.
I got to thinking that back inna day, when I was at Portal, the entire net content, whether via usenet, the web, gopher, archie (yeah, I’m actually that old. One of the few that ever did IP networking while the net was still only university and government), etc – the combined filesize of everything, – the whole lot of the net – was estimated to be a whopping 2.6 TB. Less storage than I have on my home computer now. The amount of contiguous storage I have here under LVM would have cost billions of dollars just 10 short years ago.
It just amazes me sometimes.
To think that when my 13 year old self upgraded my TRS80 to 32k of RAM, and bought a 128k floppy reader, I thought that it was more storage and ram than I could ever use in my life. *lol* We now have games that take up more storage than all of Norad had available until the early 90s.
Now think of all that storage for all those servers online… just for advertising and porn… lol.
The Editor of WIRED was speaking at a keynote speach during Videofest and recalled the day he realized that the new cheap bargin bin PC he just bought his son at a retailer had more storage space than all the computers in the WIRED head office combined (including the server)
The coolest thing about your raid array definitely has to be the whole “Time Machine” aspect.
The fact that you are posting this at 3:19 this afternoon is fucking amazing. Don’t suppose you feel so inclined to post the winning lottery numbers for tonights draw, would ya?
Huh? I posted at 3:19 am this morning. My system clock is right, and the post looks like it was at 3:19 am to me. Huh?
I agree something is screwy. On the post, it says “@ 2009-02-20 15:19:00″.
3ware controller or what?
Very very odd.
Nothing to do with my controller. seems to have something to do with LJ.
Tried posting from another linux box, and got the date time 12 hours ahead as well. Had to go in and backdate the entry.
tried IE and FF too. bizzare.
Re: Very very odd.
I didn’t expect to have to be anything to do with the controller. Just something with LJ as you note.
What make/model is the controller tho? You don’t say anywhere. My desktop has a 3ware 9650SE-8LPML these days, just running 4x750GB in a RAID10 config tho. I’m waiting for good SSDs to come out, and then they’ll go in the last 4 slots.
Re: Very very odd.
Surprisingly it is an NVIDIA onboard controller with the 780i chipset. The 680i was frankly crap, so I’m shocked at the improvement. I set up raid 5, largely because I needed the redundancy, but deep down, I really wanted a raid 0 for performance. NV raid has a manager that allows you to step down to a raid 0 on a live set if you want to, so I decided to try 5 first and then step down to 0 or 0+1 if the disk writes were as abysmal as I remembered from the 680. (something awful like 20mb / sec).
Of course, I spent a lot of time making sure my stripe and blocksize multiples would result in aligning on the stripe boundary, making sure aligning the disks, and that my partions started in the right places, etc.
Still I was only expecting maybe up to 100m write, not the almost full sata2 speed that I’m getting.
pretty impressed with this onboard controller I gotta say.
Never used a 3ware controller myself, but with a multi-lane backplane, I’d expect that they are pretty smokin.
Most of the stuff I’ve read about SSD and hardware raid is not great right now. Seems that linux software raid loves ‘em tho.
Lemme know how it works out if you go that route.
Re: Very very odd.
Regarding the alignment on disk, stop using MS-DOS partition tables. Use GPT exclusively, and the problem is MUCH easier to deal with.
I stick to RAID10 these days, because I’ve been horrified by some of the multi-disk failures that are popping up. See a recent thread in the linux-ide mailing list, where people are starting to use DIFFERENT brands of disks deliberately to reduce their failure rates.
Re: Very very odd.
Interesting. I hadn’t even thought of GPT. Old habits die hard I suspect.
Thanks for the tip.
The failure rate is more due to the massive size of disks these days and the chance of partiy writing issues. there’s a interesting formula posted on wikipedia that shows that by the time you get to a ~10T array, your chance of a parity issue is near 100%
Disks from disparate sources or maufactruing batches is a common practice, but there’s been no creditable study that i’m aware of that has shown a real world advantage. be interesting to see one tho.
Re: Very very odd.
Yes, different sources/batches but the same model is common already. It was just surprising that they were using totally different brands and models. One fellow with bad luck was running a 3-disk RAID1: Hitachi, Seagate, WD.
1+0 0+1
Ouch.
How big were the drives? It’s interesting running the mean time to failure math on various combinations of stripe size, disk count, and total sector count. Sometimes RAID is actually worse than using SLED. lol. Especially now that we are getting over the 1TB per platter threshold.
I’ve always thought that RAID1 is about as complicated as you wanna go with small servers or workstations, but those of us who have more complex setups or huge data stores kinda have to look at other solutions.
I like the idea of 1+0, for the fact that you can’t split the mirror of the raid 1 which is definitely nice to be able to do in a pinch. So if I have the disks, I do prefer the 0+1 method, although in theory, it does mean having to regen more disks on a failure…
For my home workstation, 5 seems to be fine though. ‘s not like I have to so a lot of high speed multi-user services on this machine.
Poor dude with the wipe tho. having multiple disks fail at the same time… ugh. It smells like an environmentally induced failure. Co-failure of any two disks is statistically unthinkable unless they are statistically linked. Did he have a heat problem? Any idea?
I have a seperate cooling unit and fan for my drive cages in each of my cases. A lesson learned by an ole IBM deskstar disk that actually caught fire back in 2000…
Re: 1+0 0+1
For the 3-disk RAID1 guy, that’s what he was on now, after a failure of a 2-disk heterogeneous setup. IIRC that faillure was 250GB drives, WD+Maxtor, and now he went with 3x 750GB I described. No high performance needs, just integrity and very seldom idle (continuous slow logging). He was lucky to also have sane weekly backups to tape. From everything I could see that he detailed, he covered all the bases from clean input power to temps in keeping a good environment. Just unlucky.
RAID10 (stripe on top of 2 mirrors) is nice for being able to take one disk from each mirror and still have perfect data – You claim to not be able to do it, but that’s just that some controllers can’t do it. It’s the risk of the second-disk failure in RAID0+1 why I prefer RAID10.