Forum Settings
       
Reply To Thread

Choosing hard drivesFollow

#1 Apr 23 2012 at 3:24 PM Rating: Excellent
Meat Popsicle
*****
12,018 posts
So here's an odd question. We're looking for some hard drives for the lab. We've run across some slightly unusual requirements, so we're bit at a loss of where to start. Basically we don't care so much about size, a few hundred gigabytes is fine, but instead we're looking to buy drives that have a couple of features.

1) The ability to quickly write large numbers of files. A lot of the files we generate are smaller text files, 1-10kb in size, but we generate them by the millions. Having the ability to read/write those quickly is a plus.

2) Long term stability. Something like a 2-year warranty is out. Hopefully will find something that will last longer than that without having to be pitched. Data would be archived other places than these disks, so that isn't so much the concern, but keeping them running over a time-frame like 5-6 years without mechanical difficulties, corruption, etc. would be nice. Having to re-analyze lost data isn't fun. Smiley: frown

Any thoughts? I've never shopped for drives like this before, so wasn't even sure how to start comparing things.


____________________________
That monster in the mirror, he just might be you. -Grover
#2 Apr 23 2012 at 3:53 PM Rating: Good
****
7,820 posts
I don't know that I can help on the speed aspect, but I've got a Samsung HD in my home PC, and it's been chugging along with zero problems for over 5 years now.
____________________________
People don't like to be meddled with. We tell them what to do, what to think, don't run, don't walk. We're in their homes and in their heads and we haven't the right. We're meddlesome. ~River Tam

Sedao
#3 Apr 23 2012 at 6:03 PM Rating: Excellent
Avatar
******
29,905 posts
are the files being written sequentially, or in mass batches all at once? Also, when you say "long term stability" do you need to access those same files continually throughout those same 5 years, or can you offload those to backup as they are processed.

You have a couple of options here, and there are going to be tradeoffs in terms of speed and price whichever route you go.

At the top of the heap, you have things like the Texas Memory systems RamSAN http://www.ramsan.com/ Which essentially employes a boatload of solid state drives in a fiberchannel Storage Array Network (SAN) attached array. It isn't going to get any faster with publically available technology at the moment then that or one of the FusionIO type systems. You are looking at a minimum of $20,000 to get into one of their small servers, plus a fiberchannel attached server ~$7,000ish (something like a HP dl580 G8 server), and possibly a fiberchannel switch ~$2,000 - ~8,000, depending on options. You're going to want a multiple core server with as many processors and cores as you can possibly cram in it, and a fair amount of very high speed ram. Solid State operating system drives in the server too.

If the cost of a small luxury car is not in the budget, you can build a fairly fast disk array on a desktop class system. Something based on a fusion-IO ioDrive ioFX for example (http://www.solidstateworks.com/) would run about $2,000 for the drive + machine costs ($2,000 - $3,000 depending on feature set). A server chassis would probably still give you better throughput with multiple processor cores than a desktop would.

If you need to go low budget, simple solid state drives would probalby work. you want the fastest one available, with at least a 6GB / second SATA III controller. A Single Level Cache or SLC is going to be faster than the MLC chips, but much more expensive.

You also want to think about backup windows and hardware redundancy. With the really expensive SAN approach, drive failures are accounted for in the chassis, and can usually be replaced without even shutting down. An enterprise class backup solution of some sort like ArcServ R16 with the open file agent could be configured to back your files up nightly, weekely, or monthly, etc without impacting operations on the ones that were open. In the desktop solution, you're more looking at either running drives in a raid 1 or raid 5 array for hardware redundancy, and then installing a secondary drive array to push backups to as needed. then off to tape.

Without knowing an approximate budget, I can't really give you a good idea of the best possible approach, but that should at least give you a few things to think over. There are "Build your own SAN" options out there, and lots of drives is going to tend to be better suited for lots of little file writes. For small files in all cases a solid state drive is going to be better for throughput these days. In some cases for large files, a 15,000 RPM rotational drive in a raid 0 array might have a faster write rate.

If possible, bring a few different solutions in and then run benchmarking on them using hard drive specific benchmark utilities such as attodisk. Here are a few individual drive results. http://www.harddrivebenchmark.net/high_end_drives.html
____________________________
Arch Duke Kaolian Drachensborn, lvl 95 Ranger, Unrest Server
Tech support forum | FAQ (Support) | Mobile Zam: http://m.zam.com (Premium only)
Forum Rules
#4 Apr 24 2012 at 10:23 AM Rating: Excellent
Meat Popsicle
*****
12,018 posts
Dread Lörd Kaolian wrote:
are the files being written sequentially, or in mass batches all at once?


Files are written sequentially from a single process, but multiple processes will be going on at once. The processes themselves don't take very much overhead, they're all basically reading and parsing text files and writing output. But when you stack up a dozen or so, add in some zipping, deleting, and other file generation stuff things get busy.

Dread Lörd Kaolian wrote:
Also, when you say "long term stability" do you need to access those same files continually throughout those same 5 years, or can you offload those to backup as they are processed.


The files themselves get offloaded and archived on a regular basis, so that's not a problem in my mind at least. However the drives we're using seem to be having issues as time goes on. We lost one a couple of months ago, were just now replacing another which is taking about 10x as long as normal to access data on it, and there's another we're suspecting has problems. For reference we're mostly using these; there's a scattering of sizes.

To give you an idea of what we're working with, we have several of these guys (or similar) we've bought over the last few years usually spending about $4k-5k a pop. An ideal solution would be mounted internally, but that's not to say other solutions are off the table.

Going the server route has been beyond our price range for the most part. Like you say, $20k-30k seems to be about the low end of what we've seen, and that's a bit much on our budget. Though ironically we probably have about that much invested in hardware over the last 5 years or so.

As for a budget, we've only spent a couple hundred dollars per drive in the past, and have been going with the mentality that's the price range we'd like to stay. However given all the problems with the drives we have, we're rethinking that a bit. Still we have about a half dozen computers we'd be looking to replace drives on, so things will add up fast. Smiley: frown
____________________________
That monster in the mirror, he just might be you. -Grover
#5 Apr 24 2012 at 12:44 PM Rating: Excellent
Avatar
******
29,905 posts
Ah. THe westernd Digital "green" series drives are the slowest pieces of junk on the entire planet. they are in fact slower than a 5200 RPM hard drive. They try to make up for it with cache, which for your process does you almost no good at all. Solid state drives would be a major improvement. FusionIO cards should be a massive improvement. The fiberchannel SAN should be able to handle multiple computers, and the T5500 is essentially a server motherboard more than a workstation, so you can add a fiberchanne. card to it pretty easily.

I'd bring in some SSD's or a couple fusionIO cards forst to test and see if that improves the situation enough, and if not keep those in play as OS drives and bring in a fast SAN to write things to. If nothing else, you shouldn't be able to find anything slower than a western digital green drive. Floppy drives might be an improvement.
____________________________
Arch Duke Kaolian Drachensborn, lvl 95 Ranger, Unrest Server
Tech support forum | FAQ (Support) | Mobile Zam: http://m.zam.com (Premium only)
Forum Rules
Reply To Thread

Colors Smileys Quote OriginalQuote Checked Help

 

Recent Visitors: 20 All times are in CST
Anonymous Guests (20)