First, let me give you some background...
Over the last 5 years I've been working on making an Extreme Media Server. What frankly started out as a competition between me and another tech at PC Perfect has ended up where all these things end up - one person going over the top and pushing beyond any level of sanity. I'm happy to report that someone is me!
The first solution was simple enough, a basic PC with a few hard drives hanging off it. Definitely NOT EXTREME, very standard and, if I may say so, really really really boring!
The second solution was a small rolling rack of Super Micro 3U cases. A nice solution no doubt, but what I call an "Uncle Sam Solution". You know - one you'd come up with if you have unlimited funds and don't care about energy usage, cooling, etc. - essentially solving the problem with money.
That brings us up to this year, when I came across a great article from BackBlaze talking about how they build their storage pods for their online backup service (http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/). Wow - Wow - Wow! Someone had come up with a way to store 45 Drives in one reasonably sized case and had done it on a budget. With a little tweaking, I knew I could take their core design and end up with awesome Media Server. Sadly, my wife recognized the look in my eyes and feared the boxes she knew would be coming soon as well!
Before we continue, I want to take a moment and thank the great folks over at BackBlaze. They openingly published their design and have been great in answering what seems like a million of my questions and providing me invaluable guidance in tracking down parts to make this happen. Without their generosity, time, and openness, this frankly wouldn't have been possible.
For those of you who are long time readers of techdirt.com, you'll be familar with concept of sharing information and building a business model around it. It is very rare these days to find a company like BackBlaze that is so open with their designs/intellectual property. Most businesses protect and horde this type of information. I have the upmost respect for the folks running BackBlaze - they're definitely forward thinkers on top of their game!
Next up, the game plan/design criteria...
Subscribe to:
Post Comments (Atom)
Posts
I can't wait for you future posts. I want to build a media server too. Well, a bigger one than my 8TB that has a giant case and external case for HDs. The blaze will be great to sit in the closet and just run a cat 5 to it and setup a wireless connector. Hope Windows XP Pro works.
ReplyDeleteMy thoughts exactly - I've been messing around over the years trying to find a "perfect" storage solution, and only when I saw Backblaze open-design I thought to myself: This is it!
ReplyDeleteSure, I won't be using nearly all of the available drive space right from the start (maybe start with 7 or 8 hard drives) - but to know that we can plug in extra drives at any time... that's awesome.
From my calculations we can get a stripped-down-entry-level backblaze system for $1100. (google translated link, sorry)
By the way, is there any good (software) solution for limitless "growing/expanding" storage space adding discs - preferably keeping some form of redundancy (as RAID5 or better)? That would be my main concern right now...
>> From my calculations we can get a stripped-down-entry-level backblaze system for $1100.
ReplyDeleteThere are a couple areas that will kill you on cost. The Case and PM Backplanes are big ones. The Case was a bit of a surprise as when you go to order just one the cost is about $100 more than what BackBlaze has listed on their site (single qty versus multiple qty that they typically order). I think the case alone cost me almost $1k with shipping. I also haven't found a low quantity low cost source for the PM backplanes. I ended up ordering 10 of them and was fairly happy to pay around $50/each for them plus shipping and this was from a wholesaler that actually shipped them in from Twain to me. (I ordered 10 to have a spare just in case.)
Of course, you still need to get all the little extras that don't come with the case - nylon mounts for the backplanes, nlyon screws, right angle molex connectors to make custom power cables for the PMs, fans, rubber-band sleves and so on. All these come with nice shipping charges as well and have to be sourced from individual suppliers.
I guess I'm saying it just adds up fairly quickly. If you are building 10 or 100 of them, then it is easy to control the costs - much harder when making just one of them.
Also, after you start working in/on the case, you'll just want all the wiring done so you don't have to go back and mess with it later. The idea of just adding a PM down the line sounds good, but in practice it is a major PITA as you have to literally rip everything out to do it.
>> is there any good (software) solution for limitless "growing/expanding" storage space adding discs
I'll being using a standard Windows/SMB Share with Mount Points (i.e.\\MSERVER\HDs\HD1-1, HD1-2, HD1-3, ..., HD3-11, HD3-14, HD3-15). This will allow easy access on the network and setup on the server side is literally just a few clicks for each new drive. Basically each hard drive will be a folder within a share on the network.
This probably isn't actually what you were hoping for though. I also remember seeing a Linux solution for a software based RAID that could easily grow/expand. Just add a new hard drive and get more storage. I'm not a big RAID fan, but it looked like an interesting solution when I read an article on it a few years back. Unfortunately I can't find any reference to it any longer - maybe some of the Linux gurus can help ???
Don
@Don
ReplyDeleteYes, you point out a few details I was worrying about: being just one unit, the prices will be higher; and some/most components are better installed right from the start.
I'll look forward for your next posts detailing the assembling process and the actual final cost of everything.
Meanwhile, I think one of my 1TB backup external HD just got busted this morning... which reminds me never to trust on any single drive again.
That's why I think a Raid solution is the only way to go in such large scale... The bigger the hard drives (2TB) the more you lose.
It would be nice to be able to assemble multiple raid units (for example, each group of 5 discs in RAID5) - but then we'd also need an easy way to actually identify which HD is which, in case any of them fails. (We wouldn't want to remove/replace a good one instead of the bad one! :)
In Linux you do have software raid (dmraid I believe) though there are also other - perhaps more suitable - solutions for "expandable and reliable file systems" (I've been toying around with ZFS in FreeNas.)
But in Windows there are also "software raid" solutions (most "low-cost" Raids are.)
Well, the day will come when our Internet connection will make all this redundant, and we'll be able to access our data at Backblaze servers just as fast as if it were at home.
(My ISP just recently began offering 1Gbit net connection, although at 200eur/month. - though I can't complain: we now have 50Mbits+cable TV+phone at around 60Eur/month :)
... Just another thought.
ReplyDeleteWhat's your take on using low-noise fans to move the air around the case?
(I'm particularly fond of using Nexus 9/12dB fans in my setups and have been quite happy with it - my latest "power PC" is actually my less noisy computer (and it has dual Nvidia GTX 275 cards.)
And with the power management that keeps most drives idle, reduced air flow would be even less of an issue.
Have you analyzed disk temps in normal/extreme cases just to know how "bad" it could get? :)
Hi Guys,
ReplyDeleteI am about to start building one of these in the UK.
As for linux RAID reshaping, see
http://neil.brown.name/blog/20090817000931
http://scotgate.org/2006/07/03/growing-a-raid5-array-mdadm/
To identify the dives I would recommend writing the serial # on the top of them then running
hdparm -I /dev/sd? | grep Serial
To get the serial for each drive
You may like this script
"""
#!/bin/sh
letters=(a b c d e f g h i j k l m n o p q r s t u v w x y z
aa bb cc dd ee ff gg hh ii jj kk ll mm nn oo pp qq rr ss tt uu vv ww xx yy zz)
for i in "${letters[@]}"
do
if [ -e /dev/sd$i ]; then echo "/dev/sd$i : " $(hdparm -I /dev/sd$i | grep Serial); fi
done
"""
(There is probably a neater way of generating "letters")
Don - which wholesaler did you use for the backplanes?
I have some experience with ZFS and will post my experiences with OpenSolaris (which has port multipliers working now) and FreeBSD on my blog
Cheers,
thattommyhall
>> Don - which wholesaler did you use for the backplanes?
ReplyDeleteThese guys were able to hook me up.
http://chyangfun.com/
They typically won't bring in less than 100 PM units, but I got lucky and was able to get just 10 from them - they had stock already in Twain.
Nice and helpful folks, light on the English though and wholesaler mindset.
Don
>> I have some experience with ZFS and will post my experiences with OpenSolaris (which has port multipliers working now) and FreeBSD on my blog
ReplyDeleteCan you post a link to your blog as well on here for us?
Thanks!
Don
Carlos,
ReplyDelete>> What's your take on using low-noise fans to move the air around the case?
I've actually run into a bit of problem with the low noise/volume fans. Okay if drives in power management mode, but I have some heat issues otherwise.
This will be a bit of a balancing act. For now, I'll stick with the low volume/noise fans since I have power management enabled and the media server is contained in room with a dedicated A/C unit.
If I was going the RAID route for all my drives, I'd go with the biggest baddest fans that could move a ton of air to be safe.
Don
>> To identify the dives I would recommend writing the serial # on the top of them then running
ReplyDeleteI love this script! That's awesome - thank you for posting.
If anyone has anything like this for Windows it would be a great additional.
In Windows, Disk Manager isn't very good at getting detailed info on the drive (serial #, etc.). As far as I tell, looking at a drive in Disk Manager, Explorer, etc., it would be near impossible to figure out what actual drive it was for this kind of setup.
Silicon Image installs an applet in the Control Panel when you load the drivers. This utility is great for showing the brand, make/model, and serial number of each drive and what port it is in. Unfortunately, it doesn't also add in what drive letter or mount point it is associated with.
It appears all the information is there in the system. Just a matter of presented it in a more useful way.
Looks like maybe another project :)
Don
@Don
ReplyDeleteCan you post a screengrab of that applet, I wouldn't mind trying to code a C# program to list all the data and mount points of the disks in that "more useful way". :)
Hi Guys,
ReplyDeleteMy blog is at www.thattommyhall.com
The most relevant articles are at http://www.thattommyhall.com/category/zfs/
I have just found a place in the UK that could fab the case, next step is to test OpenSolaris and FreeBSD to see if they support the port multipliers.
Tom
thattommyhall@gmail.com
(This is where I should have posted the other comment, sorry.)
ReplyDeleteApple abandons ZFS.
Tom, any idea on how this will affect (or not) ZFS in the long term?
RE:ZFS
ReplyDeleteNot really, it is released as part of OpenSolaris under the CDDL, Apple and FreeBSD have ported the code to their kernels (for licence compatibility reasons Linux cannot do the same, though someone has done it in Fuse).
I have ordered the port multiplier to test support in OSs other than Linux and windows and have someone working on the design of the case to make it work with punchout machines rather than laser cutting to save money (apparently the original design has curves on the fan grills that complicate it, I don't really understand)
I have been hired by an ISP to investigate clustering software to layer on top of these, possibly presenting a windows share via Samba too, I will keep my blog updated with any progress.
thattommyhall
@Tom
ReplyDeleteThanks, will be interesting to know what you'll come up with.
@Don
ReplyDeleteHere's a script for Windows that will output something similar to the linux one...it uses WMI instead of talking the scsi disk driver like hdparm, but it might get you what you're looking for. run it via cscript //nologo for parseable output:
'Begin script
Set wmi = GetObject("WinMgmts:")
Set disks = wmi.ExecQuery("SELECT * FROM Win32_DiskDrive")
For Each disk In disks
WScript.StdOut.Write disk.DeviceID & ": " & Trim(disk.SerialNumber) & vbCrLf
Next
'End script
Since you're using Win2008, you could probably use get-wmiobject, im just more familiar with vbs/js than ps on Windows.
Reverendjw -
ReplyDeleteJust wanted to thank you for posting the vbScript. Just used it tonight and it worked like a champ and will come in extremely handy and will be essential when replacing bad drives if you use a Windows RAID and so on.
Thanks!
Don