Sunday, October 18, 2009

Wiring up the PMs for Power

Let’s delve into wiring the power for the Port Multipliers. This is probably one of the most important areas to get right or you’ll be chasing intermittent issues and you’ll have to rip apart the server which is time consuming.

In my case, I went with just a single 1200W Thermaltake Power Supply. I did this mainly because I hate cramped cases and 2 power supplies will make the case cramped and difficult to access components on the motherboard. The second reason and probably more important reason is that I already had a 1200W PS lying around!

If you do go the two Power Supply route instead, I’d recommend a 750W or larger power supply to power the Drive/PM Section. On initial spin up, these drives pull a lot of power.

Also, just make sure you get a good quality power supply – this is the one area where you want to get the best quality possible.

So in my design, the 1200W Thermaltake Power Supply has 4 main power rails, 3 of which we can tap into for power. The first power rail (black modular connectors on the power supply) is your standard peripheral power rail which consists of 4 Modular Connectors that support standard 5V/12V devices. We’ll be using this power rail to provide 5V power to the port multipliers (3 PMs per Power Supply Modular Connector) and the 12V lines will provide power for the case fans as needed.

The second power rail which we don’t have real access to is for the motherboard.

The 3rd and 4th power rails (red module connectors on the power supply) are designed to support high-end graphic cards or power hungry PCI-E cards and only supply 12V (no 5V taps available), but we have a total of 6 module connectors which is over 20 actual 12V dedicated yellow/12V wires – more than enough for a dedicated 12V run for each PM.




One of the nice things about the PCI-E Power Connectors is that they can support up to 72A at 12V (36A per RAIL). These will be perfect for providing the 12V power to the port multipliers. Here’s a chart of the max power available on each rail.



For the actual wiring on the PM side, I took standard 90 degree Molex Y connector type cable (http://www.svc.com/fc444-28.html) and cut it. Please note that you will need the 90 degree Molex connectors as the standard Molex power connectors are too large and the port multipliers won’t fit into the case if you use them.

I then took some red, yellow and black 16AWG stranded core wire and started making my cable assemblies, being careful to bundle the 5V Ground Line with the 5V/Red Cable and the 12V Ground Line with the 12V/Yellow Cable. I’m not sure if it is okay to cross the ground lines across the rails, but better safe than sorry.

When you wire up the actual PM, what you DO NOT want to do is something like this:

16AWG PS Line -> 20AWG PWR CONN on PM -> ... -> 20AWG PWR CONN

Instead you want something like this:

16AWG PS Line -> 20AWG PWR CONN ON PM
              -> 20AWG PWR CONN ON PM






The reason the second design works better is because most of the Y cables that you'll purchase will use 20AWG or even thinner wire. By having a nice thick 16AWG wire feed from the power supply, we allow the PMs to tap into a "16AWG Power Bus" versus tapping into a "20AWG Power Bus". The wire thickness may not seem like much, but in my experience it makes the difference between having 5 drives on a PM BackPlane that power up and 5 drives that don't.

When hooking up to the actual power supply, here's the basic layout of connections:

Rail 1 – Black PS Plug 1 -> 5V -> PMs 1 to 3
Rail 1 – Black PS Plug 2 -> 5V -> PMs 4 to 6
Rail 1 – Black PS Plug 3 -> 5V -> PMs 7 to 9
Rail 1 – Black PS Plug x -> 12V -> Fans in System


Rail 3 – Red PS Plug 1 -> 12V Wire 1 -> PM1
Rail 3 – Red PS Plug 1 -> 12v Wire 2 -> PM2
Rail 3 – Red PS Plug 1 -> 12v Wire 3 -> PM3


Rail 3 – Red PS Plug 2 -> 12v Wire 1 -> PM4
Rail 3 – Red PS Plug 2 -> 12v Wire 2 -> PM5


Rail 4 – Red PS Plug 5 -> 12V Wire 1 -> PM6
Rail 4 – Red PS Plug 5 -> 12V Wire 2 -> PM7
Rail 4 – Red PS Plug 5 -> 12V Wire 3 -> PM8


Rail 4 – Red PS Plug 6 -> 12v Wire 1 -> PM9





As for the actual wiring of the PMs, I crimped 4 lengths of ~30” 16AWG wire (1 yellow, 1 red, and 2 blacks) to the PM “Y-Connectors”. I then cut the modular ends off of the PCI-E/PS connectors that would typically go to the video cards and peripherals, and then crimped these wires to them per the layout above. Any unused wires were just taped off.

For all the actual crimps, I just use standard AMP style crimp connectors. Done with the right crimp tool and technique, I'd take these any day over soldering wire.




If you haven’t done it already, time to get your hands dirty!!

27 comments:

  1. Have you tried this with all 45 drives?

    Seagate ST31500341AS for example uses 0.7A on 5V (operating) which totals 31.5A with 45 drives. This is more than your PSU is rated, and there will be other things going on too.

    ReplyDelete
  2. All this stuff leaves me wondering... how long will it take until someone starts offering a Backblaze "barebone" system, with all port-multipliers and wiring ready; where we can just drop in a motherboard and drives and be done with it? :)

    I know it's not Backblaze business-line, but it would be great if they could supply such a thing. (They'd make it much cheaper than us, due to volume discounts, and could sell it to the public at "1-unit" prices, giving them a nice margin.)

    ReplyDelete
  3. Thank you for sharing this with us! You gave some very good information!

    ReplyDelete
  4. >> Have you tried this with all 45 drives?

    I currently have 42 drives in the EMS and haven't had any issues (knock on wood!).

    I am using the WD Green 1TB and 1.5TB Drives though for the majority of them.

    Interesting point on the 5V draw. I looked at WD's site and couldn't get the numbers on the 5V versus 12V line though.

    Worse case, I'll have to put in some 12V to 5V converters to move the load around on the rails.

    For what it's worth, in my setup, I won't be accessing all the drives concurrently. In the typical usage scenario, 39 to 41 of the current 42 drives will be asleep/standby mode. I can't think of a single instance where I would have all 42 (or 45 for that matter) drives actively reading/writing data for my type of usage.

    With that said, now that I know it might be an issue, I'll be pulling out the AMP meter and see if I need to re-rig some of the 5V power lines for my drives.

    This brings up a side note, with this many drives, you definitely want to stay away for the uber performance, uber power eating drives. Not only do they generate more heat, but they'll make life much more difficult in managing your power budget as you get the case fully loaded.

    Thanks,
    Don

    ReplyDelete
  5. Hey Carlos -

    The wholesaler that got me the PMs asked if I thought there would be any interest for a 30 bay "mini" version.

    I think he is looking at something like a 100+, but if you are interested, it might be worth dropping the company a note.

    http://chyangfun.com/

    Don

    ReplyDelete
  6. Keep it coming Don, this is great stuff.

    ReplyDelete
  7. Thanks for the interesting articles.

    You don't mention using staggered spin-up to reduce the peak power requirements at startup. Are you using a controller/bios/driver combination that supports staggered spin-up?

    ReplyDelete
  8. Yeroc,

    >> Are you using a controller/bios/driver combination that supports staggered spin-up?

    As far as I can tell, the PM boards nor the SATA SilImage Based Cards support this option.

    On some WD drives, there is an option for what appears to be standby mode on power up, but I couldn't find a way to get the drives to ever spin up after power was applied when using this jumper. Maybe there is an expert out there that nows of a low-level command/API for this???

    My testing indicated that I draw about 900 Watts during initial spin-up with ~40 drives on the system. With the 1200 Watt power supply, that draw appears acceptable with room to spare.

    While it is working without staggered spin-up now, I would have to have a staggered spin-up option. My gut tells me with the existing H/W it isn't possible though.

    One of the ideas we've tossed around is a power relay board that when powered is applied would stagger the spin up of each PM/cluster of 5 drives. If we go this route, I'd probably add a USB interface to the power relay board and a way to control it via software so you could power down PMs without shutting down the server.

    Also, FYI, if you take a look at the prior post, there are pictures of the wattage used for each stage of operation: spin-up/power on, etc.

    Don

    ReplyDelete
  9. Possible power supply changes...

    >> Seagate ST31500341AS for example uses 0.7A on 5V (operating) which totals 31.5A with 45 drives. This is more than your PSU is rated, and there will be other things going on too.

    I finally got a chance to do some real world testing on my WD Green Drives and surprisingly they pull a significant amount of power off the 5V line despite the data I got from WD on this (about the same as the Seagates). This combined with the PM backplane would definitely kill the 5V RAIL Power Budget for the 1200W Power Supply.

    I still need to do more testing/verification as the #s I got from WD Tier 2 support indicate that the power budget should be about 1/5th of the numbers I'm seeing so I want to make sure my data is correct.

    However, at this point I have two options assuming the data is correct or you want to run more power hungry drives:

    1. Stick with a single power supply and find a 12V to 5V power adapter and switch probably 3 of the PMs over to it. This will use one of the 12V rails from 1200W PS and on initial check I appear to have enough left on those RAILs/power budget for this.

    2. Go with the more "traditional" BackBlaze Dual Power Supply arrangement.

    I'll update as I get a chance to do more verification, but for now, just be aware if you go with a single power supply that you may run into issues.

    Also, be aware that I'm currently running 42 drives without ANY problems even though I'm over my power budget. However, I use power management so the majority of drives are asleep most of the time so the voltage overload isn't an issue and the power supply seems to handle the initial surge overload on start up just fine.

    Don

    ReplyDelete
  10. All the "multi-rail" issues were part of the reason why I chose a single-rail PSU for my PC. The Corsair HX850W. :)

    ReplyDelete
  11. @Don

    There's been a few comments about using low noise fans...a definite must for keeping one of these things in the house, or if you have an apartment with no garage to keep it in ;-)

    I was curious, have you measured the actual sound output from your pod?

    also, what about something simple like shock mounting the drives or wrapping them to reduce vibration?

    ReplyDelete
  12. Don

    @reverendjw... Thanks for the script you posted on the blog - awesome! Will be giving that a try this weekend!

    As for the low noise fans - I just found some we had in our shop and went with those.

    As for noise, right now with 42 drives in the box (majority of them in sleep mode), it is putting out between 55 to 62 DBs accordingly to a Radio Shack Sound Meter that I bought years ago and never used until now ;)

    For what it's worth, I hadn't noticed the noise frankly until I read the noise comments on the blog. However, definitely wouldn't want it in a media room though as is.

    I could definitely let the fans go slower than they currently are and reduce the noise with my setup that has most of the drives in sleep mode. Just not a lot of heat going on.

    Ultimately I think what we need is a multiple fan temperature based controller to keep the noise as low as possible. If anyone has a link to one that can support a min. of 5 fans and preferable 6, please post.

    I haven't shock mounted my drives yet. I know this is "bad", but with my non-RAID type setup and only a few drives being awake at the same time, there just isn't a heck of lock of vibration going on. In short, its on my to-do list, just not at the top right now.

    I'm also working on a post about an issue that I submitted to Microsoft about accessing junction points under a Windows Share remotely - the junction points (i.e. hard drives) unfortunately just disappear from the list in Explorer but can still be accessed by typing in the full path).

    Not the end of the world and there are workarounds just kind of a pain. With any luck Microsoft will release a fix in their next Service Pack round, but I wasn't lucky enough or big enough to justify getting a hotfix made for it. It was nice though hearing that they duplicated the issue internally. It was then sad to essentially hear that they wouldn't be fixing it any time soon! If anyone works at Microsoft and can nudge this up the line, here's the case # (REG:209100574446845002).

    I also ordered in a set of 15-2GB Drives to replace my 30-1GB units. When I get those and replace them, I'll do a post on the actual 5V and 12V rail usage for each Port Multiplier. This should help those that want to try and stick with a single power supply or just want the field #s.

    Don

    ReplyDelete
  13. That definitely sucks... Detecting a problem and having an answer like that: "We won't fix it anytime soon." :(

    ReplyDelete
  14. WD doesn't have much info in their datasheets anymore, as far as power consumption on the different voltage lines, but Seagate's datasheet for their Barracuda LP series claims that the following:

    For the 1TB model:
    spinup = 2.0A @ 12V
    Idle = 0.218A @ 5V, 0.216A @ 12V
    Operating = 0.335A @ 5V, 0.350A @ 12V
    Standby and sleep = 0.225A @ 5V, 0.096A @ 12V

    For the 1.5TB and 2TB models:
    Spinup = 2.8A @ 12V
    Idle = 0.168A @ 5V, 0.337A @ 12V
    Operating = 0.360A @ 5V, 0.420A @ 12V
    Standby and sleep = 0.350A @ 5V, 0.010A @ 12V

    These figures go against what backblaze said about most of the power draw being on the 5V lines.

    ReplyDelete
  15. Hmm... decided to check out the consumption of the non-LP 1.5TB barracuda (ST31500341AS)

    Spinup: 2.8A @ 12V
    Idle: 0.330A @ 5V, 0.530A @ 12V
    Operating: 0.700A @ 5V, 0.610A @ 12V (40% r/w, 40% seek, 20% inop.)
    Seeking: 0.430A @ 5V, 0.663A @ 12V (random, 20% idle)
    Standby/Sleep: 0.130A @ 5V, 0.021A @ 12V

    so, while operating, it appears that this drive uses a bit more power on 5V than 12V. Another thing I just noticed is that the measurements were taken with an ambient temperature of 25 degrees Celsius, so as things get warmer, power usage might change.

    ReplyDelete
  16. Hello,

    I'm in the process of ordering all of my parts. I've already ordered the 1200W power supply, having not yet read all of these insightful comments.

    Did anyone come up with a firm resolution to this potential disaster or am a screwed?

    I will most certainly be using all 45 drives and they won't usually be sleeping or idling. I haven't been able to find a 12v-to-5v "computer" converter, most everything seems to be both large and automotive in nature. Do I need to just add another power supply?

    Any help would be appreciated.

    ReplyDelete
  17. >> Do I need to just add another power supply?

    The short answer is YES! If you are going to have a live set of 45 drives, just get a second 400/500 Watt PS for the MB, Boot HD, and Fans. It is a nice way to segment things out as well as the system seems to work better when you power up the 45 drives first and then fire up the motherboard.

    I've gotten away with a single PS and 45 live drives, but if all drives are going, the system was unstable which is no fun.

    Don

    ReplyDelete
  18. I realize this is a necropost, but more on the power consumption. I just looked at the WD data for their green drives.

    spinup: 1.75A @ 12V
    read/write: 296mA@12v, 503mA@5v
    idle: 243mA@12v, 172mA@5v
    standby/sleep: 3mA@12v, 160mA@5v

    Again, spinup is a killer.

    With some electronics knowledge, it would be possible to make some 12V pcie connector to 12/5 4-pin power connector boards. An LM7505, some transistors to add more current capacity (the LM7505 usually only can provide up to 1A by itself), and a few other parts, really.

    ReplyDelete
  19. Why would you need to use any electronics rather than just connect it all together?

    ReplyDelete
  20. @Cockroach:

    To supply more current on the 5v rail? Modern PSUs supply a lot more current to the 12v rail(s) than the 5v rail. Adding the circuit I mentioned provides more 5v current.

    45 of the WD green drives would require a maximum current of 22.64A from the 5V rail, if all are reading or writing at once, which I suspect is going to be fairly likely at boot. I don't think I have seen a PSU with that kind of 5V rating.

    ReplyDelete
  21. I should clarify.. I haven't seen PSUs that can supply enough on the 5V rail to satisfy that 22.64A in addition to the draw from the rest of the system: fans, mobo/cpu/ram/cards, plus the port multipliers themselves.

    ReplyDelete
  22. I have 30A@5V on my PSU but it's "only" 80A@12V - so it couldn't spin-up 45 drives and the system anyway. I'll be using a second PSU when I have enough drives.

    ReplyDelete
  23. It seems to me the proper solution is to find a controller that supports staggered spin-up of the devices so you don't need to over-build your PSU just for the bootup scenario.

    ReplyDelete
  24. Additionally, most PSUs have a combined wattage limit on 3.3v and 5v that you have to be careful not to hit.

    As an example, I've got a 500W PSU up on newegg that I was looking at. Looking at the sticker, it can provide 20A on 5v and 24A on 3.3v, but 5v and 3.3v can only provide 120W max. If the 5V rail is pulling all 20A, that only leaves about 6A for the 3.3v, and conversely, if the 3.3v line is using all 24A, that only leaves about 8A for the 5v line.

    Additionally, the combination of 3.3 and 5 might also share a limit with a third voltage line before the combined limit of the PSU itself.

    ReplyDelete
  25. If you want to buy all the components, use this site: www.ybytes.eu

    ReplyDelete