It’s not always easy to correctly size your power requirements for server racks. Everything in the server world is getting smarter and faster, and this includes one of the basic parts of the server: the Power Supply Unit (PSU).
Power supplies were once a simple set of transformers and filters. Today, they are an intelligent switching and monitoring unit with their own firmware. This leads to some interesting and unforeseen issues when designing the power feeds to a rack.
The hidden costs of colocation
Power supply considerations
Here are some things to ask and take into consideration.
Are the rack feeds to be a Primary/Primary arrangement or a High Availability Primary/Redundant configuration? And how many servers and devices are single-corded or dual-corded?
Every rack has two power feeds from diverse sources. Each feed has its own PDU and associated safety circuit breakers to provide power to your equipment.
Primary/Primary arrangements are usually used with single-corded equipment where you can afford to lose power to anything plugged in should an outlet bank or main breaker trip. Primary/Primary can pack a lot of power in to a small space at the expense of providing no critical redundancy. Primary/Primary indicates you can draw up to 80% of the rated main breaker for each feed.
Have you selected Power Distribution Units (PDU) with enough outlets to supply all the devices in the rack and the correct number of circuit breaker banks protecting the outlets?
In the case of Primary/Redundant, where each piece of equipment has at least 2 power supplies, each PSU is connected to a separate PDU and feed. This keeps the equipment running in case a PDU fails or a breaker trips.
During a failure, there is a “Swing Load” as the equipment load shifts from using both PDUs to only one. Correctly-designed rack feeds for a Primary/Redundant arrangement are sized so that one PDU can supply the entire rack load to keep your critical equipment running. This is accomplished by intentionally loading no more than 40% per feed so that if one fails, the other can take the entire swing up to 80%.
We want to minimize Swing Load whenever possible. During a failure, as the power draw shifts all at once, it can overtax the remaining PDU. Swing Load in a poorly designed rack feed could cause the remaining feed to trip its breaker and fail as well, taking down the entire rack.
Are the PSUs in your equipment the older style, designed to supply half the load from each, or are they of a newer design where one provides the entire load and the other sits in standby?
In the past, dual-corded servers would draw power from their two PSUs in a balanced fashion: 50% from one and 50% from the other. Hooking one PSU to one power feed and the other PSU to the second power feed provided redundancy in case of failure. If one PSU failed or the electrical feed from the PDU failed, then the server would draw 100% from the remaining PSU.
To improve power supply efficiency, some manufacturers have redesigned intelligent PSUs. Instead of drawing 50% from each, the intelligent PSU places the entire load on one while running the other in standby — warmed up but not providing any current unless the first PSU fails.
We recently powered up a new rack with over 40 dual-corded servers with intelligent PSUs. It produced a graph like this from its two power feeds:
We placed all of the left side PSU on one PDU, and all of the right side PSU on the other PDU — a common, clean rack design.
You can see that about 11:00 on the graph, as the servers were turned on, they all started drawing their main load from the upper feed as all the PSU on one side went live. The other PSU went to standby, and all of them were connected to the lower feed, which showed no increase in power usage at all.
While the total power load was still well within the design rating for the rack, it was not balanced at all. If the upper PDU or its feed were to fail, the entire rack load would swing to the bottom feed. All 8 kilowatts would transition from one feed to the other in a fraction of a second.
When using dual-corded equipment in a Primary/Redundant configuration, have you balanced out your Swing Load?
To minimize any potential for future trouble with the Swing Load, we shut down and swapped the AC line cords for half of the servers in the rack. This placed half the favored PSU on one feed and half on the other, thus evening out the draw across both feeds and reducing the Swing Load by half should either side fail.
You can see that in the graph at 13:00 on.
Now the maximum Swing Load is only 4 kilowatts. This would not have been an issue with the old style PSU that draw equally from both sides, as the results would have been the same.
Best practice dictates keeping Swing Load to a minimum, as it takes a good deal of stress off the PDUs during failures. There will be enough alarms sounding when an entire rack of servers report loss of half their expected power that you don’t need the added worry of will the remaining feed hold up until the failure can be rectified.
asdas
Here’s another observation from the Operations crew: Make sure your network routers, firewalls, and load balancers are dual-corded as well. Keeping dual-corded servers running but unable to communicate when the single-corded router loses power is a bummer.
The better the balance of the rack loads, the better the rack will behave!
SCTG engineers can help you correctly size power requirements for your racks and assist in spec’ing and supplying the correct PDUs for your needs. Just email sales@servercentral.com.
