The second meeting in the PG&E series at the Critical Facilities Roundtable was about power distribution at data centers and was held at Cupertino Electric in San Jose.

The first speaker was Spas Lazarov of GE Energy Service, who spoke about points of power loss in data centers.

Spas Lazarov

The power use ratio in a data center varies from one data source to another, but Spas showed the ratio in the following picture.

Power use ratio in a data center

In his graph, I think the cooling ratio is a little too small, but it is within a reasonable range. The two items that consume the most power are IT and cooling. The power required for power distribution is 10–20%, small enough not to be very noticeable. There has been a lot of discussion about conserving power for IT (refreshing hardware and virtualization) and cooling (hot and cold aisle, containment, and air economizer). Although some work has been done, efforts to curb power consumption for power distribution have been minimum. Once power enters a data center, it changes its voltage and form a few times before it hits IT equipment. The following is a picture that shows the normal course of power distribution in a data center.

In a normal data center, UPSs are placed to prepare for power loss or poor quality power.

There are several UPS configurations, but Spas talked about double conversion mode and high efficiency mode. In the double conversion model, power, which entered the data center from the grid, always goes through the UPS. This is to ensure that IT equipment always receives good quality power in terms of frequency and voltage. Power from the power grid is AC. When that AC enters a UPS, AC is transformed via a rectifier to DC power for the battery inside the UPS. Then DC power is taken out of the battery and converted to AC power via an inverter. Two conversions introduce power losses, with an additional loss at the battery. If instead the UPS is operated under the high efficiency model, which bypasses the UPS in a normal situation and taps the UPS battery only in the case of an emergency, power loss is minimized, as shown in the next picture. In the figure, PUE increases because more power is fed to IT equipment rather than wasted in power distribution.

Spas pointed out that power quality in the US and most developed countries is good enough not to worry about. He gave us the example of a 50,000 square foot data center with a 5 MW load and a power price of $0.01 per kilowatt hour saving $3M annually by means of the high efficiency model. He also said that if all UPSs were operated under this model, $3B could be saved globally.

This is great, but what if the supply power is cut off from the grid? Can your servers survive the power interruption? Spas showed a diagram, below, with some details about the way power components are put together inside a server.

Normally, a server contains a capacitor. As in the power grid, which has capacitor banks at strategic locations like substations and distribution networks, this capacitor inside the server maintains power for the server in case of power loss from outside. As shown in the picture above, a typical capacitor can maintain power for 20 ms in case of power loss. When you check your server’s specification, that information should be described on it.

Spas also showed us that operating UPSs in the high efficiency model does fit within the ITI (CBEMA) Curve set by the Information Technology Industry Council. ITI is the high tech sector's advocacy organization in Washington, D.C., and in foreign capitals around the world. It deals with such issues as corporate tax reform, trade, telecommunications, cybersecurity, energy efficiency, workforce and STEM initiatives, regulatory compliance, accessibility, and environmental sustainability.

The ITI (CBEMA) Curve and application note describe an AC input voltage envelope that typically can be tolerated (no interruption in function) by most information technology equipment.

As I dive into smart grid and power engineering, I revisit the issues in electrical and mechanical functions in data centers. Although I came from the IT sector, I think I am beginning to understand the facilities perspective a little bit better.

The second presenter at the recent Critical Facilities Roundtable (CFRT) meeting was Melissa Bautista of PG&E. Her presentation was about what PG&E is doing to help its high tech customers, especially data center operators, conserve energy.

Melissa Bautista and her presentation

I’ve heard a lot in bits and pieces about what PG&E is doing regarding their support for improving energy efficiency at data centers. Melissa’s presentation was a good summary of what PG&E is doing and how you can get help from them.

Melissa reminded of us that PG&E covers a large territory, mostly northern and central California and part of Nevada. On the map she showed us, the territory seemed to cover about half of the entire state of California.

As part of power deregulation, PG&E sold some of its generation assets, but it still has 6,800 MW of generation capacity (3,896 MW hydro, 2,240 MW nuclear, 163 MW gas/oil, and 500 MW solar), according to Wikipedia. PG&E buys the rest from other power generators. In comparison, TEPCO generates about 60,000 MW. TEPCO’s territory is much smaller, but its population density and industry concentration are much higher. Besides, TEPCO owns all its generation sources and does not buy any power from others, which will change if the current renewable energy promotion law is passed.

For data centers, PG&E specifically provides the following:

What PG&E can help you with

The rebate program is available for the areas mentioned below:

Available rebate program areas

Finally, Melissa outlined how to get your rebate check:

Process of applying for the rebate program and getting a check

PG&E has a specific URL for high tech customers. Melissa recommended reading two reports on data center energy efficiency: Energy Efficiency Baselines for Data Centers (47 pages)  and PG&E’s Energy Management Solutions for Data Centers (6 pages).

I have not read them yet, but at a casual look they both seem to contain very useful information for improving energy efficiency at data centers. It is good for a utility like PG&E to invest time and money to help data center operators conserve energy and get rebates for doing so. There are three more meetings jointly scheduled by CFRT and PG&E. I will announce them when the dates are confirmed

At the recent Critical Facilities Round Table conference, there was a lot of talk about container-based data centers, which are gaining in popularity these days.

The slide below, presented by Brian Canney of IBM, shows that a container-based data center is one form of flexible data center design at IBM.

The next slide, shown by Rich Hering of M+W Group, summarized environmental standards for data centers.

The slide below shows industry reliability and data center classifications.

Rich also surveyed the current manufacturers of container-based data centers. In the beginning, there were only few offerings from a handful of manufacturers, but there are quite a few now, which indicates a growing market.

Rich also presented the reasons why the data center industry went for modularity and containers, as shown below.

He then listed the current issues with container-based data centers.

The next slide is specifically for environmental issues.

Rich then talked about how hot a data center can be. In the next slide, he showed a request from several IT companies to ASHRAE to raise the acceptable temperature standard for data centers.

In the following slide, he pointed out the problems to consider.

Finally, Mark Bramfitt talked about his report Modular/Container Data Centers Procurement Guide: Optimizing for Energy Efficiency and Quick Deployment, which I have yet to read. The link to the report is here.

I have attended several Critical Facilities Roundtable (CFRT) events in the past. CFRT is a group of data center users who share industry trends, technologies, practices, and friendship. Their meeting is usually held at a data center site, and after a few presentations, a tour is conducted at that site. Because of the group’s nature, the participants usually have the rare opportunity to tour at a production data center, such as Yahoo’s (hosted in Digital Realty Trust).

I had exchanged a few words with Bruce before but had not had a chance to sit down and talk to him. So I caught him at his new office in San Francisco right after his return from two business trips. In spite of the busy schedule of his new position, he was nice enough to give me time to interview him.

Bruce Myatt at his office

The following is a summary of our conversation.

CFRT was founded in 2003 by four people, including Bruce. The other three people left for one reason or another, and Bruce is now in charge. CFRT is a group of professional data center users and providers, and its purpose is to share new technologies and practices as reflected in the members’ day-to-day operations. The group’s focus is on Silicon Valley rather than national.

Data Center Pulse (DCP) is somewhat similar to CFRT, but DCP members interact via websites like LinkedIn, and CFRT has a hands-on approach.

CFRT consists of several committees and subgroups, and most are facilities focused; there are committees for the high-density data center, legacy data centers, and energy. However, a recent creation of the technology group has begun to change the focus and make it more diversified, concerning itself with IT issues, including cloud computing, virtualization, and high energy efficiency in server and other IT equipment technologies. The technology group is formed by a team from the previous AFCOM Northern California chapter, as I have reported before.

Over the seven to eight years since CFRT’s founding, a lot has changed in the data center marketplace. Of course, the data center field has expanded tremendously, but the technology and operational advancements are the most noteworthy. At its formation, CFRT people knew how the changes would take place, and the changes materialized, and they are happy with the result. Technology has changed so much that the way data centers are designed, constructed, and operated is completely different today.

I asked Bruce if there was an energy crisis back then. He said that the energy crisis for data centers did not come to the surface until the EPA report in 2007. And since then, it has become one of the most important issues for the data center market.

For the next five to ten years, advancements in technology will make data centers smaller yet very efficient in both energy and throughputs. A container-based data center is a test bed for that. It is standardized in form, efficient, and mobile.

Bruce predicted that every data center would be equipped with measuring and monitoring functions, based on something like SNMP, to report its minute-by-minute status.

As for one of the most-discussed subjects of IT and facilities integration, he thinks that tools like HP’s OpenView and BMC’s Patrol can collect data from both IT and facilities sides now. The question is how you use the collected data for control. This is not a technology question but an operational issue. His funny examples are whether we want to play music on a refrigerator or set an oven timer via a cell phone. This is a good example of "do you really want to do it if you have the technology to enable it?” Unless the data collected from both camps is put to use for a real reason, the situation would remain the same.

We also talked about LEED, Energy Star for data centers, Tier, and international markets for data centers. But I will blog on them in the future.