Enterprise Data Center Design and Methodology

A raised floor is generally constructed on a grounded framework, with a load surface of two- foot square tiles (also called panels). The space beneath the floor is called the plenum . Feeding conditioned air directly from the HVAC units into the plenum is simple, and gives the flexibility to channel air, in varying degrees, to the locations where it is needed. The plenum is also generally used to route cables and mount electrical outlets that feed right up to the racks. This plan keeps the cabling out of the way, eliminating the possibility of people tripping over them, or accidently unplugging a vital system.

Floor Height

The height of the floor depends on the purpose of the room. Height should be based on air conditioner design and anticipated subfloor congestion. A typical floor height between the subfloor and the top of the floor tiles is 24 inches (61 cm), though a minimum height could be 18 inches (46 cm). The floor height could go as high as 60 inches (152 cm) but, of course, you would need added HVAC to pressurize such a large plenum. The height of the floor is also relative to the total height of the floor space. A 14-foot vertical space with a 5-foot high raised floor leaves only nine feet. This doesn't allow enough ceiling height for air return.

Support Grid

The support grid for the floor has several purposes. It creates the open structure below the floor to allow for the routing of cables, supports the load surface (tiles) and equipment, and is used for part of the "signal reference grid." There are many types of support grids from different manufacturers.

The following figure shows a recommended system that utilizes bolted stringers and provides maximum rigidity for dynamic loads.

If you intend to use an alternative system, such as snap-on stringers, make sure you research them carefully to ensure that they meet the necessary load and stability specifications.

If the data center is to be located in a seismically active area, seismic bracing should be considered for the raised floor system. Verify that the floor manufacturer supplies supplemental bracing before making the decision to use a particular system. If this is not an option, bracing systems are available from several manufacturers that could work with existing equipment.

When determining the type and specifications of the support grid you must anticipate all the possible weight that could be placed on it at one time. Racks full of equipment, HVAC units, equipment on dollies, forklifts or floor jacks , a tour of people, etc. The weight specifications of the floor must exceed this potential weight.

Floor Tiles

A raised floor is typically constructed on a grounded framework, with a load surface consisting of interchangeable tiles (sometimes called floor panels). The tiles can be solid, perforated, or grated. There are many different types of floor tiles, designed for different loads, and to either prohibit air flow or allow specific amounts of air flow through them. Some tiles have custom cutouts for cable or utility passage. There is a great deal of flexibility for designing air flow patterns using tiles with specific air flow characteristics. Solid tiles can be placed to redirect air flow and create subfloor pressure. Perforated tiles can be placed to redirect air flow while also letting a certain percentage of the air flow up into the room or directly into equipment racks.

Tile Construction

Floor tiles are typically 24 in. x 24 in. (61 cm x 61 cm). Historically, the tile cores have been made of compressed wood, concrete, or an open structural metal design. These tiles usually have a point load of 500 pounds . While there are solid tiles from certain manufacturers that allow a load higher than 500 pounds , you should make sure your stretcher system is also rated to handle this type of load. Even if these solid tiles and stretchers can support higher floor load ratings, perforated tiles might not. The use of perforated tiles that can handle higher loads might be required for heavy, bottom-cooled equipment.

Choose tiles based on structural integrity and specific load requirements. Wood or concrete might not support heavier loads. Sun Microsystems Enterprise Technology Centers are now installing cast aluminum tiles to handle the floor loads. These tiles can support a point load of over 1,500 pounds, whether the tiles are solid, perforated, or even grated tiles with a 55 percent pass-through.

The following figure shows an example of a cast aluminum tile.

Note

It is best to use tiles with adequate load specifications so they don't warp or become damaged. If this happens, replace them immediately. An ill-fitting tile can pose a safety hazard to people and equipment.

The floor surface must allow for the proper dissipation of electrostatic charges. The floor tiles and grid systems should provide a safe path to ground through the tile surface, to the floor substructure and the signal reference grid. The top surface of the floor covering to understructure resistance should be between a minimum of 1.5 x 105 ohms and a maximum of 2 x 1010 ohms (as per NFPA 56A Test Method). The tile structure (not the surface laminate) to understructure resistance should be less than 10 ohms.

Never use carpeted tiles. Carpets can harbor contaminants that are agitated every time someone walks on the tiles. These tiles are more easily damaged by the movement of hardware, or when removed using specially designed tile lifters that incorporate spikes to catch the loops of the tiles. Also, carpeted tiles designed with static dissipative properties can become less effective over time.

Tile Customizations

Some tiles must be modified to fit around columns , accommodate odd room shapes , or to allow access for conduits , pipes, and cables. All modifications to tiles should be done according to the manufacturer's recommendations and guidelines. The exposed cut edges of all cut-outs must be capped with custom corners or protective trim for the safety of people handling the tiles, to avoid damage to cables, and to optimize the air pressure under the floor. Exposed corners can also shed particulate matter into the airstream.

Additional structural support might be necessary, especially where partial tiles are installed along walls, around columns, or by air conditioners.

For information on floor tile maintenance, see the manufacturer's specifications.

Plenum

A plenum (pronounced PLEH-nuhm, from Latin meaning "full") is a separate space provided for air circulation, and primarily to route conditioned air to where it is needed in the data center. It is typically provided in the space between the subfloor and raised floor, and between the structural ceiling and a drop-down ceiling. The plenum space is often used to house data and power cables. Because some cables can introduce a toxic hazard in the event of fire, special plenum-rated cables might be required in plenum areas. This is subject to local fire code.

Wireways and Outlets

An efficient method of bringing power to the racks on the floor is to put the power under the floor where it is needed. Beneath the floor tiles are outlets set into a wireway which is usually a long metal box that houses the electrical wiring and outlets. The power cables from the racks drop down through cutouts in the tiles and plug into these outlets. The outlets are connected back to circuit breakers and sub-panels by electrical wiring. You could run power cables from each breaker in a sub-panel out to the floor, but the problems of messy cabling under the floor, air flow blockages, and vertices would develop. Centralizing the runs of this electrical wiring to a few areas helps reduce this problem.

Also, power outlets need to be attached to something. A wireway is a combination of conduit housing and a place to secure outlets. Once you know where machines will be on the floor, you know where to place the wireway. Also, using your RLU definitions for each location, you know exactly how many outlets, and of what type, should go to those specific locations on the floor. Your electrical engineer and electrical contractors can then size the wireways accordingly . You want the wireways to be as small as possible, for the least amount of air flow blockage, but they must meet local electrical codes.

The following figure shows an example of a raised floor system. It shows the tile surfaces, plenum (open air space), pedestals, cable tray, and the outlets set into a wireway. It also shows the concrete subfloor.

Cable Trays

To keep cables out of harm's way, it is normal to run cables under the raised floor. While many data centers just drop the cables down into the floor, this causes quite a lot of cabling chaos (not so affectionately known as "spaghetti") under the floor. This tangle makes tracing bad cables difficult and time-consuming . Also, large numbers of cables will create air flow obstructions in the raised floor. These obstructions inhibit the free flow of air in the under-floor plenum and decrease under-floor pressure past these blockages. See Chapter 9, "Network Cabling Infrastructure" for more details.

The use of cable trays under the floor serves as a way to organize cables and limit blockages under the floor. The cable trays are generally U-shaped wire baskets (sometimes called " basket wireways") that run parallel to the wireway that houses the electrical outlets. In many cases, these trays will be joined to this wireway, either on top of the wireway or on the opposite side of the outlets. This minimizes air vertices under the floor that can lead to decreased air pressure.

Note

Make sure you factor in at least one and a half inches of space between the top of the cable tray and the bottom of the raised floor tiles to keep the cables from getting crushed. Two inches or more is preferable, but this space could be dependent on the depth of the plenum.

Placement of Wireways and Cable Trays

Before final design plans are completed, you should determine the layout of racks on the raised floor. This will tell you where the fronts and backs of the machines will be and, therefore, which aisles will be cool (intake) aisles and which will be hot (exhaust) aisles . After you know the position of the racks, you can determine precisely where the electrical wireways for your outlets will be placed and on which side of the wireways the outlets will be. It is often standard procedure for an electrician to orient all of the outlets for all of the cable trays in the same direction, unless directed to do it differently.

The following figure shows examples of a few possible options for wireway orientation.

This placement is critical. Poor planning could set things up so that the rack is on top of a tile that covers the outlet and the outlet is facing the wrong direction. Good planning can save you from having to move a machine or crawl under the raised floor to plug in a rack.

This is also a good time to determine where the under-floor cable trays will be installed. The cable trays help organize the network and storage cables and keep them out of the plenum where they would block air flow. Excess power cords should also be placed there.

• Layout A: This shows a back-to-back electrical wireway configuration. You could put the cable tray in the middle. You will still have quite a bit of dangling cable because the outlets are far from the server. This will work only if your RLU definitions have very few network and storage cables defined in them.

• Layout B: This is probably the most efficient layout. The wireway and outlets are arranged so they can be accessed by removing a tile from the aisle area. The run length from the outlet is shorter than Layout A. Excess cable can be placed in a cable tray, either on the opposite side of the outlet or on the top of the wireway.

• Layout C: If you don't look at these types of details in the design process, you could find yourself faced with Layout C for every other row of equipment in your data center. Even though you can lift the tile to get access to the wireway, you will still have to get to the other side to plug it in. If you like working with flashlights and mirrors you could use this layout, but it doesn't fit the "simple" part of the design philosophy.

• Layout D: This is the worst of the four layouts. The outlet is not only in the wrong orientation, but it is also under a floor tile that has a rack on top of it. You would have to move the machine two feet to get the tile up to access the outlet. Why is it mentioned here? Because this mistake sometimes happens and now you know to avoid it.

The following are things to consider when planning this layout:

• Does code dictate how the wireways must be placed?

• Will there be adequate space between the top of the cable tray and the bottom of the raised floor tiles? This is important to keep all cabling in the tray from getting crushed. An absolute minimum of 1.5 inches is recommended between the bottom of the raised floor tile and the top of the wireway or cable tray, whichever is higher.

• Can you get to the cabling below the floor without having to move any racks? (Moving racks that are in service is not an option.)

• What is the best layout so that the excess electrical cable can be placed in the wireway without spilling over the sides?

It might be a good idea to create a mock-up using whatever materials work best for you, from coffee stirring sticks and sugar cubes to 2x4s and cardboard boxes, to figure out the best layout for the wireways.

Routing Wires and Cables

There are four different types of cabling in a data center. The first two types are installed during the construction phase. Once these cables are installed, they should not be changed, except by professionals.

• Power wiring from breakers to outlets. These go in the wireways under the raised floor.

• Network " home run" cabling from points of distribution (PODs) on the floor to the network room. These cables should be bundled together, and their run to the network room should be routed above the raised floor. To maximize air flow under the raised floor, these are usually routed in a separate cable tray in the ceiling plenum.

The second two types of cabling are installed and removed along with racks on the floor by data center personnel. They are routed along the cable trays under the raised floor.

• Power cables to the racks. These are the power cables for the racks that come up through a cabling access in the raised floor from the power outlets.

• Network cables from network PODs to devices. These cables connect devices to PODs, or connect devices to other devices.

Ramps and Lifts

There are two methods for getting equipment up onto the raised floor: ramps and lifts.

Ramps are the most common. The structural integrity and angle of the ramp are the two biggest factors. Ramps usually go from outside the data center to the staging area. The ramp must not only support the weight of the equipment, but the weight of the pallet, packing materials in which the equipment is shipped, and the weight of the mechanical device used to move the pallet. Mechanical devices are usually hand or electrical powered pallet jacks. Electrical pallet jacks can easily weigh 800 pounds by themselves . Add that to a 2,200 pound Sun Fire 15K server with packing materials, pallet, etc., and the load weighs over 3,000 pounds. That's one and half tons. But wait, that's not all! Add a motorized pallet jack and two or three people to maneuver the pallet jack, open doors, etc., and the ramp is now supporting a rolling load of close to 4000 pounds, or two tons. It is a good practice to have a fully qualified structural engineer looking into this construction detail.

The scale of the ramp must also be considered. These can range from 1 in 12 (that's 1 inch of rise for every 12 inches of length ”a pretty steep ramp) to 1 in 20. A ratio of 1 in 20 is probably more suited to moving large equipment. But, a 1 in 20 ramp for a 24-inch raised floor must be at least 40 feet long. Also, there should be level space at the top and bottom of the ramp to ensure that the pallet jack is in the correct alignment. Add a minimum of 8 feet on each end for that and you have a ramp section 56 feet long. It will probably be 10 feet wide. That's 560 square feet of space just for the ramp.

Building a ramp to support your data center is not a trivial task. Some sites are building ramps with poured concrete. This is not as absurd an idea as it might seem. As previously described, a Sun Fire 15K server with packing material and a motorized pallet jack weighs over 3400 pounds. The unladen weight of a 2002 BMW 330i sedan is 3285 pounds. If your ramp can't handle the weight load of that BMW, it can't handle the weight load of that Sun Fire 15K server.

Lifts are platforms placed on the edge of the raised floor and can raise the equipment to the height of the raised floor surface. While lifts can save space, they are a more expensive alternative. Also, a lift will only be so large, once you size the lift, that is the size of the largest thing you can lift with it. Remember to choose lifts that will accommodate both the size of the pallet jacks you will use and the people operating them. Also, you will be subject to local code restrictions. Code might dictate that you must have a ramp as well as a lift.