Upgrading and Repairing Servers

Current servers use DDR SDRAM, DDR2 SDRAM, and in some cases RDRAM. The following sections discuss these memory technologies in detail, as these are the memory technologies you'll be most likely to encounter in building and servicing servers.

DDR SDRAM

Double data rate (DDR) SDRAM memory is a JEDEC-created standard that is an evolutionary upgrade of standard SDRAM in which data is transferred twice as quickly. Instead of doubling the actual clock rate, DDR memory achieves the doubling in performance by transferring twice per transfer cycle: once at the leading (falling) edge and once at the trailing (rising) edge of the cycle. This effectively doubles the transfer rate, even though the same overall clock and timing signals are used.

DDR SDRAM was first used for graphics cards, and starting in 2001, it began to show up in PCs, although Intel did not officially support DDR until early 2002. Most recent server designs use DDR SDRAM.

DDR DIMMs come in a variety of speeds or throughput ratings and normally run on 2.5 volts. They are basically an extension of the standard SDRAM DIMMs, redesigned to support double clocking, where data is sent on each clock transition (twice per cycle) rather than once per cycle, as with standard SDRAM. To eliminate confusion with DDR, regular SDRAM is often called single data rate (SDR). Table 5.7 compares the various types of standard DDR SDRAM modules used in servers. As you can see, the raw chips are designated by their speed, in megatransfers per second, whereas the modules are designated by their approximate throughput, in megabytes per second.

^[1] Key: MT/s = megatransfers per second; MBps = megabytes per second; DIMM = dual inline memory module; and DDR = double data rate.

The bandwidths listed in Tables 5.6 and 5.7 are per module. Many recent server and desktop chipsets support dual-channel DDR memory, in which two DDR DIMMs are installed at one time and function as a single bank with double the bandwidth of a single module. For example, the Intel E7320 and E7221 server chipsets use dual-channel DDR memory. The E7320 supports the 800MHz front side bus (FSB) version of the 64-bit Xeon processor, and the E7221 supports Pentium 4 processors with the 800MHz FSB. Both processors transfer 8 bytes (64 bits) at a time, for a bandwidth of 6,400MBps (800x8 = 6,400). With an 800MHz FSB processor installed, these boards use standard PC3200 modules, installed two at a time (dual-channel), for a total bandwidth of 6,400MBps (3,200MBps x 2 = 6,400MBps). This design allows the memory bus throughput to match the CPU bus throughput exactly, resulting in the best possible performance. Note that the AMD Opteron processor's integrated memory controller also supports dual-channel DDR memory. You can optimize PC design by ensuring that the CPU bus and memory bus both run at exactly the same speeds (meaning bandwidth, not megahertz), so that data can move synchronously between the buses without delays.

DDR2 SDRAM

JEDEC and its members began working on the DDR2 specification in April 1998 and published the standard in September 2003. DDR2 chip and module production actually began in mid-2003 (mainly samples and prototypes), and the first chipsets, motherboards, and systems supporting DDR2 appeared in mid-2004. DDR2 is supported by many of the newest server chipsets, such as the Intel E7221.

DDR2 SDRAM is simply a faster version of conventional DDR-SDRAM memory: It achieves higher throughput by using differential pairs of signal wires to allow faster signaling without noise and interference problems. DDR2 is still double data rate, just like DDR, but the modified signaling method enables higher speeds to be achieved, with more immunity to noise and cross-talk between the signals. The additional signals required for differential pairs add to the pin count; DDR2 DIMMs have 240 pins, compared to the 184 pins of DDR. The original DDR specification tops out at 400MHz, whereas DDR2 starts at 400MHz and goes up to 533MHz, 800MHz, and 1000MHz. Table 5.8 shows the various DDR2 module types and bandwidth specifications.

^[1] Key: MT/s = megatransfers per second; MBps = megabytes per second; DIMM = dual inline memory module; and DDR = double data rate.

In addition to providing greater speeds and bandwidth, DDR2 has other advantages. It uses lower voltage than conventional DDR (1.8V versus 2.5V), so power consumption and heat generation are reduced. Because of the greater number of pins required on DDR2 chips, the chips typically use fine-pitch ball grid array (FBGA) packaging rather than the thin small outline package (TSOP) chip packaging used by most DDR and conventional SDRAM chips. FPGA chips are connected to the substrate (meaning the memory module, in most cases) via tightly spaced solder balls on the base of the chip.

DDR2 DIMMs resemble conventional DDR DIMMs but have more pins and slightly different notches to prevent confusion or improper application. DDR2 memory module designs incorporate 240 pins, significantly more than conventional DDR or standard SDRAM DIMMs.

RDRAM

Rambus DRAM (RDRAM) is a fairly radical memory design found in high-end PC systems and servers from late 1999 through 2002. Intel signed a contract with Rambus in 1996, ensuring that it would support RDRAM into 2001. After 2001, Intel continued to support RDRAM in existing systems, but new chipsets and motherboards primarily shifted to DDR SDRAM, and all Intel chipsets and motherboards since that time have been designed for either conventional DDR or DDR2. RDRAM standards had been proposed that will support faster processors through 2006; however, without Intel's commitment to future chipset development and support, very few RDRAM-based systems were sold in 2003, and almost none after that. Due to the lack of industry support from chipset and motherboard manufacturers, RDRAM will most likely not play a big part in future PCs or servers.

With RDRAM, Rambus developed what is essentially a chip-to-chip memory bus, with specialized devices that communicate at very high rates of speed, a technology originally introduced for the Nintendo 64 and Sony PlayStation 2 game systems.

Conventional memory systems that use FPM/EDO or SDRAM are known as wide-channel systems. They have memory channels as wide as the processor's data bus, which for the Pentium and up is 64 bits. The DIMM is a 64-bit wide device, which means data can be transferred to it 64 bits (or 8 bytes) at a time.

RDRAM devices, on the other hand, are narrow-channel devices. They transfer data only 16 bits (2 bytes) at a time (plus 2 optional parity bits), but at much faster speeds. This is a shift away from a more parallel to a more serial design and is similar to what is happening with other evolving buses in the PC.

16-bit single-channel Rambus inline memory modules (RIMMs) originally ran at 800MHz, so the overall throughput is 8002, or 1.6GB per second for a single channelthe same as for PC1600 DDR SDRAM. Pentium 4based desktops and servers using Rambus memory typically used two banks simultaneously, creating a dual-channel design capable of 3.2GBps, which matches the bus speed of the original Pentium 4 processors. The RDRAM design features less latency between transfers because they all run synchronously in a looped system and in only one direction.

Newer RIMM versions run at 1,066MHz or 1,200MHz in addition to the original 800MHz rate and are available in single-channel, 16-bit versions as well as multiple-channel, 32-bit versions, for throughputs up to 4.8GBps per module.

A single Rambus memory channel can support up to 32 individual RDRAM devices (the RDRAM chips), and more if buffers are used. Each individual chip is serially connected to the next on a package called a RIMM, but all memory transfers are done between the memory controller and a single device, not between devices. The individual RDRAM chips are contained on RIMMs, and a single channel typically has three RIMM sockets. The RDRAM memory bus is a continuous path through each device and module on the bus, with each module having input and output pins on opposite ends. Therefore, any RIMM sockets not containing a RIMM must be filled with a continuity module to ensure that the path is completed. The signals that reach the end of the bus are terminated on the motherboard.

Each RDRAM chip on a RIMM1600 essentially operates as a standalone module sitting on the 16-bit data channel. Internally, each RDRAM chip has a core that operates on a 128-bit wide bus split into eight 16-bit banks running at 100MHz. In other words, every 10ns (100MHz), each RDRAM chip can transfer 16 bytes to and from the core. This internally wide yet externally narrow high-speed interface is the key to RDRAM. The overall wait before a memory transfer can begin (latency) is only one cycle, or 2.5ns maximum. RDRAM is also a low-power memory system. RDRAM modules use only 2.5V and support power management.

As discussed previously, RDRAM chips are installed in modules called RIMMs. A RIMM is similar in size and physical form to current DIMMs, but RIMMs and DIMMs are not interchangeable. RIMMs are available in module sizes up to 1GB or more and can be added to a system one at a time because each individual RIMM technically represents multiple banks to a system. They have to be added in pairs if your motherboard implements dual-channel RDRAM and you are using 16-bit wide RIMMs.

An RDRAM memory controller with a single Rambus channel supports up to three RIMM modules according to the design. However, most motherboards implement only two modules per channel to avoid problems with signal noise.

RIMMs are available in three primary speed grades, with three different width versions in each grade. The 16-bit versions are usually run in a dual-channel environment, so they have to be installed in pairs, with each one of the pairs in a different set of sockets. Each set of RIMM sockets on such boards is a channel. The 32-bit version incorporates multiple channels within a single device and is therefore designed to be installed individually, eliminating the requirement for matched pairs. Table 5.9 compares the various types of RDRAM modules. Note that the once-common names for RIMM modules, such as PC800, have been replaced by names that reflect the actual bandwidth of the module to avoid confusion with DDR memory.

^[1] Key: MT/s = megatransfers per second; MBps = megabytes per second; and RIMM = Rambus inline memory module.

One of the reasons that RIMMs were formerly favored by Intel is that their throughput matched the throughput of the original 400MHz Pentium 4 and Xeon processor buses. However, with the rise of dual-channel DDR and DDR2 memory and processor buses up to 800MHz and beyond, DDR and DDR2 memory provide the throughput needed, making them the best choices for modern server platforms.