High-end Servers: IBM’s Power4-based Servers

The Big Blue has been in the big iron business for long and stories of their mainframes and high-end RISC processor-based servers are only too well known. This company introduced its own RISC architectures, for example the PowerPC and more recently the pServer line of servers. These servers feature mainframe-class reliability and scalability, broad support of open standards for the development of new applications, logical partitioning to allow multiple virtual servers to coexist in a single physical system, and capacity on demand. These pServer series uses the 64-bit RISC-based Power4 processors, and these servers are amongst the highest-end UNIX systems available today. They are being used for HPC (High-Performance Computing) applications like oil exploration and life sciences.

A word on the Power4

The Power4 series places two complete CPU cores on a single chip, speeding it up and adding high-speed connections to a max of three other pairs of Power4 CPUs. The OSs that run on these monsters are IBM AIX and Linux (supported by SUSE). The Power4’s core processor technology features a superscalar design that is pipelined and binary compatible with prior PowerPC designs. The Power4 processor has two integer functional units that are called Fixed Point Units by IBM. The core has a separate Branch and Conditional Register Unit forming eight execution units in all. All execution units have instruction queues associated with them that enable the processing of up to 200 instructions in various stages. 

Power4-based servers

Four Power4 chips are cast onto an MCM (Multi-Chip Module) to create an eight-way SMP (Symmetric Multiprocessing) building block. Four MCMs create the maximum of a 32-way SMP configuration. With this configuration, the rate of information per second between processors and memory is 204.8GB per sec. The figure on the right shows the interconnection of multiple 4-chip MCMs to form larger 32 way SMPs. When interconnecting multiple MCMs, the inter module buses act as repeaters moving requests and responses from one module to another module in a ring topology. As with the single MCM configuration, each chip always sends requests/commands and data on its own bus but reads on all buses. 

To communicate with other Power4 chips on the same module, there are logically four 16-byte buses. Physically, these four buses are implemented with six buses: three on and three off. To communicate with Power4 chips on other modules, there are two 8-byte buses: one on the chip and one off the chip operating at half the speed of the processor. Each chip has its own interface to the off-chip L3 across two 16-byte-wide buses, one on and one off, operating at one-third of the processor frequency. 

Reliability, Accessibility, Serviceability

As mentioned before, the Power4 processors are mounted on an MCM, which is connected directly to the p690 server’s backplane. This packaging methodology provides increased reliability by eliminating the tiered packaging levels of some UNIX servers, wherein separate processor modules are mounted on processor cards, which are further mounted on the
backplane.

Variable speed fans, termed as Air-Moving Devices by IBM for these servers, are there in the I/O units and allow for increased airflow to maintain proper cooling levels in case of a fan or cooling-system fault. The system design facilitates the recognition of component errors that are either corrected dynamically, or properly reported for isolation and repair. Parity on the system bus, CRC (Cyclic Redundancy Checking) on the RIO (Remote I/O) bus, and the extensive use of ECC on memory and arrays provide some of these capabilities.

The most likely failure event in a processor is a soft single bit error in one of its caches. For the L1, L2, L3 caches and their directories, hardware and firmware keeps track of whether permanent errors are being corrected beyond a threshold. If exceeded, a deferred repair error log is created. L1, L2 caches and directories on the Power4 chip are manufactured with spare bits in their arrays which can be accessed via programmable steering logic to replace faulty bits in the respective arrays.

The steering logic is activated during processor initialization and is initiated by BIST (Built-in System Test) at Power On time. If the p690 system is rebooted without such repair, the L3 cache is placed in bypass mode, and the system comes up with this cache disabled.

Ankit Khare