AI Server Systems &
Platform Design

🖥️ DGX H100 — The Exam Reference System

The DGX H100 is the most exam-tested platform. Memorize these specs completely — they appear directly in NCP-AII questions about system capacity, power, and interconnect design.

🏗️ HGX — OEM GPU Baseboard

HGX is NVIDIA's GPU subsystem for OEM partners. It includes the GPU tray (H100/H200/B200 SXM GPUs + NVSwitch chips) without the CPU, DRAM, storage, or networking. OEM partners like Dell, HPE, Lenovo, Supermicro, and Inspur integrate HGX into their own server chassis, adding their choice of CPU platform, networking, and cooling.

HGX H100 Configurations

• HGX H100 8-GPU: 8× H100 SXM5 + 4× NVSwitch 3rd gen — identical GPU/interconnect to DGX H100, different chassis/CPU/networking
• HGX H100 4-GPU: 4× H100 SXM5 + 2× NVSwitch — used in half-width 2U form factors for cloud providers with denser rack deployments
• OEM adds: CPUs (typically 2× Xeon Sapphire Rapids or AMD EPYC Genoa), system DRAM, NVMe SSDs, and ConnectX NICs

🔌 SXM vs PCIe: GPU Form Factor Comparison

NVIDIA offers H100 (and other GPUs) in two physical form factors. The choice between them is one of the most exam-critical design decisions in AI server architecture. SXM is for maximum performance; PCIe is for flexibility and standard server integration.

📊 PCIe Gen 5 — The CPU-GPU Data Path

PCIe (Peripheral Component Interconnect Express) is the standard interface connecting GPUs to the CPU and host system. Understanding PCIe bandwidth — and its bottleneck effects — is essential for AI system design.

PCIe Version Bandwidth Table

NVLink Bridge (PCIe GPUs)

For PCIe H100 deployments needing some GPU-to-GPU bandwidth above PCIe, NVIDIA offers the NVLink Bridge — a physical connector linking 2 or 4 PCIe GPUs via NVLink. However, the NVLink Bridge provides far less bandwidth than a full NVSwitch topology (limited to 2–4 GPU pairs, not a full mesh), and is not a substitute for the SXM/NVSwitch architecture in large training workloads.

🌐 AI Server Network Architecture

AI servers require multiple distinct network fabrics for different traffic types. Mixing them onto a single network causes congestion and performance collapse. The NCP-AII exam tests your understanding of which network handles which traffic, and the hardware involved.

🔧 ConnectX-7 — The AI Server NIC

ConnectX-7 is NVIDIA's 7th-generation network adapter, standard in DGX H100 (one per GPU). It supports both InfiniBand NDR (400 Gb/s) and 400GbE, making it dual-mode capable. Each ConnectX-7 directly services one H100 GPU, ensuring the GPU's network bandwidth is not shared with other GPUs.

GPUDirect RDMA

GPUDirect RDMA (Remote Direct Memory Access) allows a ConnectX-7 NIC to transfer data directly between a remote server's GPU memory and the local GPU's HBM — completely bypassing the CPU and system DRAM. This eliminates two PCIe crossings per transfer and can double effective network bandwidth for GPU-to-GPU operations across nodes.

InfiniBand NDR vs Spectrum-X for AI

💾 Storage for AI Workloads

AI training has extreme storage demands: large dataset reads during training, frequent checkpointing of multi-hundred-GB model weights, and burst I/O patterns. The storage architecture must match these demands — a slow filesystem becomes the training bottleneck even with 8 fast H100s.

GPUDirect Storage (GDS)

GPUDirect Storage enables a DMA engine to transfer data directly between NVMe SSDs and GPU HBM memory, bypassing the CPU and system DRAM entirely. This is critical for checkpointing large models (e.g., saving a 140 GB LLaMA 70B checkpoint) without saturating PCIe with CPU-routed copies.

// GPUDirect Storage read example (cuFile API)
CUfileDescr_t cf_desc;
CUfileHandle_t cf_handle;
cuFileDriverOpen();
cuFileHandleRegister(&cf_handle, &cf_desc);
// Direct NVMe → GPU DMA — no CPU data copy
cuFileRead(cf_handle, gpu_buffer_ptr, read_size, file_offset, 0);

Shared Filesystems for AI Clusters

📡 Scale-Out: DGX POD → SuperPOD

Single DGX nodes are the building block. NVIDIA provides reference architectures — DGX POD and DGX SuperPOD — that define how to interconnect multiple DGX nodes into validated AI clusters with known-good performance, networking, and storage configurations.

GB200 NVL72 — Rack-Scale Computing

The GB200 NVL72 takes scale-up further: 72 B200 GPUs and 36 Grace CPUs in a single liquid-cooled rack, all connected in one NVLink 5 domain. Every GPU can communicate with every other GPU at 1.8 TB/s without leaving the rack — eliminating the inter-node InfiniBand bottleneck for models that fit within 72 GPUs.

• Total GPU memory: 13.8 TB HBM3e — fits a 70T parameter model with room for KV cache
• FP4 performance: ~1.44 ExaFLOPS — enables real-time inference of trillion-parameter models
• Rack power: ~120 kW — requires facility-level direct liquid cooling infrastructure
• CPU: Grace (ARM-based) — tight CPU-GPU memory bandwidth via NVLink-C2C (900 GB/s coherent interface)

🧪 Practice Quiz — AI Server Systems

🃏 Flashcards — Tap to Flip

Click any card to reveal the answer

🤖 AI Server Systems Advisor

Select a topic for exam-focused guidance:

🧩 Platform Mnemonics