Data Center, Networking & DPUs

GPU TDP (Thermal Design Power)

H100 SXM5 = 700W per GPU. 8-GPU DGX H100 server = ~10.2 kW total (GPUs + CPUs + cooling fans + networking). B200 = up to 1,000W per GPU.

Rack Power Density

Traditional IT racks = 5–10 kW. AI GPU racks = 30–60+ kW. A 4U rack housing 2 DGX H100 systems = ~21 kW. This density is 3–6× traditional servers.

Power Distribution

Requires high-amperage PDUs; 3-phase power distribution; dedicated electrical circuits per rack. DGX H100 requires 200–240V AC, 60A circuit per server.

Power Redundancy

N+1 = one extra PSU per server. 2N = fully duplicated power path from UPS to rack. AI production environments require at minimum N+1 for PSUs and dual power feeds from separate UPS/PDUs.

UPS and Backup Power

Uninterruptible Power Supply (UPS) provides ride-through during power events. Generator backup for extended outages. Checkpointing AI training jobs protects against unexpected power loss.

Power Usage Effectiveness (PUE)

Ratio of total facility power to IT equipment power. Ideal = 1.0. Good AI DC = 1.1–1.3. High GPU cooling load makes low PUE challenging. Liquid cooling improves PUE significantly.

Thermal Challenge

H100 GPU generates 700W of heat in a small package. 8 GPUs = 5.6 kW of heat from GPUs alone. Traditional air cooling struggles at densities above 20–25 kW per rack.

Air Cooling (CRAC/CRAH)

Computer Room Air Conditioning/Handler. Hot-aisle/cold-aisle containment; chilled water coils. Adequate for racks up to ~30 kW with careful airflow management. DGX H100 can be air-cooled.

Direct Liquid Cooling (DLC)

Coolant flows through cold plates directly on GPU/CPU chips. Highest thermal efficiency. Supports rack densities of 100+ kW. DGX H100 supports DLC. Required for B200-based systems at full TDP.

Rear-Door Heat Exchangers

Liquid-cooled door mounted at rack rear. Captures hot air exhaust and transfers to chilled water loop. Retrofittable to existing racks. Good for densities up to 60 kW.

Immersion Cooling

Servers submerged in dielectric fluid (single-phase or two-phase). Extreme density (100+ kW per tank). Highest PUE improvement. High upfront cost. Not yet mainstream for GPU clusters.

Cooling Selection Guide

Up to 30 kW/rack = air cooling. 30–60 kW = DLC or rear-door. 60+ kW = full liquid cooling or immersion. B200 systems essentially require DLC.

Floor Loading

Standard raised floors handle ~500–1,000 lbs/sq ft. DGX H100 = ~130 kg / 290 lbs for a 10U system. High-density GPU deployments may require floor reinforcement or specific rack placement.

Space and Rack Planning

DGX H100 = 10U. Determine rack units (U) needed per server. Plan for cable management, hot-aisle/cold-aisle, and service access in high-density GPU layouts.

Electrical Infrastructure

GPU clusters require dedicated electrical panels. DGX POD (20 nodes) ≈ 200 kW for compute alone. Add 30–50% overhead for cooling infrastructure.

Network Cabling

InfiniBand cables: copper DAC for short runs (<3m); fiber optic AOC/optical for longer runs. Each GPU server has multiple InfiniBand HCAs (one per GPU in rail-optimized topology).

Environmental Controls

ASHRAE Class A3/A4 allows up to 40°C inlet temperature. NVIDIA recommends 18–27°C cold aisle for air-cooled systems. Humidity: 40–60% relative humidity.

Raised Floor vs Overhead Cabling

Raised floor enables cold air distribution. Overhead cabling reduces floor obstructions. Modern AI DCs often use a combination. Overhead power busbars for high-amperage distribution.

RDMA (Remote Direct Memory Access)

Allows one server to directly read/write another server's memory without involving the remote CPU. Zero-copy; latency ~1–2 microseconds. Essential for efficient all-reduce in distributed training.

InfiniBand (IB)

Purpose-built HPC network. Lossless by design; built-in flow control; native RDMA. Speeds: EDR (100Gb/s) → HDR (200Gb/s) → NDR (400Gb/s) → XDR (800Gb/s upcoming). NVIDIA Quantum switches.

RoCE (RDMA over Converged Ethernet)

RDMA semantics over standard Ethernet. Requires lossless Ethernet (Priority Flow Control + ECN). Lower infrastructure cost than IB. NVIDIA Spectrum-X optimizes RoCE for AI.

TCP/IP Ethernet

Standard networking. Lossy by design (TCP retransmission). No native RDMA. Suitable for management networks, inference serving, storage access. 1/10/25/100/200/400GbE options.

MPI and NCCL

MPI (Message Passing Interface): programming model for distributed computing — defines collectives (all-reduce, broadcast, scatter, gather). NCCL: NVIDIA's GPU-optimized collective communications library used by AI frameworks.

Subnet Manager

Required for InfiniBand networks. Manages topology discovery, routing, and QoS. OpenSM (open-source) or NVIDIA UFM (Unified Fabric Manager) for production. Must be highly available in large clusters.

NVIDIA Quantum-2 InfiniBand (NDR)

400Gb/s per port. Non-blocking fat-tree switch (QM9700 = 64 ports). Latency <130ns. Used in DGX POD/SuperPOD. Supports SHARP in-network aggregation.

NVIDIA Spectrum-X (Ethernet for AI)

RoCE-optimized 400GbE Ethernet. Combines Spectrum-4 switch + BlueField-3 DPU. NVIDIA Adaptive Routing and congestion control. Achieves near-InfiniBand performance on Ethernet for large-scale AI training.

NVLink (Intra-Node)

Direct GPU-to-GPU interconnect within a server. H100 NVLink 4: 900 GB/s bidirectional per GPU pair. Not a traditional network — only within a node (or GB200 NVLink domain with NVLink Switch).

Storage Networking

InfiniBand can carry storage traffic (RDMA/NFS, Lustre, GPFS over IB). Dedicated storage Ethernet (25/100GbE) for object storage. GPUDirect Storage uses IB RDMA for direct GPU-storage transfers.

Out-of-Band Management Network

Separate 1GbE Ethernet for BMC/iDRAC access (IPMI). Always on even when server is off. Used for: remote power on/off, console access, BIOS updates, health monitoring. Never share with data plane.

Bandwidth Planning Example

8-GPU node with NDR IB = 8 ports × 400Gb/s = 3.2 Tb/s compute fabric bandwidth per node. Must be connected to non-blocking switch to avoid oversubscription in large-scale training.

What Is a DPU?

"Third pillar" of computing (alongside CPU and GPU). A programmable network processor with ARM cores, high-bandwidth network connectivity, and hardware accelerators for networking, storage, and security.

NVIDIA BlueField-3 Specs

400Gb/s InfiniBand or Ethernet connectivity. 16 Arm Cortex-A78 cores. Hardware offload for RDMA, cryptography, packet processing, storage. PCIe Gen5 to host CPU.

DPU Use Cases in AI Data Centers

• Network offload — move RDMA/IB processing off CPU
• Storage offload — NVMe-oF target/initiator processing
• Security — encrypted networking, zero-trust enforcement
• Isolation — host vs. tenant isolation in cloud environments

DPU vs SmartNIC

SmartNIC = network card with some on-card processing. DPU = full programmable data center infrastructure accelerator with its own OS (Linux on ARM), persistent services, and hardware accelerators — much more capable than SmartNIC.

How DPUs Help AI Workloads

CPU cores freed from networking/storage I/O can be used for data preprocessing. RDMA offload reduces latency for all-reduce. Enables microsegmentation and encrypted data flows without performance penalty.

NVIDIA DOCA SDK

Data Center Infrastructure on a Chip Architecture. SDK for programming BlueField DPUs. Provides APIs for networking, storage, security services. Enables containerized services running on DPU ARM cores.

All-Reduce Bandwidth Requirement

Gradient synchronization generates traffic proportional to model size × 2 × (N-1)/N per step. For large models, hundreds of GB per step across all nodes. Fabric must be non-blocking.

Congestion Control

Multiple flows compete for bandwidth in large IB/RoCE fabrics. Without congestion control, packet drops (RoCE) or credit stalls (IB) reduce throughput. NVIDIA Adaptive Routing + ECN-based congestion control address this.

Latency Sensitivity

Synchronous training means all GPUs wait for slowest all-reduce. Even small latency spikes (straggler nodes) impact cluster efficiency. Low-latency networking (IB) minimizes waiting time.

SHARP — In-Network Computing

Scalable Hierarchical Aggregation and Reduction Protocol. NVIDIA Quantum-2 performs gradient aggregation inside the switch — not at end nodes. Significant bandwidth savings for large-scale training.

Topology and Oversubscription

Non-blocking fat-tree has no oversubscription — ideal for training. 2:1 oversubscription halves effective bandwidth. Training workloads should use non-blocking; inference may tolerate some oversubscription.

Storage Network Separation

Always separate training data plane (GPU-to-GPU) from storage network. Mixing causes contention. Use dedicated IB partitions or separate physical networks for storage traffic.

Control and Customization

On-prem: full control over hardware configuration, networking design (IB topology), cooling choices, and software stack. Cloud abstracts this — you get VMs/instances, not bare metal usually.

Procurement and Lead Times

On-prem GPU hardware: lead times of 3–6+ months for large orders. Cloud allows instant access to current-generation GPUs. Critical advantage for time-sensitive projects.

Cost Model

On-prem = CapEx (large upfront) + OpEx (power, cooling, staff). Cloud = pure OpEx (pay per hour). 3-year TCO: for >50% GPU utilization, on-prem often wins; below that, cloud wins.

Colocation (Colo)

Company-owned hardware in a shared data center facility. Company provides servers; colo provides power, cooling, physical security, connectivity. Good middle ground — infrastructure control without building/operating own DC.

NVIDIA DGX Cloud

NVIDIA-managed DGX systems accessible via cloud API. Combines DGX performance with cloud accessibility. 1-, 3-month commitments. Available on Oracle Cloud, Azure, GCP, AWS.

Sustainability Considerations

On-prem allows custom renewable energy procurement. Cloud providers publish sustainability metrics (AWS, Google, Azure all have carbon commitments). Liquid cooling + renewables are key for GPU-intensive AI workloads.