GPU Architecture &
Compute Fundamentals

⚙️ The Streaming Multiprocessor (SM)

The Streaming Multiprocessor is the fundamental compute unit of every NVIDIA GPU. All CUDA kernels execute on SMs. A GPU's throughput scales with SM count — the H100 SXM has 132 SMs, compared to 108 in the A100 and 192 in the B200. Understanding what's inside an SM is critical for the NCP-AII exam.

🔄 Warps and SIMT Execution

A warp is a group of 32 threads that execute together. The warp is the hardware scheduling unit — the warp scheduler selects which warp to issue an instruction to each clock cycle. Because warps have independent program counters, the GPU can interleave warps to hide memory latency: while one warp waits for a memory operation to complete, the scheduler switches to another warp with ready operands.

Warp Scheduling and Latency Hiding

The H100 SM has 4 warp schedulers, each capable of issuing one instruction per clock cycle. With up to 64 active warps per SM, the GPU can tolerate long memory latencies (hundreds of clock cycles) by keeping the schedulers busy with other warps. This is why maximizing occupancy (the ratio of active warps to the theoretical maximum) is crucial for performance.

Warp Divergence

Because all 32 threads in a warp execute the same instruction in lockstep (SIMT), warp divergence occurs when threads take different branches. The hardware serializes both paths, executing threads that take the "true" branch while masking threads that take the "false" branch, then vice versa. This can halve throughput in the worst case. NCP-AII candidates should understand why data-dependent branching in GPU kernels is expensive.

🎯 Core Types: CUDA vs Tensor vs RT Cores

4th-Generation Tensor Core (H100)

The H100's Tensor Cores are the engine behind its massive throughput advantage over A100. Key improvements over 3rd-gen (A100):

• FP8 support — native 8-bit floating-point (E4M3 and E5M2 formats) for training and inference, enabling the Transformer Engine to cut model size and double throughput vs FP16
• Asynchronous execution — Tensor Core operations can overlap with other SM operations using the wgmma (Warpgroup Matrix Multiply-Accumulate) instruction
• Larger tile sizes — H100 operates on 16×16×16 or 16×8×16 tiles in a single instruction vs 8×8×4 on A100
• TF32 speedup — 989 TF32 TFLOPS on H100 vs 312 on A100 (3.2×)

🧠 The GPU Memory Hierarchy

GPU memory is organized in a hierarchy that trades capacity for latency and bandwidth. Understanding this hierarchy is essential for both system design (NCP-AII) and performance optimization. The H100's HBM3 delivers 3.35 TB/s — the dominant reason GPUs outperform CPUs for AI.

📏 Bandwidth Numbers — Exam-Critical

📈 The Roofline Model & Arithmetic Intensity

The roofline model determines whether a kernel is compute-bound or memory-bound by comparing its arithmetic intensity against the machine's peak performance / bandwidth ratio (the "ridge point").

H100 Roofline Landmarks

Practical Arithmetic Intensity by Operation

Shared Memory Optimization

Shared memory is programmer-managed cache (L1-speed) within each SM. High-performance GPU kernels like cuBLAS, FlashAttention, and CUTLASS use shared memory to cache tile blocks from HBM, perform computation, then write results back — dramatically reducing the number of HBM accesses. The H100 allows configuring up to 228 KB of shared memory per SM:

// Configure max shared memory for H100 kernel
cudaFuncSetAttribute(myKernel,
  cudaFuncAttributeMaxDynamicSharedMemorySize, 228 * 1024);

📈 GPU Generations: Ampere → Hopper → Blackwell

Each GPU generation from NVIDIA represents a major architectural leap for AI. The NCP-AII exam tests your knowledge of what changed between generations and the quantitative improvements. Know these numbers cold.

⚡ Key Innovations Per Generation

Ampere — A100

• Multi-Instance GPU (MIG) — first GPU with full hardware partitioning. Up to 7 isolated MIG instances with guaranteed QoS. Each instance has its own SMs, HBM, L2 cache, and memory controllers.
• 3rd-gen Tensor Cores with sparsity — 2:4 structured sparsity support doubles effective throughput (312 → 624 TFLOPS). Works by pruning 50% of weights in a structured pattern.
• TF32 precision — new "TensorFloat-32" format: 10-bit mantissa (like FP16) but 8-bit exponent (like FP32). Drops into FP32 code paths automatically for ~10× speedup vs FP32 CUDA Cores.
• NVLink 3 — 600 GB/s bidirectional per GPU; 12 NVLink lanes.
• PCIe Gen 4 — connects to host; DGX A100 uses NVSwitch 2nd gen.

Hopper — H100 ★ Most tested on NCP-AII

• Transformer Engine — dedicated hardware that automatically selects FP8 or BF16 precision on a per-layer, per-tensor basis during training. No accuracy loss through dynamic per-tensor scaling. This is the mechanism behind H100's 4× throughput advantage vs A100 for LLM training.
• FP8 Tensor Cores (4th gen) — native 8-bit floating-point in E4M3 (higher precision) and E5M2 (higher range) formats. 3,958 TFLOPS dense FP8 throughput.
• HBM3 — 80 GB, 3.35 TB/s (vs 2.0 TB/s on A100). Critical for large model weight storage.
• NVLink 4 — 900 GB/s bidirectional; 18 NVLink lanes. DGX H100 uses NVSwitch 3rd gen (4 switches connecting 8 GPUs).
• Confidential Computing — Trusted Execution Environment (TEE) for secure multi-tenant AI workloads.
• PCIe Gen 5 — 128 GB/s host-device bandwidth.

Blackwell — B200

• FP4 Tensor Cores (5th gen) — 4-bit floating-point support for inference, enabling up to 18,000 TFLOPS. Requires careful calibration but 2× inference throughput vs FP8.
• 2nd-gen Transformer Engine — extends dynamic precision management to FP4, with per-block scaling for higher accuracy at ultra-low precision.
• HBM3e — 192 GB capacity (2.4× vs H100), 8.0 TB/s bandwidth (2.4× vs H100). Enables fitting 405B+ parameter models on a single node.
• NVLink 5 — 1.8 TB/s bidirectional per GPU (2× vs H100).
• RAS Engine — dedicated Reliability, Availability, Serviceability engine for datacenter-grade fault detection and recovery without halting workloads.

🔄 Transformer Engine — Deep Dive

The Transformer Engine is one of the most exam-critical features of H100. Understanding how it works distinguishes expert candidates from those who just memorize numbers.

Traditional mixed-precision training (AMP) uses FP16 for forward/backward passes with a static loss scaler. The Transformer Engine goes further by using FP8 (8-bit) computation with per-tensor dynamic scaling — automatically selecting the right scale factor for each tensor to prevent overflow/underflow, without programmer intervention.

The Transformer Engine exposes the transformer_engine Python library. Wrapping standard layers with TE equivalents (e.g., te.Linear, te.LayerNorm) enables FP8 compute automatically with no manual scaling code.

🔗 NVLink: Intra-Node GPU Interconnect

NVLink is NVIDIA's proprietary high-speed GPU-to-GPU interconnect, providing bandwidth that is an order of magnitude higher than PCIe. It is the critical enabler of multi-GPU AI training within a single server — allowing model parallelism and fast AllReduce communication without leaving the server chassis.

NVLink Switch System — Scale Beyond a Single Node

NVIDIA's NVLink Switch System (introduced with Hopper) enables NVLink connectivity across multiple servers using NVLink Switch chips in a separate fabric. Up to 576 GPUs can be connected in a single NVLink domain, enabling scale-up configurations with GPU-speed bandwidth across nodes — blurring the line between scale-up and scale-out.

🔀 MIG: Multi-Instance GPU

MIG (Multi-Instance GPU), introduced with Ampere, enables a single physical GPU to be partitioned into up to seven independent, hardware-isolated instances. Unlike CPU virtualization which is software-based, MIG creates actual hardware boundaries — each instance gets its own slice of every physical resource.

What MIG Partitions

H100 MIG Profiles

MIG vs MPS vs Time-Slicing

🧪 Practice Quiz — GPU Architecture

🃏 Flashcards — Tap to Flip

Click any card to reveal the answer

🤖 GPU Architecture Advisor

Select a topic for exam-focused recommendations:

🧩 Memory Mnemonics