AI Software Stack & Cluster Deployment

CUDA Toolkit vs CUDA Driver: Two distinct components. The driver (installed with GPU driver package) is the low-level kernel interface. The toolkit (nvcc, runtime libraries) compiles and links CUDA code. Toolkit version must be ≤ driver version for compatibility.

Compute Capability: A version number (major.minor) identifying GPU architecture features. Code compiled for compute capability 9.0 can only run on H100 or newer. Always compile for the target GPU's compute capability.

CUDA Runtime vs Driver API: Runtime API is higher-level (implicit context management). Driver API is lower-level (explicit control). Most frameworks use Runtime API.

# Check driver and CUDA toolkit versions
nvidia-smi
# Shows: Driver Version, CUDA Version (max supported)

nvcc --version
# Shows: installed CUDA toolkit version

# Check GPU compute capability
nvidia-smi --query-gpu=compute_cap \
  --format=csv,noheader

# Compile for H100 (sm_90)
nvcc -arch=sm_90 kernel.cu -o kernel

MIG partitions a single physical GPU into multiple isolated GPU Instances (GIs), each with:

• Dedicated SM partition (no sharing between instances)
• Dedicated HBM memory slice
• Dedicated L2 cache partition
• Dedicated memory controllers and bandwidth
• Full fault isolation — one instance's errors don't affect others

H100 MIG profiles: 1g.10gb (7×), 2g.20gb (3×), 3g.40gb (2×), 4g.40gb (1×), 7g.80gb (1× — full GPU). The "g" = GPC slices; "gb" = HBM allocation.

Use case: Running multiple independent inference workloads on one GPU; multi-tenant environments where isolation is required.

# Enable MIG mode on GPU 0
sudo nvidia-smi -i 0 \
  -mig 1

# List available MIG profiles
sudo nvidia-smi mig \
  -lgip

# Create 7 × 1g.10gb instances on GPU 0
sudo nvidia-smi mig \
  -cgi 19,19,19,19,19,19,19 -C

# List all MIG instances
sudo nvidia-smi mig -lgi

# Disable MIG mode
sudo nvidia-smi -i 0 -mig 0

NGC provides pre-built, optimized containers from nvcr.io (NVIDIA Container Registry). Each container includes framework + CUDA toolkit + cuDNN + libraries, validated against specific GPU generations.

Key container families: PyTorch, TensorFlow, JAX, TensorRT, Triton Inference Server, NeMo (LLM training), RAPIDS, CUDA base images.

Versioning: Tagged as YY.MM (e.g., 24.01) indicating release month. Monthly releases track upstream framework + NVIDIA library updates.

# Pull NGC PyTorch container
docker pull \
  nvcr.io/nvidia/pytorch:24.01-py3

# Run with GPU access
docker run --gpus all \
  --rm -it \
  nvcr.io/nvidia/pytorch:24.01-py3

# Run with specific GPU + NVMe mount
docker run --gpus '"device=0,1"' \
  -v /lustre/data:/data \
  -v /lustre/checkpoints:/ckpt \
  nvcr.io/nvidia/pytorch:24.01-py3 \
  python train.py

SLURM (Simple Linux Utility for Resource Management) is the dominant HPC/AI cluster scheduler. GPU resources are managed through GRES (Generic Resource Scheduling).

Pyxis: SLURM plugin (from NVIDIA) that integrates container execution natively into srun/sbatch. Replaces the need for docker run wrappers. Uses Enroot as the container runtime underneath.

Enroot: Lightweight container runtime that converts OCI images (Docker/NGC) to unprivileged sandboxes — optimized for HPC shared filesystems.

#!/bin/bash
#SBATCH --job-name=llm-train
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --gres=gpu:8
#SBATCH --time=24:00:00
#SBATCH --partition=dgx-h100

# Launch with Pyxis (--container-image)
srun \
  --container-image=nvcr.io/nvidia/pytorch:24.01-py3 \
  --container-mounts=/lustre/data:/data \
  python -m torch.distributed.run \
    --nproc_per_node=8 \
    --nnodes=4 \
    train.py

NVIDIA GPU Device Plugin: Kubernetes DaemonSet that advertises GPUs as allocatable resources (nvidia.com/gpu). Required on every GPU node. Enables GPU requests in Pod specs.

MIG-aware Scheduling: The GPU device plugin supports MIG device advertisement. Pods can request specific MIG profiles (e.g., nvidia.com/mig-1g.10gb).

NVIDIA Operator: Kubernetes operator that automates deployment of GPU driver, device plugin, DCGM exporter, and MIG manager — the recommended production deployment path for GPU clusters on Kubernetes.

Kubeflow: ML workflow orchestration on Kubernetes — manages training jobs (PyTorchJob, TFJob), hyperparameter tuning, and pipelines.

# Pod spec requesting 2 H100 GPUs
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: trainer
    image: nvcr.io/nvidia/pytorch:24.01-py3
    resources:
      limits:
        nvidia.com/gpu: 2
    env:
    - name: NVIDIA_VISIBLE_DEVICES
      value: all

---
# Request MIG instance instead:
    resources:
      limits:
        nvidia.com/mig-1g.10gb: 1

NCCL implements optimized GPU-to-GPU collective operations — the backbone of all distributed deep learning. It automatically selects the best transport: NVLink (intra-node), InfiniBand RDMA (inter-node), or Ethernet.

Key collectives:

• All-Reduce: Sum (or mean) a tensor across all GPUs, distribute result back to all. Used in DDP gradient synchronization.
• All-Gather: Each GPU contributes a shard; all GPUs receive the concatenated result. Used in FSDP parameter reconstruction.
• Reduce-Scatter: Sum across GPUs, then scatter one shard to each. Used in FSDP gradient reduction.
• Broadcast: One GPU sends a tensor to all others.

Ring algorithm: Default all-reduce topology. Each GPU sends to its neighbor in a ring — O(N) data volume but O(1) bandwidth overhead regardless of GPU count. Optimal for NVLink rings.

# NCCL environment variables for tuning

# Use InfiniBand interface hca0
export NCCL_IB_HCA=mlx5_0

# Specify network interface for socket fallback
export NCCL_SOCKET_IFNAME=ib0

# Enable NCCL debug logging
export NCCL_DEBUG=INFO

# Disable IB (force Ethernet)
export NCCL_IB_DISABLE=1

# Set number of rings (often = NVLink lanes)
export NCCL_P2P_LEVEL=NVL

# Verify NCCL is using NVLink
NCCL_DEBUG=INFO python train.py \
  2>&1 | grep -i nvlink

TensorRT takes a trained model (ONNX, PyTorch, TensorFlow) and produces an optimized serialized engine for a specific GPU target. Optimizations include:

• Layer fusion: Combines sequential ops (e.g., Conv + BN + ReLU → single kernel)
• Precision calibration: FP32 → FP16 → INT8 → FP8 with calibration dataset
• Kernel auto-selection: Benchmarks alternative CUDA kernels, picks fastest for target GPU
• Memory optimization: Reuses buffers across layers to minimize HBM allocation

Engine files are GPU-specific: An engine built for H100 (SM 9.0) cannot run on A100 (SM 8.0). Always build on the deployment target GPU.

# Export PyTorch model to ONNX
torch.onnx.export(model, dummy_input,
  "model.onnx",
  opset_version=17,
  input_names=["input"],
  output_names=["output"],
  dynamic_axes={"input": {0: "batch"}})

# Build TensorRT engine (FP16)
trtexec \
  --onnx=model.onnx \
  --fp16 \
  --saveEngine=model_fp16.engine \
  --minShapes=input:1x3x224x224 \
  --optShapes=input:32x3x224x224 \
  --maxShapes=input:64x3x224x224

# Run benchmark
trtexec \
  --loadEngine=model_fp16.engine \
  --batch=32 --iterations=100

Triton (not to be confused with the Triton kernel language) is NVIDIA's production inference serving system. It serves models from multiple frameworks simultaneously and exposes HTTP and gRPC endpoints.

Key features:

• Dynamic batching: Automatically groups requests from multiple clients into batches for higher GPU utilization
• Concurrent model execution: Multiple models run simultaneously on different GPU contexts
• Model ensembles: Chain multiple models in a DAG (e.g., pre-process → model → post-process) as a single endpoint
• Backends: TensorRT, ONNX Runtime, PyTorch (LibTorch), TensorFlow, Python, FIL (forests)
• Metrics: Prometheus-compatible endpoint for latency, throughput, queue depth per model

# Model repository structure
models/
  resnet50/
    config.pbtxt        # model config
    1/
      model.plan        # TensorRT engine
  bert-large/
    config.pbtxt
    1/
      model.onnx

# Launch Triton server
docker run --gpus all \
  -p 8000:8000 \   # HTTP
  -p 8001:8001 \   # gRPC
  -p 8002:8002 \   # Metrics
  -v ./models:/models \
  nvcr.io/nvidia/tritonserver:24.01-py3 \
  tritonserver \
    --model-repository=/models

# Health check
curl localhost:8000/v2/health/ready

What's inside a NIM container: Triton Inference Server + pre-built TensorRT-LLM engines + model weights + API server (OpenAI-compatible REST API). Supports continuous batching for LLMs (token-by-token generation with dynamic slot allocation).

TensorRT-LLM: NVIDIA's LLM-specific inference library. Implements paged KV cache, in-flight batching (continuous batching), speculative decoding, and FP8 quantization for H100. Substantially higher token throughput vs vanilla PyTorch inference.

# Deploy Llama 3 70B NIM
docker run --gpus all \
  -e NGC_API_KEY=$NGC_API_KEY \
  -p 8000:8000 \
  -v /local/nim/cache:/opt/nim/.cache \
  nvcr.io/nim/meta/llama3-70b-instruct:latest

# Inference via OpenAI-compatible API
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta/llama3-70b-instruct",
    "messages": [{"role":"user","content":"Hello"}],
    "max_tokens": 200
  }'