CUDA Kernel Lab

CUDA Kernel Lab is a profiling-driven GPU optimization lab for LLM-shaped primitives. It focuses on reproducible benchmark evidence, profiler validation, and clear performance conclusions.

This repo studies kernel paths: memory traffic, reductions, fusion, launch/config tuning, Tensor Core validation, and synthetic decode-step replay. Full inference serving, request scheduling, queueing, and cluster experiments belong outside this repo.

Quick Start

Create the development environment:

uv sync --group dev

Run local checks:

uv run pytest
uv run ruff check .

Install GPU dependencies before CUDA/Triton benchmarks:

uv sync --group dev --extra gpu
uv run gpu-info

Run a small local benchmark:

uv run benchmark-memory --backend all --op all --numel 16777216 --dtype float32

Live GPU Loop

Runpod is the default live-GPU provider. A standard evidence run is:

./scripts/up
./scripts/benchmark --run-id <run-id>
./scripts/down

The benchmark script copies raw JSONL to experiments/results/runpod/<run-id>/ and writes a generated report to experiments/reports/runpod/<run-id>.md. Use --platform aws only for the legacy EC2 fallback.

For focused suites, decode-step runs, H200 matmul autotune, and profiling replay, use the workflow docs linked below.

What Is Here

src/cuda_kernel_lab/
├── benchmarks/     # benchmark entry points
├── kernels/        # PyTorch baselines and Triton kernel implementations
├── ops/            # backend-neutral traffic and FLOP models
├── benchmark.py    # timing and result metadata
└── benchmark_cli.py

Implemented kernel evidence tracks:

• memory primitives: copy, scale, vector_add, reduction_sum
• fused row-wise softmax
• RMSNorm and LayerNorm forward kernels
• fused SwiGLU elementwise activation
• tiled matmul progression and Tensor Core validation sweeps
• contiguous KV-cache decode-attention PyTorch baseline
• synthetic decode-step benchmark for eager, whole-step CUDA Graph, piecewise CUDA Graph, and dynamic trace replay

Read Next

• Documentation map
• Project architecture
• Benchmark workflow
• Runpod live GPU workflow
• Profiling workflow
• Interpreting results
• Optimization strategies
• Optimization techniques
• Legacy AWS EC2 workflow

License

MIT