RTX-StreamEngine

High-Performance Hybrid C++/CUDA Local Inference Pipeline for Quantized Edge LLMs. RTX-StreamEngine is a zero-overhead local C++ inference runtime designed to run quantized Large Language Models (LLMs) efficiently on NVIDIA RTX consumer GPUs. It integrates custom CUDA kernels with TensorRT execution contexts to maximize token generation speed and lower Time-To-First-Token (TTFT) latency.

---

Key Features

• Custom Fused CUDA Kernels: Implements inline RMSNorm + Rotary Position Embedding (RoPE) fusion (.cu) to minimize VRAM memory traffic.
• Dual Runtime Backends: C++ abstract runtime supporting TensorRT-LLM execution and ONNX Runtime C++ API.
• Quantization Export Pipeline: Python script for model compression using SmoothQuant and INT4 AWQ.
• Asynchronous Execution: Non-blocking CUDA streams (cudaStream_t) to overlap compute with token streaming.

---

System Architecture

[ PyTorch Model ] ──► [ Quantizer (INT4 AWQ) ] ──► [ TensorRT Engine Plan ]
                                                              │
                                                              ▼
                                            ┌──────────────────────────────────┐
                                            │     RTX-StreamEngine (C++)       │
                                            ├──────────────────────────────────┤
                                            │ • Async Memory Allocator (VRAM)  │
                                            │ • Fused RMSNorm + RoPE CUDA Kernel│
                                            │ • Dynamic KV-Cache Management    │
                                            └──────────────────────────────────┘