Ultra-Fast C++ JSON Transaction Parser

This repository contains a high-performance, single-threaded C++17 JSON parser designed to process massive JSON datasets (e.g., 10 GB+) without encountering Out-Of-Memory (OOM) errors.

In standardized benchmarking, this parser effortlessly processes 10.2 GB of unstructured JSON in 14.5 seconds (~700 MB/s) on a standard consumer CPU.

The 3 Pillars of Performance

Standard JSON parsers build a full Document Object Model (DOM) tree in memory, which crashes when file sizes exceed physical RAM. This parser utilizes three advanced techniques to bypass the OS file-stream bottleneck and eliminate heap allocations:

1. Zero-Copy File I/O (mmap)

Instead of using standard std::ifstream and expensive system calls to chunk data into memory, this program uses the POSIX mmap (Memory Mapping) system call.

• The Concept: mmap maps the file directly into the virtual address space of the process. The Linux Kernel's page cache streams the file into RAM precisely as needed.
• The Result: The program uses practically zero physical memory and avoids double-copying data from kernel-space to user-space.

2. Zero Heap Allocations (std::string_view)

Instead of extracting JSON string keys and dynamically allocating memory on the heap (e.g., creating millions of std::string objects), this parser implements a custom linear scanner using C++17 std::string_view.

• The Concept: A string_view acts as a lightweight pointer and length directly into the mmap memory. It scans the JSON purely by moving pointers.
• The Result: Absolute zero heap allocations during the core parsing loop, preventing CPU cache destruction.

3. Locale-Independent Fast Parsing (std::from_chars)

Standard C++ numeric parsing (std::stod or atof) is surprisingly slow due to locale-checking overhead and the requirement for null-terminated strings.

• The Concept: This parser utilizes the C++17 <charconv> library, specifically std::from_chars, which is a low-level, locale-independent, and highly optimized string-to-float converter.
• The Result: Safely parses floating-point revenue amounts at maximum CPU speed directly from the memory-mapped views.

Build & Usage

The project is dependency-free (no external libraries like nlohmann/json). All that is required is a C++17 compatible compiler (e.g., GCC 10+).

Compiling the Parser

make

This will compile the source code and place the executable in bin/app.

Running the Parser

./bin/app <path_to_input.json>

Output Format:

1. Total Revenue of completed transactions.
2. Failure Rate (percentage of failed transactions).
3. The Category with the highest revenue (alphabetically tie-broken if equal).

Benchmarking & Data Generation

To verify the speed claims, this repository includes a high-performance JSON generator (tools/fast_gen.cpp) capable of instantly generating massively randomized valid JSON transaction logs.

1. Compile the Data Generator:

make tools

2. Generate a 10 GB Test File:

# Usage: ./bin/fast_gen <size_in_megabytes> <output_filename.json>
./bin/fast_gen 10240 massive_10gb.json

3. Run the Benchmark:

time ./bin/app massive_10gb.json

Note: Due to mmap page caching, your first run will be disk I/O bound. Subsequent runs will be served entirely from RAM, demonstrating the true parser throughput.