diff --git a/decode_dse/README.md b/decode_dse/README.md new file mode 100644 index 0000000..efbfa41 --- /dev/null +++ b/decode_dse/README.md @@ -0,0 +1,326 @@ +# PackedKV decode-chip study + +This package evaluates a dedicated PLENA decode chip for disaggregated +Qwen3-32B and Llama-3.1-8B serving. It keeps numerical accuracy, compiler +support, emulator support, RTL validation, timing calibration, and power +calibration as separate claims. The two pinned model configurations use the +same precision search and publication contracts; Llama-3.1-8B is the first +execution target, followed by Qwen3-32B. + +The organising principle is that **no result inherits evidence from another**. +A precision profile that is accurate says nothing about whether the compiler can +emit it; a compiler that emits it says nothing about whether the RTL executes +it; and an emulator cycle count is not an RTL cycle count. Every number carries +the tier it was measured at, and a comparison between two tiers is refused +rather than silently made. + +## Contents + +- [Execution contract](#execution-contract) — how prefill, admission and decode divide +- [Canonical search](#canonical-search) — the 3,585-profile grid and what admits to hardware pricing +- [Workflow](#workflow) — the four protocols, launch checks, and module map +- [Validity scope](#validity-scope) — what evidence does and does not carry over +- [Publication outcomes](#publication-outcomes) — how results may be framed +- [Pipeline stages](#pipeline-stages) — what `sweep pipeline` runs, in order + +To run the study on an execution host, follow `docs/server_setup.md`; this +document describes the design, not the bring-up. + +## Execution contract + +- Prefill runs in BF16 on a separate chip and produces the first token. +- The BF16 prompt cache crosses the chip boundary unchanged. +- Decode admission quantizes K/V once into the selected PackedKV format. +- New `q_len=1` K/V entries are quantized directly into that format. +- Admission time and energy belong to TTFT; cached decode contributes TPOT. +- Embeddings remain BF16. The headline path executes the BF16 LM head as a + calibrated reserved endpoint on the same prefill chip after each decode-chip + final RMSNorm; no third device is introduced. + +The evaluator uses an immutable BF16 prefill artifact and a content-addressed +admission catalog. One KV format is deterministically recomputed and verified +at a time, then released before the next format, so the approximately 149 GiB +Qwen logical admission write volume is not retained simultaneously. Screening +and refinement report `post_handoff_greedy_conditioned_nll`: the +prefill-greedy handoff token is unscored, and later dataset tokens use cached +`q_len=1` teacher forcing with exact cache positions and one-entry growth. It +is not standard WikiText-2 perplexity. Cache-free perplexity is not part of +this path. + +## Canonical search + +The main block size is 8, the native RTL MX block. W, A, and shared K/V each +use: + +```text +MXINT2 MXINT4 MXINT8 E1M2 E2M1 E3M4 E4M3 E5M2 +``` + +Every W/A/KV point is evaluated with: + +```text +FP_E3M2 FP_E2M3 FP_E6M5 FP_E5M6 FP_E4M7 FP_E8M5 +``` + +The deterministic manifest contains: + +- 3,072 quantized profiles; +- 512 vector-BF16 controls; and +- one split-execution BF16 software reference. + +The 3,585 IDs are enumerated exactly. Optuna is not used for this numerical +grid. Blocks 16 and 32 are reserved for a selected-profile numerical +sensitivity study and are not deployment candidates for the native block-8 +datapath. Split K/V precision is confined to promoted-profile refinement. + +Accuracy is evaluated once for every precision profile and model, while the +hardware search evaluates the complete precision-by-hardware cross-product. +Final selection is joint under the accuracy, area, and HBM constraints; there +is no staged partial-objective decomposition of the canonical search. + +Static datapath legality admits 574 rows to hardware pricing: 492 quantized +profiles and 82 vector-BF16 controls. The complete structural spaces contain +1,413,216 Qwen and 1,848,096 Llama candidates, giving 811,185,984 and +1,060,807,104 raw profile-candidate pairs. Exact compiler geometry admits +79,488 Qwen and 272,160 Llama candidates. The factorized artifact stores +476,928 and 1,632,960 physical-cost evaluation rows, respectively, plus the +ordered memberships required to reconstruct every eligible conceptual join; +it never materializes the raw cross-product. The scheduler compresses this +work only by exact physical-cost equivalence, prices each equivalence class +once, and joins the result back to every accuracy row. Its +provenance records the raw, hard-accuracy-gated, hard-resource-gated, +simulator-priced, and joined counts. It performs no performance, bandwidth +demand, memory-bound, or objective-based pruning. + +## Workflow + +Four protocols govern the study, each held separately from the code: + +- the **exhaustive sweep** protocol defines immutable inputs, the measured pilot, + state-isolation checks, runtime rebinding, homogeneous-device sharding, and + launch gates; +- the **stack validation** protocol defines compiler, emulator, RTL, and DC + evidence; +- the **precision refinement** protocol defines GPTQ/Erry, selective rotation, + split K/V variants, and doomed-profile handling; and +- the **final benchmarks** protocol defines the four-configuration WikiText-2, + IFEval, and GSM8K study, with the complete 4K/8K/16K/32K RULER set enabled + only when long-context budget is available. + +The full sweep must not start until its measured preflight report passes. +Structural capability alone cannot become a deployment claim. + +Both model configurations require explicit single-device placement, BF16, +`sm_100`, minimum package versions, and declarative resource requirements. +Planning includes the full immutable manifest, run plan, provenance, logical +write volume, concurrency-bounded peak footprint, device requirement, and the +42-hour ceiling. Preflight reports only the machine on which it actually runs; +it does not substitute or project a remote host. It also does not invent a +wall-clock projection before the first completed measured profile; the runner +records that first duration and updates the ETA continuously. + +The deterministic, write-free launch checks are: + +```bash +python -m decode_dse.software.sweep stage plan \ + --config decode_dse/configs/llama3_1_8b.json \ + --output-dir /data/plena/llama3_1_8b \ + --device-label b200 --dry-run + +python -m decode_dse.software.sweep stage plan \ + --config decode_dse/configs/qwen3_32b.json \ + --output-dir /data/plena/qwen3_32b \ + --device-label b200 --dry-run +``` + +Remove `--dry-run` only on a host that passes every reported package, model, +tokenizer, simulator, GPU, BF16, host-memory, and workspace-capacity gate. +Persisted (non-dry-run) planning additionally requires `--prompt-manifest`, +produced by `sweep inputs samples`. The complete execution-host bring-up — +repository layout, model/dataset staging, evidence-artifact production +(`decode_timing_evidence.json`, the stack-validity stage reports, and the +measured BF16 head service), and the full per-model launch order — is +documented in `decode_dse/docs/server_setup.md`. + +The hardware evaluator optionally accepts `--handoff-artifact` for the E1 +prefill-to-decode schedule analysis. That JSON must bind the pinned model and +reference workload; provide BF16 prefill latency and energy evidence for every +searched batch; and state admission, readiness, idle/stall power, direct-link, +and host-link inputs explicitly. A requested missing or mismatched artifact +fails before the study starts. Omitting it leaves steady-state decode ranking +unchanged; supplying it adds fully-pipelined, back-pressure, and host-buffered +TTFT, utilization, energy, and balanced prefill:decode chip-ratio results to +each hardware row without changing decode feasibility or Pareto ranking. + +The principal modules are: + +```text +profiles.py canonical precision profiles and the grid +legality.py profile legality and PackedKV capability +manifest.py exact manifest and restart journal +simulator_bridge.py analytic decode performance model +plots.py publication figures + +software/precision_bindings.py MASE phase-split quantization bindings +software/decode_evaluator.py cached one-token decode evaluator +software/cached_decode.py teacher-forced cached decode primitives +software/token_samples.py sweep and refinement token sample bundles +software/cache_artifacts.py BF16 prefill and admitted-cache artifacts +software/sweep_plan.py run plan, manifests, shared executor context +software/sweep_runner.py restartable per-profile sweep execution +software/sweep.py inputs / stage / shards / pipeline commands +software/preflight.py check / evidence commands +software/refinement_schedule.py refinement schedule and shard plans +software/refinement_runner.py prepare / run / launch / merge commands +software/refinement_evaluator.py sealed GPTQ, clipping, and rotation banks +software/benchmark_runner.py manifest / configuration / benchmark commands +software/benchmark_evaluator.py benchmark suites over split-cached decode +software/gpu_baseline.py measured BF16 GPU cached one-token baseline +software/block_size_sensitivity.py block 16/32 sensitivity for selected points +software/runtime_environment.py deterministic runtime policy and identity + +hardware/design_space.py exact enumeration and factorized artifacts +hardware/evaluation.py simulator and power evaluation +hardware/power_model.py calibrated area and energy ingestion +hardware/selection.py Pareto promotion and final selection +hardware/packedkv_claims.py capacity-first PackedKV claim gate +hardware/synthesis_anchor.py selected-candidate DC/SAIF evidence +hardware/admission_cost.py decode-cache admission cost and evidence +hardware/workload_events.py decode event counts for dense decoders +hardware/lm_head_service.py BF16 output-head service boundary +hardware/calibration.py holdout gates for area, power, timing +hardware/hbm_sensitivity.py post-selection HBM technology schedule +hardware/statistics.py deterministic statistical helpers +``` + +## Validity scope + +Software accuracy is portable across hardware candidates. Successful compiler, +emulator, and RTL evidence applies only to the exact measured geometry and +batch. Fitted power is rankable only inside its measured interpolation +envelope. Deployment requires a candidate-exact full-chip DC/SAIF anchor whose +activity is bound to the serving batch. Off-target success becomes unknown +until matching evidence exists; observed failures remain false. + +The PackedKV compiler emits a record for the exact synchronous equal-length +batch requested by the candidate. Compiler, emulator, RTL, and timing evidence +never carries over to a different batch, even when its geometry otherwise +matches. + +## Publication outcomes + +PackedKV is framed capacity first: + +1. physical KV bytes decrease; +2. the feasible serving batch increases; +3. capacity-limited throughput increases. + +A TPOT claim is optional and requires matching compiler, emulator, RTL, and +timing-overlap evidence. Reports keep the ideal-pipeline algorithmic limit +separate from the realized memory, serialization, or compute limit; neither is +rewritten to force a memory-bound result. + +Power and timing are rankable only after their holdout gates pass. BF16 is the +software accuracy reference; it is not represented as a PLENA BF16 matrix-chip +realization. + +The local BF16 LM head is an explicitly unrankable hardware sensitivity. A +locally quantized LM head is accuracy-only and cannot supply hardware costs or +enter deployment selection. + +HBM technology does not multiply the 3,585-point numerical manifest. Four +selected native-datapath hardware profiles receive a separate 20-point +HBM2/HBM2E/HBM3/HBM3E/HBM4 sensitivity schedule with geometry, batch, +interface-unit count, and chip count fixed. HBM4 is a conservative 11 Gb/s +lower-bound representation of Micron's stated greater-than-11 Gb/s 36 GB +device and is technology-peak sensitivity only. Cross-generation ranking is +disabled until each generation, pin rate, and interface-unit count has matching +measured bandwidth calibration. + +## Pipeline stages + +`sweep pipeline` is the publication launch path for each model study, run after +that model's plan and preflight evidence pass. It executes in this order: + +1. **Measured GPU baseline.** The immutable run plan lists every + cached-`q_len=1` BF16 baseline batch; the pipeline measures those batches on + its first visible GPU before any sweep work. +2. **Hardware-validation shards**, evaluated with compiler-trace timing. +3. **Joint source selection** over the verified partitions. +4. **Refinement**, following the declared protocol. +5. **Repricing** of the selected refined profiles. +6. **Publication figures**, rendered only once the above have completed. + +### Restartability + +Every command carries an immutable identity and a completion receipt. An +interrupted command is repeated; completed commands and complete directory trees +are checksum-verified and skipped. The contract, terminal rows, report and stage +receipt are all bound to the same sweep provenance. + +### The final deployment table + +It verifies and records the measured GPU report and stage receipt, the benchmark +contract and report, the exact refined hardware artifact, and the post-accuracy +selection. Evidence tiers appear side by side, and: + +- the throughput ratio stays **empty** until a same-tier measured PLENA result + exists; +- headline ratios accept measured evidence on both sides only; +- A100 and H100 peak-roofline values remain a separate labelled table and never + acquire a ratio. + +### Measured baseline energy + +Each successful baseline row also attempts board energy, measured over only the +synchronized post-warmup decode region. The NVML cumulative-energy counter is +preferred; a timestamped NVML board-power trace with trapezoidal integration is +the fallback. The report retains the raw counters or trace samples, the physical +GPU UUIDs, the sampling interval, tokens per joule, and EDP. + +If neither meter is supported, throughput remains valid and energy is reported +as explicitly unavailable. No analytic value may stand in for it in a measured +energy comparison. + +### The `external/` evidence boundary + +`publication_pipeline` declares every post-accuracy input, output and resource +choice. Three inputs live under the workspace `external/` boundary: + +1. **Passing decode timing evidence.** Currently produced at the emulator tier + (mode `emulator_serialized`), because RTL does not execute at the anchor + geometry; the artifact carries `evidence_tier: emulator` and every consumer + surfaces that tier. An RTL-tier artifact satisfies the same gate. +2. **The full-model independent-request compiler trace set.** +3. **The BF16 output-head service calibration.** + +The third cannot be produced by timing `lm_head` on the execution GPU. The +headline boundary places a dedicated BF16 endpoint on the prefill chip, so the +artifact requires measurements taken at that physical endpoint: repeated and +holdout logits, remote-link request and response timing, component dynamic +energy, and leakage. The first pipeline command validates it against the pinned +model and every searched batch, and fails before any GPU sweep work if it is +missing or incomplete. No component energy or link measurement is ever inferred +from GPU timing. + +### Refinement and benchmarks + +Refinement is partitioned into exactly four logical source shards from one +immutable master schedule, run in two waves across the two-GPU execution pool. +The merge accepts only complete, checksum-verified terminal coverage, and every +accuracy-selected refinement profile is repriced against the exact hardware +space before publication configurations are sealed. + +Before any GPU stage, the enabled publication path enumerates complete +WikiText-2, IFEval and GSM8K splits from their declared dataset ID, +configuration, immutable revision, split, expected row count and local cache +root. Datasets are read offline in two independent ways — Hugging Face offline +controls and direct prepared-Arrow reads — and the Hub-aware loader is never +called, so a missing snapshot fails rather than downloading or silently +switching revisions. Ordered item content and every local source file are +checksum-sealed into the benchmark manifest. + +RULER is either omitted entirely or supplied as the complete 4K/8K/16K/32K set; +a partial set is not accepted. Contract construction, benchmark execution, the +post-accuracy exact hardware join, and the fail-closed final selection receipt +all remain inside the same restartable pipeline. diff --git a/decode_dse/__init__.py b/decode_dse/__init__.py new file mode 100644 index 0000000..79de8bd --- /dev/null +++ b/decode_dse/__init__.py @@ -0,0 +1,34 @@ +"""Decode-only precision, accuracy, and hardware co-design interfaces.""" + +from decode_dse.legality import ( + CrossStackCapability, + PackedKVRuntimeTarget, + StackValidity, + constrain_stack_validity, + evaluate_profile_legality, + evaluate_stack_capability, +) +from decode_dse.profiles import DecodePrecisionProfile, enumerate_decode_profiles +from decode_dse.simulator_bridge import ( + CompilerTracePointDescriptor, + CompilerTraceRequestBinder, + DecodeMetrics, + DecodeSimulator, + Precision, +) + +__all__ = [ + "CrossStackCapability", + "CompilerTracePointDescriptor", + "CompilerTraceRequestBinder", + "DecodeMetrics", + "DecodePrecisionProfile", + "DecodeSimulator", + "PackedKVRuntimeTarget", + "Precision", + "StackValidity", + "constrain_stack_validity", + "enumerate_decode_profiles", + "evaluate_profile_legality", + "evaluate_stack_capability", +] diff --git a/decode_dse/configs/llama3_1_8b.json b/decode_dse/configs/llama3_1_8b.json new file mode 100644 index 0000000..7cf98e4 --- /dev/null +++ b/decode_dse/configs/llama3_1_8b.json @@ -0,0 +1,466 @@ +{ + "_comment": "Deterministic decode-only precision sweep for Llama-3.1-8B-Instruct.", + "model_name": "meta-llama/Llama-3.1-8B-Instruct", + "model_revision": "0e9e39f249a16976918f6564b8830bc894c89659", + "tokenizer_revision": "0e9e39f249a16976918f6564b8830bc894c89659", + "sim_model": "llama-3.1-8b", + "model_architecture": { + "hidden_size": 4096, + "intermediate_size": 14336, + "num_hidden_layers": 32, + "num_attention_heads": 32, + "num_key_value_heads": 8, + "head_dim": 128, + "vocab_size": 128256, + "tie_word_embeddings": false, + "attention_bias": false, + "use_qk_norm": false + }, + "model_placement": { + "policy": "single_device", + "device_count": 1, + "automatic_device_map": false + }, + "runtime_requirements": { + "compute_capability": "sm_100", + "minimum_package_versions": { + "torch": "2.7.0", + "transformers": "4.51.0", + "datasets": "3.0.0", + "numpy": "1.26.0", + "nvidia-ml-py": "13.0.0" + } + }, + "device": "cuda:0", + "dtype": "bfloat16", + "hf_cache_dir": "/data/models", + "local_files_only": true, + "trust_remote_code": false, + "evaluation_data": { + "dataset_name": "Salesforce/wikitext", + "dataset_config": "wikitext-2-raw-v1", + "dataset_revision": "b08601e04326c79dfdd32d625aee71d232d685c3", + "split": "validation", + "text_column": "text", + "document_separator": "\n\n", + "cache_dir": "/data/datasets" + }, + "executor": { + "sample_bundle": "workspace://inputs/llama3_1_8b_samples.json", + "prefill_artifact_root": "workspace://artifacts/prefill_bf16", + "admission_artifact_root": "workspace://artifacts/decode_admission", + "stack_validity_manifest": "workspace://stack_validity.json", + "layout_id": "packed-gqa-mlen1024-block8-native-encoding", + "mase_src": "../mase/src", + "decode_microbatch_size": { + "numerical_screen": 16, + "hardware_validation": 8 + }, + "max_cpu_cache_gib": 24, + "artifact_policy": "content_addressed_recompute_per_format", + "artifact_space_safety_factor": 1.05, + "artifact_space_reserve_gib": 8, + "serialize_weight_bank_builds": true + }, + "phase_contract": { + "prefill_precision": "BF16", + "prefill_kv_precision": "BF16", + "decode_query_length": 1, + "first_token_owner": "prefill", + "decode_kv_admission": "quantize_once" + }, + "output_head_contract": { + "headline_location": "external_bf16_service", + "headline_precision": "BF16", + "local_bf16": "sensitivity_unrankable", + "local_low_precision": "accuracy_only" + }, + "reference_workload": { + "batch": 4, + "input_seq": 512, + "output_seq": 3072, + "hbm_gen": "HBM2", + "hbm_channels": 32 + }, + "search": { + "weight_w": [ + "MXINT2", + "MXINT4", + "MXINT8", + "E1M2", + "E2M1", + "E3M4", + "E4M3", + "E5M2" + ], + "kv": [ + "MXINT2", + "MXINT4", + "MXINT8", + "E1M2", + "E2M1", + "E3M4", + "E4M3", + "E5M2" + ], + "act_w": [ + "MXINT2", + "MXINT4", + "MXINT8", + "E1M2", + "E2M1", + "E3M4", + "E4M3", + "E5M2" + ], + "block": [ + 8 + ], + "mixed_weight": false, + "front_fp_setting": [ + [ + 3, + 2 + ], + [ + 2, + 3 + ], + [ + 6, + 5 + ], + [ + 5, + 6 + ], + [ + 4, + 7 + ], + [ + 8, + 5 + ] + ], + "vector_fp": [ + "FP_E3M2", + "FP_E2M3", + "FP_E6M5", + "FP_E5M6", + "FP_E4M7", + "FP_E8M5" + ], + "include_vector_bf16_controls": true, + "include_bf16_reference": true, + "expected_quantized_profiles": 3072, + "expected_vector_bf16_controls": 512, + "expected_total_profiles": 3585, + "fp_setting_attention": true, + "quant_attn_matmuls": true + }, + "software_search": "deterministic_exhaustive", + "search_budget": 3585, + "sampler": "deterministic_grid", + "use_rotation": false, + "fp_ppl_tol": 0.01, + "gptq": { + "nsamples": 64, + "seqlen": 1024, + "cali_batch_size": 8 + }, + "refinement": { + "schema_version": "decode-refinement-config/v1", + "sample_bundle": "workspace://refinement/inputs/samples.json", + "prefill_artifact_root": "workspace://refinement/artifacts/prefill_bf16", + "admission_artifact_root": "workspace://refinement/artifacts/decode_admission", + "calibration_artifact": "workspace://refinement/inputs/gptq_calibration.pt", + "calibration_receipt": "workspace://refinement/inputs/gptq_calibration.json", + "checkpoint_root": "workspace://refinement/checkpoints", + "calibration_samples": 128, + "calibration_sequence_length": 2048, + "calibration_batch_size": 8, + "calibration_seed": 20260725, + "sample_selection_seed": 20260725, + "gpu_min_free_mb": 36000, + "max_cpu_cache_gib": 24, + "calibration_data": { + "dataset_name": "Salesforce/wikitext", + "dataset_config": "wikitext-2-raw-v1", + "dataset_revision": "b08601e04326c79dfdd32d625aee71d232d685c3", + "split": "train", + "text_column": "text", + "document_separator": "\n\n", + "cache_dir": "/data/datasets" + } + }, + "publication": { + "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", + "chat_template_asset": "decode_dse/configs/publication_chat_template_llama3_1_8b.json", + "benchmark_datasets": { + "wikitext2": { + "dataset_name": "Salesforce/wikitext", + "dataset_config": "wikitext-2-raw-v1", + "dataset_revision": "b08601e04326c79dfdd32d625aee71d232d685c3", + "split": "test", + "cache_dir": "/data/datasets", + "content_columns": [ + "text" + ], + "id_column": null, + "source_item_count": 4358 + }, + "ifeval": { + "dataset_name": "google/IFEval", + "dataset_config": "default", + "dataset_revision": "39ed06ce3906b51290c6e95b7c697e928c8a7b00", + "split": "train", + "cache_dir": "/data/datasets", + "content_columns": [ + "key", + "prompt", + "instruction_id_list", + "kwargs" + ], + "id_column": "key", + "source_item_count": 541 + }, + "gsm8k": { + "dataset_name": "openai/gsm8k", + "dataset_config": "main", + "dataset_revision": "e53f048856ff4f594e959d75785d2c2d37b678ee", + "split": "test", + "cache_dir": "/data/datasets", + "content_columns": [ + "question", + "answer" + ], + "id_column": null, + "source_item_count": 1319 + } + }, + "thinking_mode": "disabled", + "enable_thinking": false, + "greedy": true, + "temperature": 0.0, + "token_budgets": { + "wikitext2": 32768, + "ifeval": 2048, + "gsm8k": 2048, + "ruler": 512 + }, + "output_head_location": "external_bf16_service", + "output_head_precision": "BF16" + }, + "eval_ppl_nsamples": 32, + "eval_ppl_seqlen": 2048, + "tasks": [ + "gsm8k", + "ifeval" + ], + "task_batch_size": 8, + "front_size": 6, + "hardware_space": { + "MLEN": [ + 16, + 32, + 64, + 128, + 256, + 512, + 1024, + 2048, + 4096 + ], + "BLEN": [ + 2, + 4, + 8, + 16, + 32, + 64 + ], + "HLEN": [ + 16, + 32, + 64, + 128 + ], + "BATCH": [ + 1, + 4, + 8, + 16, + 32, + 64, + 128, + 256 + ], + "HBM_CHANNELS": [ + 8 + ], + "HBM_GENERATION": "HBM2", + "CHIP_COUNT": [ + 1, + 2, + 4, + 8, + 16 + ], + "TP": [ + 1, + 2, + 4, + 8, + 16 + ], + "KVP": [ + 1, + 2, + 4, + 8, + 16 + ], + "LINK_PORTS": [ + 1, + 2, + 4 + ], + "SRAM_POLICY": [ + "streaming", + "projection_resident", + "kv_resident_25", + "kv_resident_50", + "kv_resident_75", + "kv_resident_100" + ], + "KV_HEAD_REUSE": [ + false, + true + ], + "DRAIN_OVERLAPPED": [ + false, + true + ], + "FP_SRAM_DEPTH": 512, + "RESOURCE_BUDGET": { + "reference_system": "A100x4", + "aggregate_area_limit_mm2": 3634.4, + "aggregate_hbm_capacity_limit_bytes": 320000000000, + "aggregate_hbm_bandwidth_limit_bytes_per_s": 8156000000000.0 + } + }, + "hbm_sensitivity": { + "schema_version": "decode-hbm-sensitivity/v2", + "source_profile_count": 4, + "generations": [ + "HBM2", + "HBM2E", + "HBM3", + "HBM3E", + "HBM4" + ], + "preserve_geometry_batch_channels": true, + "cross_generation_ranking": false + }, + "gpu_baseline": { + "attention_implementation": "sdpa", + "warmup_steps": 16, + "measured_steps": 128, + "repetitions": 3, + "batch_sizes": [ + 1, + 2, + 4, + 8, + 16, + 32 + ], + "precision": "BF16", + "q_len": 1, + "first_gpu_only": true, + "energy_meter_priority": [ + "nvml_total_energy_counter", + "nvml_power_trace_trapezoidal" + ], + "power_trace_sample_interval_ms": 10 + }, + "publication_pipeline": { + "schema_version": "decode-publication-pipeline", + "artifacts": { + "admission_receipt": "workspace://admission_preparation.json", + "timing_evidence": "workspace://external/decode_timing_evidence.json", + "compiler_trace_artifacts": "workspace://external/compiler_trace_artifacts.json", + "request_memory_calibration": "simulator://analytic_models/disagg_serve/calibration_dma_requests.csv", + "head_service_calibration": "workspace://external/bf16_output_head_service.json", + "handoff_artifact": null, + "power_calibration": null, + "area_config": null, + "exact_dc_anchors": null, + "hardware_study": "workspace://hardware/exact_hardware_study.jsonl", + "refined_hardware_study": "workspace://hardware/refined_hardware_study.jsonl", + "refinement_validity": null, + "refinement_schedule": "workspace://refinement/schedule.json", + "refinement_promotion": "workspace://refinement/source_selection.json", + "refinement_results": "workspace://refinement/results", + "publication_configurations": "workspace://publication/inputs/configurations.json", + "publication_benchmarks": "workspace://publication/inputs/benchmarks.json", + "publication_chat_template": "workspace://publication/inputs/chat_template.json", + "publication_contract": "workspace://publication/contract.json", + "publication_results": "workspace://publication/results", + "final_selection": "workspace://publication/final_selection.json", + "packedkv_evidence": null, + "decode_analysis": null, + "figures": "workspace://publication/figures" + }, + "resources": { + "stride": 1, + "runtime_hbm_reserve_bytes": 8589934592, + "rtl_source_tree_sha256": null, + "refinement_enabled": true, + "refinement_execution": "four_logical_shards", + "refinement_decode_microbatch_size": 8, + "bootstrap_replicates": 2000, + "publication_enabled": true, + "publication_executor": "decode_dse.software.benchmark_evaluator:create_executor", + "publication_timing_tier": "stage_calibrated_analytic", + "figure_formats": [ + "png", + "pdf", + "svg" + ] + } + }, + "n_chips": 1, + "baseline_hbm_channels": 32, + "seed": 0, + "scratch_dir": "workspace://scratch", + "gpu_min_free_mb": 24000, + "max_parallel_points": 4, + "compiler_trace_preflight": { + "schema_version": "decode-compiler-trace-preflight/v1", + "max_unique_family_artifacts": 64, + "max_unique_lowering_instantiations": 100000, + "max_lazy_trace_instantiations": 25000, + "max_projected_trace_bytes": 68719476736, + "max_context_timing_resolutions": 10000000, + "max_joined_identities": 10000000, + "max_joined_bytes": 68719476736, + "max_digest_updates": 20000000, + "projected_joined_row_bytes": 4096, + "digest_updates_per_joined_identity": 2 + }, + "doomed_ppl_abs": 100.0, + "doomed_ppl_ratio": 3.0, + "reference_chip": { + "MLEN": 1024, + "BLEN": 8, + "VLEN": 1024, + "HLEN": 128 + }, + "bw_model": "calibrated", + "runtime": { + "persistent": true, + "max_attempts": 3 + } +} diff --git a/decode_dse/configs/publication_chat_template.json b/decode_dse/configs/publication_chat_template.json new file mode 100644 index 0000000..169dab8 --- /dev/null +++ b/decode_dse/configs/publication_chat_template.json @@ -0,0 +1,9 @@ +{ + "schema_version": "decode-chat-template", + "model_name": "Qwen/Qwen3-32B", + "model_revision": "9216db5781bf21249d130ec9da846c4624c16137", + "tokenizer_revision": "9216db5781bf21249d130ec9da846c4624c16137", + "enable_thinking": false, + "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8", + "chat_template": "{%- if tools %}\n {{- '<|im_start|>system\\n' }}\n {%- if messages[0].role == 'system' %}\n {{- messages[0].content + '\\n\\n' }}\n {%- endif %}\n {{- \"# Tools\\n\\nYou may call one or more functions to assist with the user query.\\n\\nYou are provided with function signatures within XML tags:\\n\" }}\n {%- for tool in tools %}\n {{- \"\\n\" }}\n {{- tool | tojson }}\n {%- endfor %}\n {{- \"\\n\\n\\nFor each function call, return a json object with function name and arguments within XML tags:\\n\\n{\\\"name\\\": , \\\"arguments\\\": }\\n<|im_end|>\\n\" }}\n{%- else %}\n {%- if messages[0].role == 'system' %}\n {{- '<|im_start|>system\\n' + messages[0].content + '<|im_end|>\\n' }}\n {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n {%- set index = (messages|length - 1) - loop.index0 %}\n {%- if ns.multi_step_tool and message.role == \"user\" and message.content is string and not(message.content.startswith('') and message.content.endswith('')) %}\n {%- set ns.multi_step_tool = false %}\n {%- set ns.last_query_index = index %}\n {%- endif %}\n{%- endfor %}\n{%- for message in messages %}\n {%- if message.content is string %}\n {%- set content = message.content %}\n {%- else %}\n {%- set content = '' %}\n {%- endif %}\n {%- if (message.role == \"user\") or (message.role == \"system\" and not loop.first) %}\n {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>' + '\\n' }}\n {%- elif message.role == \"assistant\" %}\n {%- set reasoning_content = '' %}\n {%- if message.reasoning_content is string %}\n {%- set reasoning_content = message.reasoning_content %}\n {%- else %}\n {%- if '' in content %}\n {%- set reasoning_content = content.split('')[0].rstrip('\\n').split('')[-1].lstrip('\\n') %}\n {%- set content = content.split('')[-1].lstrip('\\n') %}\n {%- endif %}\n {%- endif %}\n {%- if loop.index0 > ns.last_query_index %}\n {%- if loop.last or (not loop.last and reasoning_content) %}\n {{- '<|im_start|>' + message.role + '\\n\\n' + reasoning_content.strip('\\n') + '\\n\\n\\n' + content.lstrip('\\n') }}\n {%- else %}\n {{- '<|im_start|>' + message.role + '\\n' + content }}\n {%- endif %}\n {%- else %}\n {{- '<|im_start|>' + message.role + '\\n' + content }}\n {%- endif %}\n {%- if message.tool_calls %}\n {%- for tool_call in message.tool_calls %}\n {%- if (loop.first and content) or (not loop.first) %}\n {{- '\\n' }}\n {%- endif %}\n {%- if tool_call.function %}\n {%- set tool_call = tool_call.function %}\n {%- endif %}\n {{- '\\n{\"name\": \"' }}\n {{- tool_call.name }}\n {{- '\", \"arguments\": ' }}\n {%- if tool_call.arguments is string %}\n {{- tool_call.arguments }}\n {%- else %}\n {{- tool_call.arguments | tojson }}\n {%- endif %}\n {{- '}\\n' }}\n {%- endfor %}\n {%- endif %}\n {{- '<|im_end|>\\n' }}\n {%- elif message.role == \"tool\" %}\n {%- if loop.first or (messages[loop.index0 - 1].role != \"tool\") %}\n {{- '<|im_start|>user' }}\n {%- endif %}\n {{- '\\n\\n' }}\n {{- content }}\n {{- '\\n' }}\n {%- if loop.last or (messages[loop.index0 + 1].role != \"tool\") %}\n {{- '<|im_end|>\\n' }}\n {%- endif %}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|im_start|>assistant\\n' }}\n {%- if enable_thinking is defined and enable_thinking is false %}\n {{- '\\n\\n\\n\\n' }}\n {%- endif %}\n{%- endif %}" +} diff --git a/decode_dse/configs/publication_chat_template_llama3_1_8b.json b/decode_dse/configs/publication_chat_template_llama3_1_8b.json new file mode 100644 index 0000000..0dbb2ef --- /dev/null +++ b/decode_dse/configs/publication_chat_template_llama3_1_8b.json @@ -0,0 +1,9 @@ +{ + "chat_template": "{{- bos_token }}\n{%- if custom_tools is defined %}\n {%- set tools = custom_tools %}\n{%- endif %}\n{%- if not tools_in_user_message is defined %}\n {%- set tools_in_user_message = true %}\n{%- endif %}\n{%- if not date_string is defined %}\n {%- set date_string = \"26 Jul 2024\" %}\n{%- endif %}\n{%- if not tools is defined %}\n {%- set tools = none %}\n{%- endif %}\n\n{#- This block extracts the system message, so we can slot it into the right place. #}\n{%- if messages[0]['role'] == 'system' %}\n {%- set system_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n{%- else %}\n {%- set system_message = \"\" %}\n{%- endif %}\n\n{#- System message + builtin tools #}\n{{- \"<|start_header_id|>system<|end_header_id|>\\n\\n\" }}\n{%- if builtin_tools is defined or tools is not none %}\n {{- \"Environment: ipython\\n\" }}\n{%- endif %}\n{%- if builtin_tools is defined %}\n {{- \"Tools: \" + builtin_tools | reject('equalto', 'code_interpreter') | join(\", \") + \"\\n\\n\"}}\n{%- endif %}\n{{- \"Cutting Knowledge Date: December 2023\\n\" }}\n{{- \"Today Date: \" + date_string + \"\\n\\n\" }}\n{%- if tools is not none and not tools_in_user_message %}\n {{- \"You have access to the following functions. To call a function, please respond with JSON for a function call.\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n{%- endif %}\n{{- system_message }}\n{{- \"<|eot_id|>\" }}\n\n{#- Custom tools are passed in a user message with some extra guidance #}\n{%- if tools_in_user_message and not tools is none %}\n {#- Extract the first user message so we can plug it in here #}\n {%- if messages | length != 0 %}\n {%- set first_user_message = messages[0]['content']|trim %}\n {%- set messages = messages[1:] %}\n {%- else %}\n {{- raise_exception(\"Cannot put tools in the first user message when there's no first user message!\") }}\n{%- endif %}\n {{- '<|start_header_id|>user<|end_header_id|>\\n\\n' -}}\n {{- \"Given the following functions, please respond with a JSON for a function call \" }}\n {{- \"with its proper arguments that best answers the given prompt.\\n\\n\" }}\n {{- 'Respond in the format {\"name\": function name, \"parameters\": dictionary of argument name and its value}.' }}\n {{- \"Do not use variables.\\n\\n\" }}\n {%- for t in tools %}\n {{- t | tojson(indent=4) }}\n {{- \"\\n\\n\" }}\n {%- endfor %}\n {{- first_user_message + \"<|eot_id|>\"}}\n{%- endif %}\n\n{%- for message in messages %}\n {%- if not (message.role == 'ipython' or message.role == 'tool' or 'tool_calls' in message) %}\n {{- '<|start_header_id|>' + message['role'] + '<|end_header_id|>\\n\\n'+ message['content'] | trim + '<|eot_id|>' }}\n {%- elif 'tool_calls' in message %}\n {%- if not message.tool_calls|length == 1 %}\n {{- raise_exception(\"This model only supports single tool-calls at once!\") }}\n {%- endif %}\n {%- set tool_call = message.tool_calls[0].function %}\n {%- if builtin_tools is defined and tool_call.name in builtin_tools %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- \"<|python_tag|>\" + tool_call.name + \".call(\" }}\n {%- for arg_name, arg_val in tool_call.arguments | items %}\n {{- arg_name + '=\"' + arg_val + '\"' }}\n {%- if not loop.last %}\n {{- \", \" }}\n {%- endif %}\n {%- endfor %}\n {{- \")\" }}\n {%- else %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' -}}\n {{- '{\"name\": \"' + tool_call.name + '\", ' }}\n {{- '\"parameters\": ' }}\n {{- tool_call.arguments | tojson }}\n {{- \"}\" }}\n {%- endif %}\n {%- if builtin_tools is defined %}\n {#- This means we're in ipython mode #}\n {{- \"<|eom_id|>\" }}\n {%- else %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n {%- elif message.role == \"tool\" or message.role == \"ipython\" %}\n {{- \"<|start_header_id|>ipython<|end_header_id|>\\n\\n\" }}\n {%- if message.content is mapping or message.content is iterable %}\n {{- message.content | tojson }}\n {%- else %}\n {{- message.content }}\n {%- endif %}\n {{- \"<|eot_id|>\" }}\n {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n {{- '<|start_header_id|>assistant<|end_header_id|>\\n\\n' }}\n{%- endif %}\n", + "chat_template_sha256": "e10ca381b1ccc5cf9db52e371f3b6651576caee0a630b452e2816b2d404d4b65", + "enable_thinking": false, + "model_name": "meta-llama/Llama-3.1-8B-Instruct", + "model_revision": "0e9e39f249a16976918f6564b8830bc894c89659", + "schema_version": "decode-chat-template", + "tokenizer_revision": "0e9e39f249a16976918f6564b8830bc894c89659" +} diff --git a/decode_dse/configs/qwen3_32b.json b/decode_dse/configs/qwen3_32b.json new file mode 100644 index 0000000..70fd217 --- /dev/null +++ b/decode_dse/configs/qwen3_32b.json @@ -0,0 +1,467 @@ +{ + "_comment": "Deterministic decode-only precision sweep for Qwen3-32B.", + "model_name": "Qwen/Qwen3-32B", + "model_revision": "9216db5781bf21249d130ec9da846c4624c16137", + "tokenizer_revision": "9216db5781bf21249d130ec9da846c4624c16137", + "sim_model": "qwen3-32b", + "model_architecture": { + "hidden_size": 5120, + "intermediate_size": 25600, + "num_hidden_layers": 64, + "num_attention_heads": 64, + "num_key_value_heads": 8, + "head_dim": 128, + "vocab_size": 151936, + "tie_word_embeddings": false, + "attention_bias": false, + "use_qk_norm": true + }, + "model_placement": { + "policy": "single_device", + "device_count": 1, + "automatic_device_map": false + }, + "runtime_requirements": { + "compute_capability": "sm_100", + "minimum_package_versions": { + "torch": "2.7.0", + "transformers": "4.51.0", + "datasets": "3.0.0", + "numpy": "1.26.0", + "nvidia-ml-py": "13.0.0" + } + }, + "device": "cuda:0", + "dtype": "bfloat16", + "hf_cache_dir": "/data/models", + "local_files_only": true, + "trust_remote_code": false, + "evaluation_data": { + "dataset_name": "Salesforce/wikitext", + "dataset_config": "wikitext-2-raw-v1", + "dataset_revision": "b08601e04326c79dfdd32d625aee71d232d685c3", + "split": "validation", + "text_column": "text", + "document_separator": "\n\n", + "cache_dir": "/data/datasets" + }, + "executor": { + "sample_bundle": "workspace://inputs/qwen3_32b_samples.json", + "prefill_artifact_root": "workspace://artifacts/prefill_bf16", + "admission_artifact_root": "workspace://artifacts/decode_admission", + "stack_validity_manifest": "workspace://stack_validity.json", + "layout_id": "packed-gqa-mlen1024-block8-native-encoding", + "mase_src": "../mase/src", + "decode_microbatch_size": { + "numerical_screen": 16, + "hardware_validation": 8 + }, + "max_cpu_cache_gib": 24, + "artifact_policy": "content_addressed_recompute_per_format", + "artifact_space_safety_factor": 1.05, + "artifact_space_reserve_gib": 8, + "serialize_weight_bank_builds": true + }, + "phase_contract": { + "prefill_precision": "BF16", + "prefill_kv_precision": "BF16", + "decode_query_length": 1, + "first_token_owner": "prefill", + "decode_kv_admission": "quantize_once" + }, + "output_head_contract": { + "headline_location": "external_bf16_service", + "headline_precision": "BF16", + "local_bf16": "sensitivity_unrankable", + "local_low_precision": "accuracy_only" + }, + "reference_workload": { + "batch": 4, + "input_seq": 512, + "output_seq": 3072, + "hbm_gen": "HBM2", + "hbm_channels": 32 + }, + "search": { + "weight_w": [ + "MXINT2", + "MXINT4", + "MXINT8", + "E1M2", + "E2M1", + "E3M4", + "E4M3", + "E5M2" + ], + "kv": [ + "MXINT2", + "MXINT4", + "MXINT8", + "E1M2", + "E2M1", + "E3M4", + "E4M3", + "E5M2" + ], + "act_w": [ + "MXINT2", + "MXINT4", + "MXINT8", + "E1M2", + "E2M1", + "E3M4", + "E4M3", + "E5M2" + ], + "block": [ + 8 + ], + "mixed_weight": false, + "front_fp_setting": [ + [ + 3, + 2 + ], + [ + 2, + 3 + ], + [ + 6, + 5 + ], + [ + 5, + 6 + ], + [ + 4, + 7 + ], + [ + 8, + 5 + ] + ], + "vector_fp": [ + "FP_E3M2", + "FP_E2M3", + "FP_E6M5", + "FP_E5M6", + "FP_E4M7", + "FP_E8M5" + ], + "include_vector_bf16_controls": true, + "include_bf16_reference": true, + "expected_quantized_profiles": 3072, + "expected_vector_bf16_controls": 512, + "expected_total_profiles": 3585, + "fp_setting_attention": true, + "quant_attn_matmuls": true + }, + "software_search": "deterministic_exhaustive", + "search_budget": 3585, + "sampler": "deterministic_grid", + "use_rotation": false, + "fp_ppl_tol": 0.01, + "gptq": { + "nsamples": 64, + "seqlen": 1024, + "cali_batch_size": 8 + }, + "refinement": { + "schema_version": "decode-refinement-config/v1", + "sample_bundle": "workspace://refinement/inputs/samples.json", + "prefill_artifact_root": "workspace://refinement/artifacts/prefill_bf16", + "admission_artifact_root": "workspace://refinement/artifacts/decode_admission", + "calibration_artifact": "workspace://refinement/inputs/gptq_calibration.pt", + "calibration_receipt": "workspace://refinement/inputs/gptq_calibration.json", + "checkpoint_root": "workspace://refinement/checkpoints", + "calibration_samples": 128, + "calibration_sequence_length": 2048, + "calibration_batch_size": 8, + "calibration_seed": 20260725, + "sample_selection_seed": 20260725, + "gpu_min_free_mb": 90000, + "max_cpu_cache_gib": 24, + "calibration_data": { + "dataset_name": "Salesforce/wikitext", + "dataset_config": "wikitext-2-raw-v1", + "dataset_revision": "b08601e04326c79dfdd32d625aee71d232d685c3", + "split": "train", + "text_column": "text", + "document_separator": "\n\n", + "cache_dir": "/data/datasets" + } + }, + "publication": { + "chat_template_asset": "decode_dse/configs/publication_chat_template.json", + "chat_template_sha256": "a55ee1b1660128b7098723e0abcd92caa0788061051c62d51cbe87d9cf1974d8", + "benchmark_datasets": { + "wikitext2": { + "dataset_name": "Salesforce/wikitext", + "dataset_config": "wikitext-2-raw-v1", + "dataset_revision": "b08601e04326c79dfdd32d625aee71d232d685c3", + "split": "test", + "cache_dir": "/data/datasets", + "content_columns": [ + "text" + ], + "id_column": null, + "source_item_count": 4358 + }, + "ifeval": { + "dataset_name": "google/IFEval", + "dataset_config": "default", + "dataset_revision": "39ed06ce3906b51290c6e95b7c697e928c8a7b00", + "split": "train", + "cache_dir": "/data/datasets", + "content_columns": [ + "key", + "prompt", + "instruction_id_list", + "kwargs" + ], + "id_column": "key", + "source_item_count": 541 + }, + "gsm8k": { + "dataset_name": "openai/gsm8k", + "dataset_config": "main", + "dataset_revision": "e53f048856ff4f594e959d75785d2c2d37b678ee", + "split": "test", + "cache_dir": "/data/datasets", + "content_columns": [ + "question", + "answer" + ], + "id_column": null, + "source_item_count": 1319 + } + }, + "thinking_mode": "disabled", + "enable_thinking": false, + "greedy": true, + "temperature": 0.0, + "token_budgets": { + "wikitext2": 32768, + "ifeval": 2048, + "gsm8k": 2048, + "ruler": 512 + }, + "output_head_location": "external_bf16_service", + "output_head_precision": "BF16" + }, + "eval_ppl_nsamples": 32, + "eval_ppl_seqlen": 2048, + "tasks": [ + "gsm8k", + "ifeval" + ], + "task_batch_size": 8, + "front_size": 6, + "hardware_space": { + "MLEN": [ + 16, + 32, + 64, + 128, + 256, + 512, + 1024, + 2048, + 4096 + ], + "BLEN": [ + 2, + 4, + 8, + 16, + 32, + 64 + ], + "HLEN": [ + 16, + 32, + 64, + 128 + ], + "BATCH": [ + 1, + 4, + 8, + 16, + 32, + 64, + 128, + 256 + ], + "HBM_CHANNELS": [ + 8 + ], + "HBM_GENERATION": "HBM2", + "CHIP_COUNT": [ + 1, + 2, + 4, + 8, + 16 + ], + "TP": [ + 1, + 2, + 4, + 8, + 16 + ], + "KVP": [ + 1, + 2, + 4, + 8, + 16 + ], + "LINK_PORTS": [ + 1, + 2, + 4 + ], + "SRAM_POLICY": [ + "streaming", + "projection_resident", + "kv_resident_25", + "kv_resident_50", + "kv_resident_75", + "kv_resident_100" + ], + "KV_HEAD_REUSE": [ + false, + true + ], + "DRAIN_OVERLAPPED": [ + false, + true + ], + "FP_SRAM_DEPTH": 512, + "RESOURCE_BUDGET": { + "reference_system": "A100x4", + "aggregate_area_limit_mm2": 3634.4, + "aggregate_hbm_capacity_limit_bytes": 320000000000, + "aggregate_hbm_bandwidth_limit_bytes_per_s": 8156000000000.0 + } + }, + "hbm_sensitivity": { + "schema_version": "decode-hbm-sensitivity/v2", + "source_profile_count": 4, + "generations": [ + "HBM2", + "HBM2E", + "HBM3", + "HBM3E", + "HBM4" + ], + "preserve_geometry_batch_channels": true, + "cross_generation_ranking": false + }, + "gpu_baseline": { + "attention_implementation": "sdpa", + "warmup_steps": 16, + "measured_steps": 128, + "repetitions": 3, + "batch_sizes": [ + 1, + 2, + 4, + 8, + 16, + 32 + ], + "precision": "BF16", + "q_len": 1, + "first_gpu_only": true, + "energy_meter_priority": [ + "nvml_total_energy_counter", + "nvml_power_trace_trapezoidal" + ], + "power_trace_sample_interval_ms": 10 + }, + "publication_pipeline": { + "schema_version": "decode-publication-pipeline", + "artifacts": { + "admission_receipt": "workspace://admission_preparation.json", + "timing_evidence": "workspace://external/decode_timing_evidence.json", + "compiler_trace_artifacts": "workspace://external/compiler_trace_artifacts.json", + "request_memory_calibration": "simulator://analytic_models/disagg_serve/calibration_dma_requests.csv", + "head_service_calibration": "workspace://external/bf16_output_head_service.json", + "handoff_artifact": null, + "power_calibration": null, + "area_config": null, + "exact_dc_anchors": null, + "hardware_study": "workspace://hardware/exact_hardware_study.jsonl", + "refined_hardware_study": "workspace://hardware/refined_hardware_study.jsonl", + "refinement_validity": null, + "refinement_schedule": "workspace://refinement/schedule.json", + "refinement_promotion": "workspace://refinement/source_selection.json", + "refinement_results": "workspace://refinement/results", + "publication_configurations": "workspace://publication/inputs/configurations.json", + "publication_benchmarks": "workspace://publication/inputs/benchmarks.json", + "publication_chat_template": "workspace://publication/inputs/chat_template.json", + "publication_contract": "workspace://publication/contract.json", + "publication_results": "workspace://publication/results", + "final_selection": "workspace://publication/final_selection.json", + "packedkv_evidence": null, + "decode_analysis": null, + "figures": "workspace://publication/figures" + }, + "resources": { + "stride": 1, + "runtime_hbm_reserve_bytes": 8589934592, + "rtl_source_tree_sha256": null, + "refinement_enabled": true, + "refinement_execution": "four_logical_shards", + "refinement_decode_microbatch_size": 8, + "bootstrap_replicates": 2000, + "publication_enabled": true, + "publication_executor": "decode_dse.software.benchmark_evaluator:create_executor", + "publication_timing_tier": "stage_calibrated_analytic", + "figure_formats": [ + "png", + "pdf", + "svg" + ] + } + }, + "n_chips": 1, + "baseline_hbm_channels": 32, + "seed": 0, + "scratch_dir": "workspace://scratch", + "_scratch_comment": "Cross-process weight-bank construction locks and refinement checkpoints use this path.", + "gpu_min_free_mb": 70000, + "max_parallel_points": 2, + "compiler_trace_preflight": { + "schema_version": "decode-compiler-trace-preflight/v1", + "max_unique_family_artifacts": 64, + "max_unique_lowering_instantiations": 100000, + "max_lazy_trace_instantiations": 25000, + "max_projected_trace_bytes": 68719476736, + "max_context_timing_resolutions": 10000000, + "max_joined_identities": 10000000, + "max_joined_bytes": 68719476736, + "max_digest_updates": 20000000, + "projected_joined_row_bytes": 4096, + "digest_updates_per_joined_identity": 2 + }, + "doomed_ppl_abs": 100.0, + "doomed_ppl_ratio": 3.0, + "reference_chip": { + "MLEN": 1024, + "BLEN": 8, + "VLEN": 1024, + "HLEN": 128 + }, + "bw_model": "calibrated", + "runtime": { + "persistent": true, + "max_attempts": 3 + } +} diff --git a/decode_dse/docs/server_setup.md b/decode_dse/docs/server_setup.md new file mode 100644 index 0000000..6470d44 --- /dev/null +++ b/decode_dse/docs/server_setup.md @@ -0,0 +1,66 @@ +# Execution-host bring-up + +## Repository layout + +The four repositories are cloned side by side; the sweep plan hashes +`PLENA_RTL/PLENA_Tools/plena_quant` as a quantizer source root and fails +closed without it, so `PLENA_RTL` must be present even though the sweep is +software-only. + +``` +/PLENA_Software branch sr1325-decode_dse +/PLENA_Simulator branch sr1325-dev (compiler submodule initialized) +/PLENA_RTL branch sr1325-dev (PLENA_Compiler submodule initialized) +/mase branch sr1325/decode-phase-quant +``` + +Byte-identity invariants must hold before any run: +`PLENA_RTL/PLENA_Compiler == PLENA_Simulator/compiler` and +`PLENA_RTL/PLENA_Tools == PLENA_Simulator/PLENA_Tools` +(`diff -qr … -x __pycache__ -x '*.pyc' -x .git -x .pytest_cache` is empty). + +## Environment + +All pipeline invocations go through `decode_dse/scripts/launch_pipeline.sh`, +which exports `PLENA_SIMULATOR_PATH` and prepends +`PLENA_Simulator/compiler` and `PLENA_Simulator/PLENA_Tools` to `PYTHONPATH`. +These exports are required: the compiler frontend imports `asm_templates` as a +top-level package, which the in-process `sys.path` injection does not cover. +Do not rely on ambient shell state. + +Models and datasets are staged under the paths pinned in +`decode_dse/configs/.json` (`cache_dir`, `model_revision`, dataset +revisions). The Llama tokenizer/model requires an authenticated Hugging Face +fetch at the pinned revision. + +## Evidence artifacts + +`workspace://external/` must contain, before the pipeline's evidence gate: + +| file | origin | +|---|---| +| `decode_timing_evidence.json` | copied from `PLENA_Simulator/analytic_models/performance/evidence/`; self-labelled emulator tier | +| `compiler_trace_artifacts.json` | written by the pipeline's first command (`sweep compiler-trace-artifacts`) | +| `bf16_output_head_service.json` | measured on this host by `decode_dse/hardware/measure_bf16_head_service.py`, one run per model, on two idle exclusively-held GPUs | + +The stack-validity stage reports and calibration artifacts ship inside the +repositories and the workspace; they are content-hash bound and must not be +regenerated unless their bound sources changed. + +## Per-model launch order + +``` +scripts/launch_pipeline.sh inputs samples --config --output-dir ... +scripts/launch_pipeline.sh stage plan --config --output-dir --dry-run +scripts/launch_pipeline.sh stage plan --config --output-dir --prompt-manifest /prompt_manifest.json +scripts/launch_pipeline.sh inputs prefill --config --output-dir ... +scripts/launch_pipeline.sh inputs admission --config --output-dir ... +# stage the three external artifacts (table above) +scripts/launch_pipeline.sh pipeline --config --output-dir --device-label