Welcoming Community Models && Using Model Spec v1 for New Models

[0xShug0]

Thank you to everyone contributing to audio.cpp. Your work is greatly appreciated!

Audio.cpp introduces a dedicated community_models area for community-contributed model ports. The review bar for community models is intentionally lighter than core model integrations, so contributors can share useful implementations earlier.

• RTF should be below 1.0.
• VRAM usage should stay stable across multiple requests. If memory needs to be optimized, use mem_saver to balance performance and VRAM instead of hiding leaks.
• Long-form generation should work correctly. The shared long-form test cases live in tools/audiocpp_cli/audiocpp_cli_longform_tts_clone_cases.json.
• Use existing framework modules and patterns and helpers as much as possible (CacheSlot, Text/Audio chukers, etc) so the models will benefit from future framework optimizations.
• I'm constantly refactoring the framework modules. If you need to touch existing framework modules, the changes must be additive and opt-in for the new model. The preferred approach is to separate the framework changes and the model implementation into different PRs. Alternatively, you can implement local variants with comments calling this out, and I'll promote them into the framework later.

Community models still run through the normal audio.cpp CLI and server paths, but the new layout makes ownership clearer: core framework models stay under models, while contributor-led ports live under community_models.

PS: If you plan to work on something, please open a draft PR early to help avoid duplicated work or merge conflicts, since I may already be sneakily working on the same area, or doing some framework refactor that could lead to significant merge conflicts.

Model Spec v1 Preview

Confucius4-TTS is being published early as the current example of the new schema-v1 model spec flow.

The goal of spec v1 is to make model_specs/<family>.json the single source of truth for model metadata, supported tasks, options, CLI/help surfaces, package layout, and future UI/model-manager behavior.

One immediate benefit: v1 models can use the generic spec-backed loader, so the old repetitive per-model loader.cpp/loader.h pattern is no longer needed. Confucius4-TTS now declares its loader factory next to the session and lets the framework derive metadata/options from the spec.

For new models, put the new spec in the model_specs directly and add "schema_version": 1 so the new spec system can recognize and use it. No need to create two specs under model_specs and model_specs_v1. The specs under model_specs_v1 are the expected shape of the old models. The migration will be done slowly.

model_manager.py (-> model_manager_deprecated.py) is deprecated. model_manager_v2.py will read model links from specs directly. Safetensor conversion is no longer supported.

If you create GGUF files (which are preferred over safetensors) please make them self-contained by embedding the model spec. Safetensors source files are optional.

Naming Rules That Are Easy To Miss

One pain is many models have different option names pointing to the same concept (e.g., top_k vs topK vs top_K). Please use existing normalized option names whenever the concept already exists. You can refer to the specs under model_specs_v1 for examples and the naming conventions. Please do not copy Python/CLI/internal names directly into the spec. Some option keys will be validated by the framework so you don't need to write code to validate it locally.

Old/ad-hoc style	Prefer
max_new_tokens	max_tokens
duration_seconds, duration, gen_duration	duration_sec
reference_duration_seconds, ref_duration	reference_duration_sec
target_duration_seconds	target_duration_sec
max_chunk_duration	audio_chunk_threshold_sec
target_chunk_duration	audio_chunk_duration_sec
cross_fade_duration	cross_fade_duration_sec
f0_up_key, pitch_shift_steps	semitone_shift
file_index	retrieval_index_path
index_rate	retrieval_blend
voice, speaker when selecting preset	voice_id
voice_ref, speaker_wav for reference audio	target_voice
*_seconds	*_sec
*_milliseconds	*_ms
file-like inputs	*_path

Session options are local in JSON but public as <family>.<name> by the framework:

{ "name": "weight_type", "type": "enum" }

becomes:

--session-option confucius4_tts.weight_type=q8_0

Request options should usually stay bare:

--option temperature=0.7
--option target_voice=ref.wav

Current Spec System

Capability	Current behavior
Validate default type/range in JSON	Yes
Expose default in model inspection / CLI metadata	Yes
Use default to build in-memory contract	Yes, as metadata
Automatically inject default into request/session options	No
Normal request reloads spec from disk to get defaults	No

Backward Compatibility

For already released models, keep the user path stable while moving the spec forward.

Case	Suggestion
Existing released model has old option names	Keep compatibility in the model/session parser or compatibility layer.
New unreleased model	Use clean spec names directly; no legacy aliases needed.
Embedded GGUF spec is older than repo spec	Treat embedded spec as the base contract; newer runtime code may add/understand newer options without breaking old packages.
Option exists in production but not old CLI	Include it if the model really supports it. Spec v1 should describe model capability, not old CLI limitations.
Old Python/internal name differs from normalized name	Normalize the spec name and map internally only when needed.
Default behavior exists	Put the default in the spec and describe it.
Model does not support the option yet	Do not claim it in the spec.

Normal requests do not reload the JSON spec from disk. The spec-backed loader builds an in-memory contract when the model is loaded, and sessions validate against that contract.