[Feature] ModelScope download - #450
Merged
0xShug0 merged 4 commits intoSep 4, 2026
Merged
Conversation
…e manager Add modelscope_snapshot as a download kind alongside huggingface_snapshot: - schema: accept modelscope_snapshot (repo required, revision optional) - manager: ms_url() + AUDIOCPP_MS_BASE_URL override (default https://www.modelscope.cn); kind-aware revision default (master for ModelScope, main for Hugging Face) - remote info via the ModelScope file-list API (per-file Size + Sha256), cached per repo+revision; sha256 doubles as the manifest etag so size checks and inventory version comparison work unchanged; falls back to HEAD + X-Linked-Etag when the listing is unavailable - downloads reuse the existing HTTP layer (302 CDN redirects already followed); auth failure message no longer hardcodes Hugging Face - tests: ModelScope endpoints in the native manager fixture, new package_manager_modelscope_test, schema kind coverage - docs: download sources section in model_manager.md and the new kind in maintainers/model_specs.md
…on manager Mirror the native package manager's ModelScope support in tools/model_manager_v2.py: - ms_endpoint() honors AUDIOCPP_MS_BASE_URL (default https://www.modelscope.cn); ms_url() builds /models/{repo}/resolve/{revision}/{path} - kind-aware revision default via package_revision(): master for modelscope_snapshot, main for huggingface_snapshot - remote info comes from the ModelScope file-list API (per-file Size + Sha256), cached per endpoint+repo+revision; sha256 doubles as the manifest etag so size checks and version_state comparison work unchanged; falls back to HEAD + X-Linked-Etag with unknown size when the listing is unavailable - downloads reuse the existing urlopen flow (302 CDN redirects already followed); 401/403 message is provider-appropriate; gated/HF-token handling stays Hugging Face-only - docs: note that both the native manager and model_manager_v2 support both download sources
…erride Allow redirecting any spec package to ModelScope at install/sizes time without editing model_specs: --source modelscope rewrites the download kind, --source-repo names the ModelScope repo (defaults to the spec's repo), and an unset or 'main' revision becomes 'master' while explicit revisions pass through. Errors under an override hint at --source-repo. Document the switch and the cross-source etag caveat.
Owner
|
@IIIIIllllIIIIIlllll The PR builds one shared request-header path and reuses it for both Hugging Face and ModelScope. Since that helper can attach I’d suggest making request headers provider-scoped: Hugging Face requests may attach Hugging Face auth, while ModelScope requests should not receive HF auth and should use a separate ModelScope auth path. |
Request headers were built by one shared helper that attached the Hugging Face token (HF_TOKEN / HUGGING_FACE_HUB_TOKEN / cached HF token) to every request, so ModelScope requests could leak the HF credential to modelscope.cn or an AUDIOCPP_MS_BASE_URL mirror. Both the native manager and tools/model_manager_v2.py now pick auth by provider: Hugging Face requests may carry the HF token, ModelScope requests carry only the new AUDIOCPP_MS_TOKEN (optional). The fixture server gained --hf-token/--ms-token guards that reject HF credentials on ModelScope paths, and the ModelScope package test now runs with both tokens set to prove the isolation.
Contributor
Author
|
@0xShug0 Fixed |
Owner
|
@IIIIIllllIIIIIlllll Merged! Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Since the repository on ModelScope has already been set up, is it necessary to add a feature to download from here? I'll go ahead and implement it first and put it here. It doesn't really make sense for two people to maintain separate repositories, and I'm not sure if it's necessary either. If all else fails, we could also add a note in the documentation to guide users who need it to that repository.
https://www.modelscope.cn/models/HereIsMark/audio.cpp-gguf
Sure, here's the English version:
1. On-demand switch (most common, no config changes)
Use the Python tool to pull any package from ModelScope:
Simple rules:
--source= exactly the same behavior as before (Hugging Face)mainrevision in the spec → translated to ModelScope's default branchmaster; any other explicit revision (e.g.v1.2) passes through unchanged--source-repoalone (without--source modelscope) is rejected--sourceyou installed with — etags are source-specific, so cross-source checks may falsely report an update2. Declared in the spec (for maintainers)
Set
download.kindtomodelscope_snapshotin_specs/*.json; both the native C++ manager and the Python tool understand it:revisionis optional (defaults tomaster).3. Mirror/test endpoints (rarely needed)
AUDIOCPP_MS_BASE_URL(defaulthttps://www.modelscope.cn), honored by both native and PythonAUDIOCPP_HF_BASE_URL, Python uses the standardHF_ENDPOINTTL;DR: when Hugging Face is unreachable, just append
--source modelscopesource-repo HereIsMark/audio.cpp-ggufto your install command.