fix(parser): decode HTTP-fetched specs as UTF-8, not transport charset

[sophie-jentic]

Problem

Importing a valid OpenAPI spec by URL fails with:

Failed to parse document with backend 'PyYAMLParserBackend':
unacceptable character #x0080: special characters are not allowed in "", position ...

whenever the origin server serves the spec as Content-Type: text/yaml without a charset parameter (e.g. docs.livetennisapi.com), and the spec contains any non-ASCII character (an em-dash, curly quote, accent, emoji…).

Pointing at a mirror that sends charset=utf-8 (e.g. raw.githubusercontent.com) "works", which is why this looked host-specific — but the byte content is identical; only the declared charset differs.

Root cause

load_uri() read resp.text, which lets requests choose the codec from the HTTP Content-Type. For a text/* response with no charset, requests follows the legacy RFC 2616 default and decodes as ISO-8859-1. That mis-decodes UTF-8 multibyte characters: an em-dash (—, UTF-8 E2 80 94) becomes three Latin-1 code points, one of which is U+0080 — the exact character PyYAML's reader rejects.

A YAML stream is UTF-8/16/32 by definition (YAML spec §5.2), so the transport charset must not decide the codec. Note the local-file branches in the same function already force encoding="utf-8"; only the HTTP branch trusted the header.

Fix

-            content = resp.text
+            content = resp.content.decode("utf-8-sig")

• Decodes the raw bytes as UTF-8 unconditionally (utf-8-sig also strips a leading BOM).
• Keeps load_uri's -> str contract, so all backends and the public OpenAPIParser.load_uri() are unaffected.
• Fixes every submitter whose host omits charset, not just one spec.

Tests

New tests/test_loader_encoding.py (61 tests):

• The reported em-dash → #x0080 case (with a negative control proving the mis-decode is real).
• Non-ASCII matrix (en/em dash, curly quotes, café, emoji, CJK, ellipsis) and a Content-Type matrix (bare text/yaml, declared utf-8, application/yaml, no header, misleading charset=iso-8859-1, …).
• BOM handling; all URL-loading backends end-to-end; local-file regression guards; error propagation; and a white-box guard that the loader reads resp.content and never resp.text.

Validation: all 61 new tests pass; reverting the one-line fix fails 49 of them (the suite genuinely exercises the bug); full parser suite: 150 passed, no regressions.

🤖 Generated with Claude Code