Skip to content

fix(eval): glm-5.3 rejects thinking.type=disabled — send reasoning_effort alone #62

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ronaldtse merged 1 commit into from
Aug 30, 2026
Merged

fix(eval): glm-5.3 rejects thinking.type=disabled — send reasoning_effort alone #62

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 11 additions & 4 deletions eval_sadeed_glm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,12 @@
unrecognized values silently default to MAX reasoning there — the
2026-08-17 finding was that reasoning mode burns minutes per long
paragraph, so a missing parameter turns a ~1h run into days. GLM-5.2
and earlier disable reasoning via thinking.type=disabled, which
GLM-5.3+ ignores. Passing an effort sends both knobs and warns if the
response still carries reasoning_content (the disable didn't take).
and earlier disable reasoning via thinking.type=disabled; GLM-5.3+
rejects that parameter outright (verified 2026-08-31: HTTP 400 code
1210, valid reasoning_effort values are exactly low, high, max —
thinking cannot be disabled at all). An effort therefore sends ONLY
reasoning_effort, and any response still carrying reasoning_content
warns that it did not take.

Usage:
python eval_sadeed_glm.py [model_id] [reasoning_effort]
Expand Down Expand Up @@ -72,10 +75,14 @@ def call(session: requests.Session, key: str, text: str, tries: int = 5) -> str:
"messages": [{"role": "user", "content": PROMPT.format(t=text)}],
"temperature": 0,
"max_tokens": 8192,
"thinking": {"type": "disabled"},
}
if EFFORT:
# glm-5.3+ rejects thinking.type=disabled outright (HTTP 400,
# code 1210: "This model always engages in thinking"); valid
# reasoning_effort values are low, high, max.
payload["reasoning_effort"] = EFFORT
else:
payload["thinking"] = {"type": "disabled"}
for attempt in range(tries):
try:
r = session.post(BASE, json=payload, timeout=300)
Expand Down
Loading