OpenAI Chat Completions ignores the refusal field #46

Activity

1. aoright commented on Sep 25, 2026

   

   aoright

   on Sep 25, 2026

   More actions

   Confirmed. When OpenAI Chat Completions returns a refusal, the payload structure typically sets finish_reason="stop" with message.content=None and message.refusal="<explanation>".

   Because _result() in src/system_one_adapter/providers/openai.py currently inspects only finish_reason and then evaluates choice.message.content or "", the provider response passes through as an empty string "". This triggers malformed JSON parsing errors and consumes corrective retry attempts against a response that the model provider explicitly rejected.

   Proposed Implementation

   In src/system_one_adapter/providers/openai.py::_result:

   1. Check not response.choices up front to record the attempt and fail fast with TypeSafeError.
   2. Record choice.finish_reason.
   3. Check getattr(choice.message, "refusal", None). If present, immediately raise TypeSafeError(f"OpenAI chat completion was a refusal: {refusal}").

   def _result(response: Any) -> ProviderResult:
       if not response.choices:
           record_response(response, finish_reason=None)
           raise TypeSafeError("OpenAI chat completion returned no choices.")
       choice = response.choices[0]
       record_response(response, finish_reason=choice.finish_reason)
       finish_reason = choice.finish_reason
       if finish_reason not in ("stop", None):
           raise TypeSafeError(f"OpenAI chat completion did not complete: {finish_reason}.")
       refusal = getattr(choice.message, "refusal", None)
       if refusal:
           raise TypeSafeError(f"OpenAI chat completion was a refusal: {refusal}")
       return ProviderResult(
           text=choice.message.content or "",
           input_tokens=getattr(response.usage, "prompt_tokens", None),
           output_tokens=getattr(response.usage, "completion_tokens", None),
       )

   Verification and Tests

   Added unit tests in tests/test_provider_nonanswers.py parametrized across sync (OpenAIProvider) and async (AsyncOpenAIProvider) in both structured and prompted modes:

   • test_chat_completion_refusal: Validates that TypeSafeError is raised with the refusal string, retry_reasons == [], len(llm_attempts) == 1, and the debug trace preserves message.refusal.
   • test_chat_completion_empty_choices: Validates handling of empty choices without throwing an unhandled IndexError.

   All 432 unit tests pass and pyrefly check reports 0 errors.

   Branch and commit for review:
   aoright@27d71f1
   (Branch: fix/openai-chat-refusal)
2. dajiaohuang commented on Sep 25, 2026

   

   dajiaohuang

   on Sep 25, 2026

   Last edited by dajiaohuang

   Author

   More actions

   Thanks for confirming the refusal payload and sharing your fix branch. I do not see a PR for it yet. Do you intend to open one from that branch, or would you prefer a separate submission? I want to avoid creating competing PRs for the same behavior.
3. aoright commented on Sep 25, 2026

   

   aoright

   on Sep 25, 2026

   More actions

   Thanks for following up! Direct pull request creation on typesafe-ai/system-one-adapter-python is restricted by repository policy to collaborators (CreatePullRequest permission required), which prevents external forks from opening pull requests directly.

   The branch fix/openai-chat-refusal on my fork (commit 27d71f1) is fully ready, tested (432/432 passed, pyrefly reports 0 errors), and addresses both refusal interception (#46) and empty choices handling (#47) with regression test coverage in tests/test_provider_nonanswers.py.

   Compare: main...aoright:system-one-adapter-python:fix/openai-chat-refusal

   Maintainers can pull directly from this branch. If collaborator permissions are granted to contributors, I can open the pull request directly from this branch.
4. dajiaohuang commented on Oct 7, 2026

   

   dajiaohuang

   on Oct 7, 2026

   Author

   More actions

   Thanks @aoright for clarifying the contribution flow and for preparing the fix branch (fix/openai-chat-refusal, commit 27d71f1) with full test coverage. Acknowledged — I understand direct PR creation is limited to collaborators here.