Skip to content

ADFA-6223 | Fix AI agent refusing general questions and migrate prompts to YAML - #114

Merged
jatezzz merged 7 commits into
mainfrom
fix/ADFA-6223-agent-refuses-general-questions
Oct 5, 2026
Merged

jatezzz merged 7 commits into
mainfrom
fix/ADFA-6223-agent-refuses-general-questions

Conversation

@jatezzz

@jatezzz jatezzz commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Description

This PR resolves an issue where the AI agent incorrectly refused general or non-Android requests by overhauling how system prompts are defined and processed.

  • Moved AI agent system prompts from hardcoded Kotlin strings into YAML configuration files located in assets/prompts/ for the Gemini, OpenAI, Local, and AI-Core plugins.
  • Added explicit instructions to the agent scopes to ensure they answer general, non-Android questions without declining.
  • Updated the Gemini backend to pass the system prompt appropriately using the systemInstruction field rather than fabricating a user turn.
  • This allows users to ask general programming questions without triggering the strict app-building workflow, while enabling prompt tuning without requiring Kotlin code changes across multiple plugins.

Details

  • Replaced hardcoded text with YAML configurations (e.g., agent.yml, rules.yml, layout.yml) parsed via PromptConfigLoader and PromptConfigParser.
  • Added conditional wording so the build workflow only triggers when a user explicitly asks to build or modify the open project.
After.mov

Added capabilities (and how to test them)

AI-Core now declares INTERNET and network.access, for the two web tools below.

Capability How to test
web_search tool. A one-off search through the active backend's own provider search (Gemini Google Search grounding, OpenAI Responses web_search). Offered only when the backend reports WebSearchBackend.canSearchWeb(). On Gemini, or OpenAI with the default base URL, ask "What is the latest stable Ktor version?" The trace shows a web_search call and the answer cites sources. Point OpenAI at Ollama/LM Studio, or use Local, and ask again: no web_search is offered or attempted.
Forced first-turn search (VerificationPolicy). The first turn must call web_search when the message holds a code fence, two or more code-like lines, a currency question ("latest", "deprecated"), or a review request with files attached. Paste a fenced Kotlin snippet using Ktor's JsonFeature and ask "Is this correct?" The first turn searches, and the answer flags JsonFeature as removed. On a backend that cannot search, nothing is forced.
fetch_url tool. Reads one http(s) page or raw file. Asks for approval before every fetch and follows redirects only within the same host. A cross-host redirect comes back as a URL, so following it needs a new call and a new approval. Ask "Summarize https://github.com/appdevforall/CodeOnTheGo/blob/stage/README.md": the approval dialog shows the raw URL, and the summary matches the file. Deny the dialog: nothing is fetched.
Answer review. A reply holding a code block gets one review pass against the run's web results before it is shown. Input stays locked for up to 180 s, after which the draft stands. Ask for a small Compose snippet on Gemini/OpenAI: the status shows the review step, then the (possibly corrected) answer.
Re-ask after a prose reply. When the model answers in prose where a tool call was required, the agent loop asks it again rather than ending the run. Covered by AgentLoopTest; on device, multi-part requests ("read X, then change Y") should finish every part.
Fenced tool calls. A reply that is only one fenced {"tool":…} block runs as a call. Fenced JSON inside prose stays an example. Covered by ToolCallExtractorTest.

Merge and publish order

  • Depends on ADFA-6281 | Share AI prompt template engine and pane helpers CodeOnTheGo#2087 (ai.prompt, ai.ui, PromptConfigStore.reload, LlmInferenceService.WebSearchBackend).
  • AI-Core, Gemini, OpenAI, Local and MCP now floor plugin.min_ide_version at 26.41. Do not publish before 26.41 ships.
  • After #2087 merges, refresh libs/ with Update libs from CodeOnTheGo before merging this PR. The committed libs/plugin-api.jar lacks these classes, so the AI plugins do not compile against it yet.

Ticket

ADFA-6223

Observation

The YAML configurations are strictly validated upon plugin activation. Any missing, unknown, or duplicate keys will be rejected with the specific file name logged, falling back to the ai-core default prompt if rendering fails.

…s [ADFA-6223]

Stop refusing general and off-domain questions, and keep working through multi-part requests.

- Move the system prompts, agent loop and tool wording for ai-core and the three backends into YAML; backends take their prompt config as a constructor argument, and ai-core runs its activation checks through PromptConfigChecks.
- Make web search, fetch_url and the device clock always available, and remove the Web search switch.
- End tool runs through respond, fix add_dependency paths, and cite only real sources. Check code-bearing answers in a second review pass that verifies API claims against the evidence.
- ToolCallExtractor ignores bare {"tool":...} JSON inside code fences, closed or not.
- Web search: a GroundingRedirect on a non-HTTP link or unchecked failure is cited as given rather than failing the search; OpenAI throws on a 2xx reply carrying an error object, and 402 and insufficient_quota classify as BillingRequired. OpenAI read timeout is 180s, reported as TimedOut.
- EXAMPLE_FILE_STEM falls back to the full name for dotfiles.
- Use plugin-api's shared prompt engine, YAML loader, store and pane helpers (ADFA-6281); each plugin keeps only its config type, PromptConfigParser and sharedPromptConfig store. snakeyaml moves to testImplementation.

@Daniel-ADFA Daniel-ADFA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 62b7656. Depends on appdevforall/CodeOnTheGo#2087, reviewed separately.

CRITICAL

IMPORTANT

  • build.gradle.kts:85 - libs/plugin-api.jar has no ai.* classes, so none of the plugins compile
  • ToolCallExtractor.kt:207 - a fenced tool call no longer runs and is shown as the answer
  • FetchUrlHandler.kt:70 - redirects reach loopback and LAN hosts without a new approval
  • ChatViewModel.kt:988 - web_search is forced on backends that cannot search

MINOR

  • AndroidManifest.xml:5 - the PR description omits the new internet access and features

NITPICK - 1 inline, not listed

No prior review rounds. The repo has no written approve/request-changes rule (CLAUDE.md, plugin-review skill), so the default grading applied.

Comment thread plugins/AI-Core/src/main/AndroidManifest.xml
Comment thread plugins/AI-Core/build.gradle.kts
Comment thread plugins/AI-Core/src/main/AndroidManifest.xml

@Daniel-ADFA Daniel-ADFA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for the CRITICAL and four IMPORTANT findings in the review above: floor the five AI plugins at the first CodeOnTheGo release that contains appdevforall/CodeOnTheGo#2087 and hold publishing until it ships, refresh libs/ so they compile, keep fenced tool calls running, stop fetch_url following cross-host redirects without approval, and force web_search only on backends that can search.

Floor AI plugins at 26.41, run fenced tool calls, stop fetch_url at cross-host redirects, offer web_search only where the backend can search.
…ADFA-6223]

Drops the preload/release copies from AI-Core, Gemini, OpenAI and Local.

@Daniel-ADFA Daniel-ADFA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 5c72a13. Depends on appdevforall/CodeOnTheGo#2087, reviewed separately.

IMPORTANT

  • FetchUrlHandler.kt:35 - Always Allow on fetch_url approves every later host for the session
  • PromptVariables.kt:224 - the web_access text tells backends without web_search to call it
  • ToolResultsPrompt.kt:66 - fetch_url results are cut to 4,000 characters
  • build.gradle.kts:85 (existing thread) - libs/plugin-api.jar still lacks the #2087 API

MINOR

  • ToolCallExtractor.kt:56 - a stray inline ``` blanks a later bare call (plausible)

NITPICK - 1 inline, not listed

Previous round, checked against the code at 5c72a13:

  • CRITICAL AndroidManifest.xml:60 - fixed: all five plugins floor at 26.41.
  • IMPORTANT build.gradle.kts:85 - not fixed yet (plan agreed); reopened.
  • IMPORTANT ToolCallExtractor.kt:211 - fixed: a reply that is one fenced block and nothing else runs as a call.
  • IMPORTANT FetchUrlHandler.kt:70 - fixed: redirects are followed within one host; a cross-host redirect comes back as a URL.
  • IMPORTANT ChatViewModel.kt:988 - fixed: web_search is offered and forced only when WebSearchBackend.canSearchWeb() says so.
  • MINOR AndroidManifest.xml:5 - fixed: the description lists each added capability with QA steps.
  • NITPICK GeminiPlugin.kt:187 - fixed: the plugins load through PromptConfigStore.reload.

The repo has no written approve/request-changes rule (CLAUDE.md, plugin-review skill), so the default grading applied.

@Daniel-ADFA Daniel-ADFA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes for the IMPORTANT findings in the re-review above: stop fetch_url from being session-approved, render the web_search wording only for backends that offer it, raise the fetch_url result limit, and refresh libs/ once appdevforall/CodeOnTheGo#2087 merges so the plugins compile.

@jatezzz
jatezzz requested a review from Daniel-ADFA October 1, 2026 14:35
jatezzz and others added 2 commits October 1, 2026 12:46
Move read_app_logs/read_ide_logs wording into tool_descriptions.yml and point LogWindowCalculator at ToolResultsPrompt.DEFAULT_CHAR_LIMIT.

Refs: ADFA-6223
@jatezzz
jatezzz merged commit 9d3c965 into main Oct 5, 2026
1 check passed
@jatezzz
jatezzz deleted the fix/ADFA-6223-agent-refuses-general-questions branch October 5, 2026 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants