fix(transfer_to_human): stop always routing to the first configured rule - #55
mateusbellozupko wants to merge 4 commits into
Conversation
When the model called transfer_to_human with only `reason` (no explicit team_id/assignee_id — the common case, since it only knows team NAMES from the docstring, not their opaque IDs), the tool silently picked whichever transfer rule happened to be first in the configured list, regardless of the actual reason. Every escalation for an agent with multiple transfer rules was routed to rule #1 — e.g. an "Imposto de Renda" request got sent to "Dep. Contábil/Fiscal" (rule #1) instead of the dedicated IRPF team (rule evolution-foundation#4), because the code never evaluated rule conditions at all (the old comment literally said "In the future, this could be enhanced to evaluate rule conditions"). Fix: - Add a `rule_index` parameter and instruct the model (via the numbered rule list already in its docstring) to always pass it when transfer_rules are configured. - If rule_index is omitted, fall back to matching `reason` against each rule's own `instructions` text by keyword overlap. - Only if neither yields a match, fall back to the first configured rule (previous behavior), now logged as a warning so misroutes are visible. Reproduced live: an "Imposto de Renda Pessoa Física" request was transferred to "Dep. Contábil/Fiscal" instead of the dedicated team.
Reviewer's GuideTransfer routing now prioritizes explicit model-selected rule indexes, otherwise matches the escalation reason against rule instructions using filtered keyword overlap, and only falls back to the first valid rule with a warning; generated model guidance and tool documentation explain the new contract and notification behavior. Flow diagram for transfer rule selectionflowchart TD
A[transfer_to_human called] --> B{Explicit assignee_id or team_id?}
B -->|Yes| C[Use explicit destination]
B -->|No| D{rule_index provided?}
D -->|Invalid| E[Return error]
D -->|Valid| F[Select numbered transfer rule]
D -->|Omitted| G{Reason keywords match rule instructions?}
G -->|Best match| H[Select highest-scoring rule]
G -->|No match| I[Select first valid rule]
I --> J[Log warning]
F --> K[Resolve user or team destination]
H --> K
J --> K
C --> L[Transfer conversation]
K --> L
File-Level Changes
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
There was a problem hiding this comment.
Hey - I've found 3 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="src/services/adk/tools/evo_crm/transfer_to_human.py" line_range="353-357" />
<code_context>
- you must provide assignee_id or team_id.{transfer_rules_doc}
-
+
+ If transfer_rules are configured, you MUST pass rule_index set to the number of
+ whichever configured rule below actually matches what the user asked about — do
+ not omit it and rely on a default, there is no single "default" rule and omitting
+ it risks the wrong team. Otherwise, provide assignee_id or team_id
+ explicitly.{transfer_rules_doc}
+
Args:
</code_context>
<issue_to_address>
**issue (broader_impact):** The new tool documentation requires the model to pass `rule_index`, but the agent-level CRM prompt still tells the model that configured rules are applied automatically and that it does not need to provide assignment parameters. Models receiving both instructions continue omitting `rule_index`; when the reason has no keyword overlap with the rules, the implementation still routes to the first valid rule.
**Triggers:** When the tool is used through `LlmAgentBuilder` with configured transfer rules and the reason does not contain a token found in any rule's instructions.
**Suggested fix:** Update the agent-level prompt to require the numbered `rule_index` and remove the statement that configured rules are selected automatically.
</issue_to_address>
### Comment 2
<location path="src/services/adk/tools/evo_crm/transfer_to_human.py" line_range="176-181" />
<code_context>
+ reason_lower = reason.lower()
+ for rule in available_transfer_rules:
+ instructions = (rule.get("instructions") or "").lower()
+ if instructions and any(
+ word in instructions
+ for word in reason_lower.split()
+ if len(word) > 3
+ ):
+ selected_rule = rule
+ logger.info(
+ "Matched transfer rule by keyword overlap between "
</code_context>
<issue_to_address>
**issue (bug_risk):** The keyword matcher treats any reason substring longer than three characters as a match, without token boundaries, stop-word filtering, or relevance scoring. A generic word such as `need`, `para`, or `help` appearing in an earlier rule's instructions selects that rule before a later rule with the actual topic match, causing another misroute.
**Triggers:** When a reason contains a common word that appears in an earlier rule's instructions and the intended rule appears later.
**Suggested fix:** Normalize and tokenize both texts, remove stop words, and require meaningful word or phrase overlap; prefer the rule with the strongest match rather than stopping at the first match.
</issue_to_address>
### Comment 3
<location path="src/services/adk/tools/evo_crm/transfer_to_human.py" line_range="162-189" />
<code_context>
+ if rule_index is not None and 1 <= rule_index <= len(available_transfer_rules):
</code_context>
<issue_to_address>
**issue (bug_risk):** An out-of-range or otherwise invalid `rule_index` is silently discarded and the code proceeds to keyword matching or the first-rule fallback. A caller that explicitly selected a nonexistent rule therefore receives a successful transfer to a different rule instead of an invalid-argument error, hiding model/tool contract failures and potentially misrouting the conversation.
**Triggers:** When the model or caller supplies a `rule_index` less than 1 or greater than the configured rule count.
**Suggested fix:** Validate a non-null `rule_index` and return an error when it is outside the configured 1-based range instead of treating it as omitted.
</issue_to_address>Sourcery assessment
Needs a human reviewer. 3 findings to address first, and a bad rule index or keyword match can assign a conversation to the wrong human or team, and that assignment may persist after the code is reverted. The conversation can be reassigned or the job rerun to repair it, but any incorrect recipient access or handling that occurred before correction would not be undone.
Blocking findings: src/services/adk/tools/evo_crm/transfer_to_human.py:357, src/services/adk/tools/evo_crm/transfer_to_human.py:181, src/services/adk/tools/evo_crm/transfer_to_human.py:189
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
- Explicit but out-of-range rule_index now returns an error instead of silently falling through to keyword matching or the first rule. - Keyword fallback scores rules by count of shared meaningful, non-stopword tokens and picks the best match instead of stopping at the first rule containing any word longer than 3 chars. - The agent-level system prompt said "the tool will automatically use the configured transfer rules, so you don't need to specify assignee_id or team_id" — directly contradicting the rule_index requirement. Now numbers the rules and explicitly requires rule_index.
|
Fixed in 5f1e8e0 — invalid rule_index now returns an explicit error instead of silently falling through; keyword fallback now scores rules by shared non-stopword tokens instead of stopping at the first word match; and the agent-level prompt (llm_agent_builder.py) no longer contradicts the rule_index requirement by claiming rules apply automatically. |
|
@sourcery-ai review — all three findings are already addressed on this branch (EVO-2247): the agent-level prompt now explicitly requires |
There was a problem hiding this comment.
Hey - I've found 1 issue
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="src/services/adk/tools/evo_crm/transfer_to_human.py" line_range="198-199" />
<code_context>
+
+ # Preferred path: the model picked a specific rule from the
+ # numbered list in its own docstring (see transfer_rules_doc).
+ if rule_index is not None:
+ selected_rule = available_transfer_rules[rule_index - 1]
+ logger.info(f"Using transfer rule #{rule_index} selected by the model")
+
</code_context>
<issue_to_address>
**issue (broader_impact):** Explicit selection and keyword scoring consider every configured entry, including rules without a valid `userId`/`teamId` or with an unsupported `transferTo` value. When such an entry is selected, `selected_rule` remains truthy but neither effective ID is set, so the function returns `Either assignee_id or team_id is required` instead of falling back to the first valid transfer rule as the previous implementation did.
**Triggers:** When the transfer_rules list contains a malformed or unsupported entry whose index or instructions match the request.
**Suggested fix:** Filter candidate rules to entries with a supported transfer target and corresponding ID before validating or selecting them, while preserving the displayed rule indexes for explicit selection.
</issue_to_address>Sourcery assessment
Needs a human reviewer. 1 finding to address first, and a wrong rule selection can assign a conversation to the wrong human or team, and that assignment remains after reverting until it is corrected or re-routed. The impact is bounded and generally repairable, but keyword matching and the retained first-rule fallback can still misroute transfers.
Blocking findings: src/services/adk/tools/evo_crm/transfer_to_human.py:199
|
Fixed — extracted the existing valid-target check (already used by the first-valid-rule fallback) into |
…ring selection Keyword matching and the first-valid-rule fallback both need a rule with a usable userId/teamId for its transferTo, but the keyword-scoring loop selected purely on instructions-text overlap without that check. A misconfigured rule (e.g. transferTo: team with no teamId) could win the keyword match on its instructions alone, then leave both effective_assignee_id and effective_team_id unset — a hard "assignee_id or team_id is required" error instead of falling through to the next, actually-usable rule. Extracted the existing valid-target check (already used by the first-valid-rule fallback) into _rule_has_valid_target and applied it to the keyword-scoring loop too. An explicit rule_index still errors on its own malformed rule rather than silently substituting another — that's the model's own contract failure, not something to paper over. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…x picks it An explicit rule_index selecting a rule without a valid userId/teamId for its transferTo type used to fall through silently to the generic "assignee_id or team_id is required" error, hiding which rule was actually broken. Now it reports which rule and why.
Summary
When the model calls
transfer_to_humanwith onlyreason(no explicitteam_id/assignee_id— the common case, since the model only knows team names from the tool's docstring, not their opaque IDs), the tool silently picked whichever transfer rule happened to be listed first, completely ignoring the actual reason. The old code's own comment admitted this: "Use the first transfer rule that matches 'human' or 'team' ... In the future, this could be enhanced to evaluate rule conditions."Practical effect: any agent with more than one transfer rule always routed every escalation to rule #1, regardless of topic.
Fix
rule_indexparameter, and instruct the model (via the numbered rule list already built into its docstring) to always pass it when transfer_rules are configured.rule_indexis omitted, fall back to matchingreasonagainst each rule's owninstructionstext by keyword overlap.Testing notes
Reproduced live against a real agent with 8 transfer rules: a user message about "Imposto de Renda Pessoa Física" (rule #4) was being routed to "Dep. Contábil/Fiscal" (rule #1) every time. Confirmed via the tool's own JSON-RPC response, which showed
team_idresolving to rule #1's team regardless of the stated reason.🤖 Generated with Claude Code
Summary by Sourcery
Route human transfers to the rule that best matches the escalation context instead of silently defaulting to the first configured rule.
New Features:
rule_indexparameter and model instructions.Bug Fixes:
Enhancements:
Tests: