Skip to content

fix(transfer_to_human): stop always routing to the first configured rule - #55

Open
mateusbellozupko wants to merge 4 commits into
evolution-foundation:developfrom
mateusbellozupko:fix/transfer-to-human-rule-routing
Open

mateusbellozupko wants to merge 4 commits into
evolution-foundation:developfrom
mateusbellozupko:fix/transfer-to-human-rule-routing

Conversation

@mateusbellozupko

@mateusbellozupko mateusbellozupko commented Aug 29, 2026 •

Copy link
Copy Markdown

Summary

When the model calls transfer_to_human with only reason (no explicit team_id/assignee_id — the common case, since the model only knows team names from the tool's docstring, not their opaque IDs), the tool silently picked whichever transfer rule happened to be listed first, completely ignoring the actual reason. The old code's own comment admitted this: "Use the first transfer rule that matches 'human' or 'team' ... In the future, this could be enhanced to evaluate rule conditions."

Practical effect: any agent with more than one transfer rule always routed every escalation to rule #1, regardless of topic.

Fix

  • Add a rule_index parameter, and instruct the model (via the numbered rule list already built into its docstring) to always pass it when transfer_rules are configured.
  • If rule_index is omitted, fall back to matching reason against each rule's own instructions text by keyword overlap.
  • Only if neither yields a match, fall back to the first configured rule (previous behavior) — now logged as a warning so misroutes are visible instead of silent.

Testing notes

Reproduced live against a real agent with 8 transfer rules: a user message about "Imposto de Renda Pessoa Física" (rule #4) was being routed to "Dep. Contábil/Fiscal" (rule #1) every time. Confirmed via the tool's own JSON-RPC response, which showed team_id resolving to rule #1's team regardless of the stated reason.

🤖 Generated with Claude Code

Summary by Sourcery

Route human transfers to the rule that best matches the escalation context instead of silently defaulting to the first configured rule.

New Features:

  • Add explicit numbered transfer-rule selection through the new rule_index parameter and model instructions.

Bug Fixes:

  • Prevent transfers without explicit IDs from silently routing to the first configured rule regardless of the escalation reason.
  • Handle malformed transfer rules without valid target IDs and report invalid explicit rule selections clearly.

Enhancements:

  • Match omitted rule selections against transfer-rule instructions using meaningful keyword overlap, with a logged warning for the final first-rule fallback.
  • Improve transfer guidance to require rule selection and support customer notification instructions.

Tests:

  • Add unit coverage for skipping unusable keyword-matched rules, reporting configurations with no valid targets, and rejecting malformed explicitly selected rules.

When the model called transfer_to_human with only `reason` (no explicit
team_id/assignee_id — the common case, since it only knows team NAMES from
the docstring, not their opaque IDs), the tool silently picked whichever
transfer rule happened to be first in the configured list, regardless of
the actual reason. Every escalation for an agent with multiple transfer
rules was routed to rule #1 — e.g. an "Imposto de Renda" request got sent
to "Dep. Contábil/Fiscal" (rule #1) instead of the dedicated IRPF team
(rule evolution-foundation#4), because the code never evaluated rule conditions at all (the
old comment literally said "In the future, this could be enhanced to
evaluate rule conditions").

Fix:
- Add a `rule_index` parameter and instruct the model (via the numbered
  rule list already in its docstring) to always pass it when transfer_rules
  are configured.
- If rule_index is omitted, fall back to matching `reason` against each
  rule's own `instructions` text by keyword overlap.
- Only if neither yields a match, fall back to the first configured rule
  (previous behavior), now logged as a warning so misroutes are visible.

Reproduced live: an "Imposto de Renda Pessoa Física" request was
transferred to "Dep. Contábil/Fiscal" instead of the dedicated team.
@sourcery-ai

sourcery-ai Bot commented Aug 29, 2026 •

Copy link
Copy Markdown

Reviewer's Guide

Transfer routing now prioritizes explicit model-selected rule indexes, otherwise matches the escalation reason against rule instructions using filtered keyword overlap, and only falls back to the first valid rule with a warning; generated model guidance and tool documentation explain the new contract and notification behavior.

Flow diagram for transfer rule selection

flowchart TD
    A[transfer_to_human called] --> B{Explicit assignee_id or team_id?}
    B -->|Yes| C[Use explicit destination]
    B -->|No| D{rule_index provided?}
    D -->|Invalid| E[Return error]
    D -->|Valid| F[Select numbered transfer rule]
    D -->|Omitted| G{Reason keywords match rule instructions?}
    G -->|Best match| H[Select highest-scoring rule]
    G -->|No match| I[Select first valid rule]
    I --> J[Log warning]
    F --> K[Resolve user or team destination]
    H --> K
    J --> K
    C --> L[Transfer conversation]
    K --> L
Loading

File-Level Changes

Change Details Files
Make transfer-rule selection explicit and visible to the model.
  • Number configured rules in generated instructions and tool documentation.
  • Add the 1-based rule_index argument and require it when rules are configured.
  • Add guidance to provide message_to_customer when rule instructions require customer notification.
src/services/adk/agents/llm_agent_builder.py
src/services/adk/tools/evo_crm/transfer_to_human.py
Replace unconditional first-rule routing with ordered selection and safer fallback behavior.
  • Reject out-of-range rule indexes with a structured error.
  • Select the explicitly indexed rule before attempting inference.
  • Preserve explicit assignee/team IDs as higher-priority inputs.
src/services/adk/tools/evo_crm/transfer_to_human.py
Add reason-to-instruction matching for transfers without an explicit rule index.
  • Tokenize reason and rule instructions using Unicode-aware word extraction.
  • Filter generic Portuguese and English stopwords and select the highest keyword-overlap score.
  • Warn when no match exists and the implementation falls back to the first valid configured rule.
src/services/adk/tools/evo_crm/transfer_to_human.py

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 3 issues

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="src/services/adk/tools/evo_crm/transfer_to_human.py" line_range="353-357" />
<code_context>
-    you must provide assignee_id or team_id.{transfer_rules_doc}
-    
+
+    If transfer_rules are configured, you MUST pass rule_index set to the number of
+    whichever configured rule below actually matches what the user asked about — do
+    not omit it and rely on a default, there is no single "default" rule and omitting
+    it risks the wrong team. Otherwise, provide assignee_id or team_id
+    explicitly.{transfer_rules_doc}
+
     Args:
</code_context>
<issue_to_address>
**issue (broader_impact):** The new tool documentation requires the model to pass `rule_index`, but the agent-level CRM prompt still tells the model that configured rules are applied automatically and that it does not need to provide assignment parameters. Models receiving both instructions continue omitting `rule_index`; when the reason has no keyword overlap with the rules, the implementation still routes to the first valid rule.

**Triggers:** When the tool is used through `LlmAgentBuilder` with configured transfer rules and the reason does not contain a token found in any rule's instructions.

**Suggested fix:** Update the agent-level prompt to require the numbered `rule_index` and remove the statement that configured rules are selected automatically.
</issue_to_address>

### Comment 2
<location path="src/services/adk/tools/evo_crm/transfer_to_human.py" line_range="176-181" />
<code_context>
+                    reason_lower = reason.lower()
+                    for rule in available_transfer_rules:
+                        instructions = (rule.get("instructions") or "").lower()
+                        if instructions and any(
+                            word in instructions
+                            for word in reason_lower.split()
+                            if len(word) > 3
+                        ):
+                            selected_rule = rule
+                            logger.info(
+                                "Matched transfer rule by keyword overlap between "
</code_context>
<issue_to_address>
**issue (bug_risk):** The keyword matcher treats any reason substring longer than three characters as a match, without token boundaries, stop-word filtering, or relevance scoring. A generic word such as `need`, `para`, or `help` appearing in an earlier rule's instructions selects that rule before a later rule with the actual topic match, causing another misroute.

**Triggers:** When a reason contains a common word that appears in an earlier rule's instructions and the intended rule appears later.

**Suggested fix:** Normalize and tokenize both texts, remove stop words, and require meaningful word or phrase overlap; prefer the rule with the strongest match rather than stopping at the first match.
</issue_to_address>

### Comment 3
<location path="src/services/adk/tools/evo_crm/transfer_to_human.py" line_range="162-189" />
<code_context>
+                if rule_index is not None and 1 <= rule_index <= len(available_transfer_rules):
</code_context>
<issue_to_address>
**issue (bug_risk):** An out-of-range or otherwise invalid `rule_index` is silently discarded and the code proceeds to keyword matching or the first-rule fallback. A caller that explicitly selected a nonexistent rule therefore receives a successful transfer to a different rule instead of an invalid-argument error, hiding model/tool contract failures and potentially misrouting the conversation.

**Triggers:** When the model or caller supplies a `rule_index` less than 1 or greater than the configured rule count.

**Suggested fix:** Validate a non-null `rule_index` and return an error when it is outside the configured 1-based range instead of treating it as omitted.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 3 findings to address first, and a bad rule index or keyword match can assign a conversation to the wrong human or team, and that assignment may persist after the code is reverted. The conversation can be reassigned or the job rerun to repair it, but any incorrect recipient access or handling that occurred before correction would not be undone.

Blocking findings: src/services/adk/tools/evo_crm/transfer_to_human.py:357, src/services/adk/tools/evo_crm/transfer_to_human.py:181, src/services/adk/tools/evo_crm/transfer_to_human.py:189


Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Comment thread src/services/adk/tools/evo_crm/transfer_to_human.py
Comment thread src/services/adk/tools/evo_crm/transfer_to_human.py Outdated
Comment thread src/services/adk/tools/evo_crm/transfer_to_human.py Outdated
- Explicit but out-of-range rule_index now returns an error instead of
  silently falling through to keyword matching or the first rule.
- Keyword fallback scores rules by count of shared meaningful,
  non-stopword tokens and picks the best match instead of stopping at
  the first rule containing any word longer than 3 chars.
- The agent-level system prompt said "the tool will automatically use
  the configured transfer rules, so you don't need to specify assignee_id
  or team_id" — directly contradicting the rule_index requirement. Now
  numbers the rules and explicitly requires rule_index.
@mateusbellozupko

Copy link
Copy Markdown
Author

Fixed in 5f1e8e0 — invalid rule_index now returns an explicit error instead of silently falling through; keyword fallback now scores rules by shared non-stopword tokens instead of stopping at the first word match; and the agent-level prompt (llm_agent_builder.py) no longer contradicts the rule_index requirement by claiming rules apply automatically.

@mateusbellozupko

Copy link
Copy Markdown
Author

@sourcery-ai review — all three findings are already addressed on this branch (EVO-2247): the agent-level prompt now explicitly requires rule_index ("You MUST pass rule_index... do not omit it, there is no reliable automatic default"), the keyword fallback now scores stopword-filtered token overlap instead of any substring match, and an out-of-range/invalid rule_index returns an explicit error instead of falling through. Requesting a fresh review to clear the stale blocking status.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've found 1 issue

Prompt for AI Agents
Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="src/services/adk/tools/evo_crm/transfer_to_human.py" line_range="198-199" />
<code_context>
+
+                # Preferred path: the model picked a specific rule from the
+                # numbered list in its own docstring (see transfer_rules_doc).
+                if rule_index is not None:
+                    selected_rule = available_transfer_rules[rule_index - 1]
+                    logger.info(f"Using transfer rule #{rule_index} selected by the model")
+
</code_context>
<issue_to_address>
**issue (broader_impact):** Explicit selection and keyword scoring consider every configured entry, including rules without a valid `userId`/`teamId` or with an unsupported `transferTo` value. When such an entry is selected, `selected_rule` remains truthy but neither effective ID is set, so the function returns `Either assignee_id or team_id is required` instead of falling back to the first valid transfer rule as the previous implementation did.

**Triggers:** When the transfer_rules list contains a malformed or unsupported entry whose index or instructions match the request.

**Suggested fix:** Filter candidate rules to entries with a supported transfer target and corresponding ID before validating or selecting them, while preserving the displayed rule indexes for explicit selection.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 1 finding to address first, and a wrong rule selection can assign a conversation to the wrong human or team, and that assignment remains after reverting until it is corrected or re-routed. The impact is bounded and generally repairable, but keyword matching and the retained first-rule fallback can still misroute transfers.

Blocking findings: src/services/adk/tools/evo_crm/transfer_to_human.py:199


Sourcery is free for open source - if you like our reviews please consider sharing them ✨

Comment thread src/services/adk/tools/evo_crm/transfer_to_human.py Outdated
@mateusbellozupko

Copy link
Copy Markdown
Author

Fixed — extracted the existing valid-target check (already used by the first-valid-rule fallback) into _rule_has_valid_target and applied it to the keyword-scoring loop too, so a misconfigured rule (valid transferTo but missing userId/teamId) can no longer win the keyword match on its instructions text alone and leave both IDs unset. An explicit rule_index still errors on its own malformed rule rather than silently substituting another, since that's the model's own contract failure. Added tests for all three cases.

mateusbellozupko and others added 2 commits September 20, 2026 07:15
…ring selection

Keyword matching and the first-valid-rule fallback both need a rule with
a usable userId/teamId for its transferTo, but the keyword-scoring loop
selected purely on instructions-text overlap without that check. A
misconfigured rule (e.g. transferTo: team with no teamId) could win the
keyword match on its instructions alone, then leave both
effective_assignee_id and effective_team_id unset — a hard "assignee_id
or team_id is required" error instead of falling through to the next,
actually-usable rule.

Extracted the existing valid-target check (already used by the
first-valid-rule fallback) into _rule_has_valid_target and applied it to
the keyword-scoring loop too. An explicit rule_index still errors on its
own malformed rule rather than silently substituting another — that's
the model's own contract failure, not something to paper over.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…x picks it

An explicit rule_index selecting a rule without a valid userId/teamId
for its transferTo type used to fall through silently to the generic
"assignee_id or team_id is required" error, hiding which rule was
actually broken. Now it reports which rule and why.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant