fix(gooddata-eval): make ranking attribute optional on 1-dim viz - #1707
Merged
Conversation
📝 WalkthroughWalkthroughRanking-filter validation now reports malformed values as errors. Normalization accepts unspecified attributes for single-dimension visualizations and uses the sole dimension. Tests cover validation, comparison, cross-reference handling, and visualization evaluation. ChangesRanking Filter Validation
Estimated code review effort: 3 (Moderate) | ~20 minutes Suggested reviewers: Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
Comment |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1707 +/- ##
==========================================
+ Coverage 78.30% 78.40% +0.09%
==========================================
Files 271 271
Lines 18689 18741 +52
==========================================
+ Hits 14634 14693 +59
+ Misses 4055 4048 -7 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/gooddata-eval/tests/test_scoring.py`:
- Around line 134-147: Update
test_validate_cross_references_never_raises_on_empty_or_none_uris so each
malformed case includes its expected validation result, marking invalid
attribute and using values as errors. Assert ok matches that expected result and
retain the existing errors-list consistency check, ensuring cases such as
attribute=[] and using=None cannot pass as valid.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 4d7617e8-da56-4ff1-9bc2-f51be8b2e785
📒 Files selected for processing (3)
packages/gooddata-eval/src/gooddata_eval/core/scoring.pypackages/gooddata-eval/tests/test_scoring.pypackages/gooddata-eval/tests/test_visualization_evaluator.py
The visualization comparator required a ranking filter's `attribute` to match exactly, but `attribute` is optional in the AAC schema -- gen-ai models it as `NotRequired[str]` / `str | None` in all three of its ranking-filter types, and when it is absent AFM ranks over every dimension of the result. On a chart with exactly one dimension that is the same filter, so the comparator was stricter than the product contract and failed correct answers. Both Anthropic models consistently omit `attribute` while getting the metric and top/bottom-N right, which made this the largest visualization failure cluster: 14 of 49 viz failures in run 30850362312 (opus48 9, sonnet46 3, bedrock 2). No GPT combo is affected. `_normalize_ranking_filter` now fills an omitted attribute in with the visualization's sole dimension URI instead of comparing it as an empty string. The substitution is gated on there being exactly ONE distinct dimension: with two or more, omitting `attribute` ranks over the dimension tuple, which is a genuinely different filter, so those stay strict. It is applied to expected and actual alike, because datasets omit `attribute` too -- without symmetry an agent that supplies the more precise filter would fail against a fixture that omits it. Missing, None and "" now normalize identically, so `attribute: null` no longer differs from an absent key. Also make `validate_cross_references` return a score instead of raising. None, "" and non-string values reached `.startswith()` / `dict.get()` and blew up with AttributeError / TypeError mid-evaluation. This affected the `using` branch as well as `attribute`. Its test asserts the expected verdict per malformed case rather than comparing `ok` against the returned error-list length, which was a tautology against an implementation that returns exactly `len(errors) == 0`; confirmed non-vacuous by mutation. Verified by re-scoring all 48 expected/actual pairs lifted from run 30850362312 with the patched module: 14 flip FAIL -> PASS, 0 checks that CI reported as True became False. Note this raises opus48's pass rate ~4.5pp for comparator reasons, not model ones. JIRA: QA-28615 risk: nonprod Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/gooddata-eval/src/gooddata_eval/core/scoring.py`:
- Around line 143-155: Reject malformed ranking filters, including non-string
values such as attribute=[], before normalization in both the visualization
evaluator and conversation.py paths. Add the validation gate before
check_filters so invalid filters cannot become equivalent to omitted attributes,
and ensure conversation.py cannot set output_correct=True without
cross-reference validation.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 830a4168-f2a0-4c49-8c77-4b4ae0758662
📒 Files selected for processing (3)
packages/gooddata-eval/src/gooddata_eval/core/scoring.pypackages/gooddata-eval/tests/test_scoring.pypackages/gooddata-eval/tests/test_visualization_evaluator.py
🚧 Files skipped from review as they are similar to previous changes (2)
- packages/gooddata-eval/tests/test_visualization_evaluator.py
- packages/gooddata-eval/tests/test_scoring.py
Comment on lines
+143
to
+155
| Missing, None and "" are all treated as "not specified"; so is a non-string, which | ||
| `validate_cross_references` reports separately rather than crashing the comparison. | ||
| """ | ||
| attr_val = filter_dict.get("attribute") | ||
| if not isinstance(attr_val, str) or not attr_val: | ||
| dim_uri = sole_dim_uri or "" | ||
| else: | ||
| dim_uri = _resolve_alias_to_uri(attr_val, fields) | ||
| using_val = filter_dict.get("using") | ||
| entry: dict = { | ||
| "type": "ranking_filter", | ||
| "metric_uri": _resolve_alias_to_uri(filter_dict.get("using", ""), fields), | ||
| "dim_uri": _resolve_alias_to_uri(filter_dict.get("attribute", ""), fields), | ||
| "metric_uri": _resolve_alias_to_uri(using_val, fields) if isinstance(using_val, str) else "", | ||
| "dim_uri": dim_uri, |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map scoring entry points before inspecting callers.
ast-grep outline packages/gooddata-eval/src/gooddata_eval/core/scoring.py --items all --view expanded
# Inspect validation and normalization call sites.
rg -n -C 12 --glob '*.py' \
'\bvalidate_cross_references\s*\(|\b_split_and_normalize_filters\s*\(' \
packages/gooddata-eval/src packages/gooddata-eval/testsRepository: gooddata/gooddata-python-sdk
Length of output: 16517
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- scoring.py ---'
sed -n '1,230p' packages/gooddata-eval/src/gooddata_eval/core/scoring.py
printf '%s\n' '--- visualization evaluator ---'
sed -n '1,180p' packages/gooddata-eval/src/gooddata_eval/core/evaluators/visualization.py
printf '%s\n' '--- evaluator and scoring usages ---'
rg -n -C 8 --glob '*.py' \
'evaluate_visualization|check_filters|cross_ref_valid|filters_correct|filter_ranking_score|EvaluationResult' \
packages/gooddata-eval/src packages/gooddata-eval/testsRepository: gooddata/gooddata-python-sdk
Length of output: 50384
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
import ast
path = Path("packages/gooddata-eval/src/gooddata_eval/core/scoring.py")
tree = ast.parse(path.read_text())
wanted = {
"validate_cross_references",
"_sole_dimension_uri",
"_normalize_ranking_filter",
"_split_and_normalize_filters",
"check_filters",
}
for node in tree.body:
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) and node.name in wanted:
print(f"--- {node.name} lines {node.lineno}-{node.end_lineno} ---")
print("\n".join(path.read_text().splitlines()[node.lineno-1:node.end_lineno]))
PY
printf '%s\n' '--- direct evaluator control flow ---'
python3 - <<'PY'
from pathlib import Path
p = Path("packages/gooddata-eval/src/gooddata_eval/core/evaluators/visualization.py")
lines = p.read_text().splitlines()
for i, line in enumerate(lines, 1):
if any(x in line for x in ("validate_cross_references", "check_filters", "return EvaluationResult")):
lo, hi = max(1, i-8), min(len(lines), i+18)
print(f"--- lines {lo}-{hi} ---")
print("\n".join(f"{n}: {lines[n-1]}" for n in range(lo, hi+1)))
PYRepository: gooddata/gooddata-python-sdk
Length of output: 12853
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- conversation scoring path ---'
sed -n '130,190p' packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.py
rg -n -C 10 --glob '*.py' \
'validate_cross_references|_evaluate_visualization|_evaluate_against_candidates|_evaluate_expected' \
packages/gooddata-eval/src packages/gooddata-eval/tests
printf '%s\n' '--- validation/scoring tests ---'
sed -n '120,175p' packages/gooddata-eval/tests/test_scoring.py
sed -n '90,145p' packages/gooddata-eval/tests/test_visualization_evaluator.pyRepository: gooddata/gooddata-python-sdk
Length of output: 36482
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
import json
def normalize_ranking_filter(filter_dict, fields, sole_dim_uri):
attr_val = filter_dict.get("attribute")
if not isinstance(attr_val, str) or not attr_val:
dim_uri = sole_dim_uri or ""
else:
dim_uri = fields.get(attr_val, attr_val)
using_val = filter_dict.get("using")
entry = {
"type": "ranking_filter",
"metric_uri": fields.get(using_val, using_val) if isinstance(using_val, str) else "",
"dim_uri": dim_uri,
}
if "top" in filter_dict:
entry["top"] = filter_dict["top"]
if "bottom" in filter_dict:
entry["bottom"] = filter_dict["bottom"]
return json.dumps(entry, sort_keys=True)
fields = {
"m_sales": "metric/net_sales",
"d_product": "label/product_id",
}
expected = {"type": "ranking_filter", "using": "m_sales", "attribute": "d_product", "top": 5}
malformed = {"type": "ranking_filter", "using": "m_sales", "attribute": [], "top": 5}
omitted = {"type": "ranking_filter", "using": "m_sales", "top": 5}
for name, value in (("expected", expected), ("malformed", malformed), ("omitted", omitted)):
print(name, normalize_ranking_filter(value, fields, "label/product_id"))
print("malformed_matches_expected:",
normalize_ranking_filter(malformed, fields, "label/product_id")
== normalize_ranking_filter(expected, fields, "label/product_id"))
print("malformed_matches_omitted:",
normalize_ranking_filter(malformed, fields, "label/product_id")
== normalize_ranking_filter(omitted, fields, "label/product_id"))
PYRepository: gooddata/gooddata-python-sdk
Length of output: 564
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 12 --glob '*.py' \
'_check_output_correct\s*\(' \
packages/gooddata-eval/src packages/gooddata-eval/tests
sed -n '1,135p' packages/gooddata-eval/src/gooddata_eval/core/agentic/conversation.pyRepository: gooddata/gooddata-python-sdk
Length of output: 10780
Reject malformed filters before normalization.
attribute=[] normalizes to the same ranking filter as a valid or omitted attribute. The visualization evaluator reports a false-positive ranking score, and conversation.py can set output_correct=True without cross-reference validation. Gate both paths before check_filters.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/gooddata-eval/src/gooddata_eval/core/scoring.py` around lines 143 -
155, Reject malformed ranking filters, including non-string values such as
attribute=[], before normalization in both the visualization evaluator and
conversation.py paths. Add the validation gate before check_filters so invalid
filters cannot become equivalent to omitted attributes, and ensure
conversation.py cannot set output_correct=True without cross-reference
validation.
henrynguyengooddata
approved these changes
Aug 5, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
JIRA: QA-28615
risk: nonprod
Summary by CodeRabbit
Bug Fixes
Noneranking attributes are accepted for single-dimension visualizations.Tests