Skip to content

(improv): add suite compliance context to HTML reports #801

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
chetan-rathore merged 1 commit into from
Sep 17, 2026
Merged

(improv): add suite compliance context to HTML reports #801

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
252 changes: 232 additions & 20 deletions common/acs_test_framework_runner/report_ui_browser_smoke.py
View file
Open in desktop

Large diffs are not rendered by default.

698 changes: 526 additions & 172 deletions common/log_parser/generate_acs_summary.py
View file
Open in desktop

Large diffs are not rendered by default.

17 changes: 17 additions & 0 deletions common/log_parser/main_log_parser.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -471,6 +471,20 @@ if [ "$YOCTO_FLAG_PRESENT" -eq 0 ]; then
"$LOGS_PATH/sbmr/sbmr_out_of_band_logs/report.html"
fi

# Keep legacy per-interface files available; the ACS page uses one SBMR view.
SBMR_COMBINED_ARGS=()
if [ $SBMR_IB_PROCESSED -eq 1 ]; then
SBMR_COMBINED_ARGS+=(--ib-json "$SBMR_IB_JSON" --ib-report "$LOGS_PATH/sbmr/sbmr_in_band_logs/report.html")
fi
if [ $SBMR_OOB_PROCESSED -eq 1 ]; then
SBMR_COMBINED_ARGS+=(--oob-json "$SBMR_OOB_JSON" --oob-report "$LOGS_PATH/sbmr/sbmr_out_of_band_logs/report.html")
fi
if [ ${#SBMR_COMBINED_ARGS[@]} -gt 0 ]; then
python3 "$SCRIPTS_PATH/sbmr/json_to_html.py" --combine \
"${SBMR_COMBINED_ARGS[@]}" \
--detailed "$HTMLS_DIR/sbmr_detailed.html" --summary "$HTMLS_DIR/sbmr_summary.html" || exit 1
fi

fi
################################################################################
# POST-SCRIPT LOG PARSING
Expand Down Expand Up @@ -1012,6 +1026,9 @@ if [ -f "$MERGED_JSON" ]; then
fi

# Finally, call generate_acs_summary.py exactly ONCE at the end
if [ $SBMR_IB_PROCESSED -eq 1 ] || [ $SBMR_OOB_PROCESSED -eq 1 ]; then
GENERATE_ACS_SUMMARY_CMD+=" --sbmr-combined-summary \"$HTMLS_DIR/sbmr_summary.html\""
fi
eval "$GENERATE_ACS_SUMMARY_CMD"

print_path=0 # For debug only
Expand Down
9 changes: 6 additions & 3 deletions common/log_parser/merge_jsons.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,10 +146,11 @@ def is_recommended_test_case(suite_entry):
srs_scope = str(suite_entry.get("SRS scope", "")).strip().lower()
return srs_scope == "recommended"

def count_fails_in_json(data):
def count_fails_in_json(data, skip_recommended=False):
"""
Inspect JSON data and count how many tests are 'FAILED' vs 'FAILED_WITH_WAIVER'.
Returns (failed, failed_with_waiver).
Recommended entries are excluded only when requested for SR OS compliance.

Expects a structure with top-level 'test_results' => [ {subtests: [...]} ]
or a top-level list for subtests. If not recognized, returns (0,0).
Expand All @@ -169,7 +170,7 @@ def count_fails_in_json(data):
return (0, 0)

for suite_entry in test_results:
if is_recommended_test_case(suite_entry):
if skip_recommended and is_recommended_test_case(suite_entry):
continue
# If testcases exist, count only testcase-level results to avoid double counting.
testcases = suite_entry.get("testcases", [])
Expand Down Expand Up @@ -536,7 +537,9 @@ def merge_json_files(json_files, output_file):
ts_dict.update(temp)
merged_results[section_name] = data

f, fw = count_fails_in_json(data)
f, fw = count_fails_in_json(
data, skip_recommended=(DT_OR_SR_MODE == "SR" and suite_key == "OS_TEST")
)
if suite_key in suite_fail_data:
suite_fail_data[suite_key]["Failed"] += f
suite_fail_data[suite_key]["Failed_with_Waiver"] += fw
Expand Down
Loading
Loading