Skip to content

Begin verification report - #551

Open
danth wants to merge 1 commit into
eclipse-score:mainfrom
etas-contrib:verification-report
Open

Begin verification report#551
danth wants to merge 1 commit into
eclipse-score:mainfrom
etas-contrib:verification-report

Conversation

@danth

@danth danth commented Aug 27, 2026

Copy link
Copy Markdown
Member

This brings in the verification report from the module template, as required for eclipse-score/score#3219.

I began to fill it using the graphs we already had, and some new tables. There is still a lot missing.

At some point we should remove the template text, but I left it in for now so we can see what is left to do.

@danth
danth requested a deployment to workflow-approval August 27, 2026 14:11 — with GitHub Actions Waiting
@danth
danth requested a deployment to workflow-approval August 27, 2026 14:11 — with GitHub Actions Waiting
@github-actions

github-actions Bot commented Aug 27, 2026

Copy link
Copy Markdown

License Check Results

🚀 The license check job ran with the Bazel command:

bazel run --lockfile_mode=error //:license-check

Status: ⚠️ Needs Review

Click to expand output 
[License Check Output]
Extracting Bazel installation...
Starting local Bazel server (8.6.0) and connecting to it...
INFO: Invocation ID: 922e3825-f8dd-4459-9581-505d64386ec1
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
Loading: 0 packages loaded
    currently loading: 
Loading: 0 packages loaded
    currently loading: 
Loading: 0 packages loaded
    currently loading: 
Loading: 0 packages loaded
    currently loading: 
Loading: 0 packages loaded
    currently loading: 
Loading: 0 packages loaded
    currently loading: 
Analyzing: target //:license-check (1 packages loaded, 0 targets configured)
Analyzing: target //:license-check (1 packages loaded, 0 targets configured)

Analyzing: target //:license-check (54 packages loaded, 10 targets configured)

Analyzing: target //:license-check (140 packages loaded, 957 targets configured)

Analyzing: target //:license-check (157 packages loaded, 6344 targets configured)

Analyzing: target //:license-check (166 packages loaded, 9394 targets configured)

Analyzing: target //:license-check (168 packages loaded, 9417 targets configured)

Analyzing: target //:license-check (169 packages loaded, 9541 targets configured)

INFO: Analyzed target //:license-check (173 packages loaded, 11555 targets configured).
[9 / 16] Creating runfiles tree bazel-out/k8-opt-exec-ST-d57f47055a04/bin/external/score_tooling+/dash/tool/formatters/dash_format_converter.runfiles [for tool]; 0s local
INFO: Found 1 target...
Target //:license.check.license_check up-to-date:
  bazel-bin/license.check.license_check
  bazel-bin/license.check.license_check.jar
INFO: Elapsed time: 22.866s, Critical Path: 0.33s
INFO: 16 processes: 4 disk cache hit, 12 internal.
INFO: Build completed successfully, 16 total actions
INFO: Running command line: bazel-bin/license.check.license_check ./formatted.txt <args omitted>
usage: org.eclipse.dash.licenses.cli.Main [-batch <int>] [-cd <url>]
       [-confidence <int>] [-ef <url>] [-excludeSources <sources>] [-help] [-lic
       <url>] [-project <shortname>] [-repo <url>] [-review] [-summary <file>]
       [-timeout <seconds>] [-token <token>]

danth
danth commented Aug 28, 2026
View reviewed changes
Comment thread docs/verification_report/index.rst
Comment on lines +44 to +61
.. grid-item::

.. needpie:: Valid requirements
:labels: not valid, valid but not tested, valid and tested
:colors: red, orange, green

type == 'comp_req' and status == 'invalid'
type == 'comp_req' and testlink == '' and (status == 'valid' or status == 'invalid')
type == 'comp_req' and testlink != '' and (status == 'valid' or status == 'invalid')

.. grid-item::

.. needpie:: Code-linked requirements
:labels: not linked, linked
:colors: red, green

type == 'comp_req' and source_code_link == ''
type == 'comp_req' and source_code_link != ''

@danth danth Aug 28, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should include AoU requirements too I think.

@MaciejKaszynski MaciejKaszynski Aug 28, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these not lower down in 63-66? Or not sure if you can link aou to source code links

@danth danth Aug 28, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least for valid requirements we should be able to include them.

@AlexanderLanin

AlexanderLanin commented Aug 28, 2026

Copy link
Copy Markdown
Member

⚠️ @danth we are currently looking into providing that via a central template

AlexanderLanin
AlexanderLanin reviewed Aug 28, 2026
View reviewed changes
Comment thread docs/verification_report/index.rst
Comment on lines +22 to +165
.. document:: Lifecycle Verification Report
:id: doc__lifecycle_verification_report
:status: draft
:version: 1
:safety: ASIL_B
:security: YES
:realizes: wp__verification_module_ver_report

This verification report is based on the :need:`gd_temp__verification_plan`.
It covers all the components of the above stated module.

**1. Verification Coverage**

**1.1. on Requirements**
- Lists of component requirements (incl. AoU satisfied by the component) tested by which test case, passed/failed and completeness verdict
(this shall be generated by tools and accompanied by progress charts to be usable also for project steering)
- For external component Assumptions of Use: coverage by platform safety manual or feature/components incl. test case, passed/failed and completeness verdict
- This is split in a list of QM requirements tested and a separate list of tests for ASIL rated requirements.
- List of component requirements (ASIL rated) linked to inspection checklist and verdict (derived from PR export)

.. grid:: 2

.. grid-item::

.. needpie:: Valid requirements
:labels: not valid, valid but not tested, valid and tested
:colors: red, orange, green

type == 'comp_req' and status == 'invalid'
type == 'comp_req' and testlink == '' and (status == 'valid' or status == 'invalid')
type == 'comp_req' and testlink != '' and (status == 'valid' or status == 'invalid')

.. grid-item::

.. needpie:: Code-linked requirements
:labels: not linked, linked
:colors: red, green

type == 'comp_req' and source_code_link == ''
type == 'comp_req' and source_code_link != ''

.. needtable:: QM requirements
:filter: (type == "aou_req" or type == "comp_req") and status == "valid" and safety == "QM"
:columns: id;title;satisfied_by as "Satisfied by";testlink as "Tested by"
:class: compact-needtable

.. needtable:: ASIL requirements
:filter: (type == "aou_req" or type == "comp_req") and status == "valid" and safety == "ASIL_B"
:columns: id;title;satisfied_by as "Satisfied by";testlink as "Tested by"
:class: compact-needtable

**1.2. on Architecture**
- List of component architecture tags tested by which test case, passed/failed and completeness verdict
(this shall be generated by tools and accompanied by progress charts to be usable also for project steering)
- This is split in a list of QM components tested and a separate list of tests for ASIL rated components.
- List of component architecture tags (ASIL rated) linked to inspection checklist and verdict (derived from PR export)

**1.3. on Detailed Design**
- List of detailed design tags tested by which test case, passed/failed and completeness verdict
(this shall be generated by tools and accompanied by progress charts to be usable also for project steering)
- This is split in a list of QM components tested and a separate list of tests for ASIL rated components.
- List of detailed design tags (ASIL rated) linked to inspection checklist and verdict (derived from PR export)

- The lists may also contain other verification methods like "Analysis" - process tbd

**2. DFA Report**
- List of the performed component DFA, pass/fail with open mitigations

**3. Safety Analysis Report**
- List of the performed component Safety Analysis, pass/fail with open mitigations

**4. Unit Verification Coverage**

**4.1. Structural Coverage**
- List of the units with C0(line) and C1(branch) coverage absolute and percentage
(this shall be generated by tools and accompanied by progress charts to be usable also for project steering)
- List includes a column for the safety rating of each unit.

**4.2. Static Code Analysis**
- List of the units with compiler warning numbers and coding rule violation numbers
(this shall be generated by tools and accompanied by progress charts to be usable also for project steering)
- List includes a column for the safety rating of each unit.

**4.3. Manual Code Inspection**
- List of components (ASIL rated) linked to inspection checklist and verdict (derived from PR export)

**5. Software component qualification verification report**
- Contains the needed verification results if for some components a qualification of pre-developed SW is performed.

**6. Test results**
- Test result per test case with status passed/failed/not_run

.. grid:: 2

.. grid-item::

.. needpie:: Test types
:labels: static-code-analysis, structural-statement-coverage, structural-branch-coverage, walkthrough, inspection, interface-test, requirements-based, resource-usage, control-flow-analysis, data-flow-analysis, fault-injection, struct-func-cov, struct-call-cov
:legend:

type == 'testcase' and test_type == 'static-code-analysis'
type == 'testcase' and test_type == 'structural-statement-coverage'
type == 'testcase' and test_type == 'structural-branch-coverage'
type == 'testcase' and test_type == 'walkthrough'
type == 'testcase' and test_type == 'inspection'
type == 'testcase' and test_type == 'interface-test'
type == 'testcase' and test_type == 'requirements-based'
type == 'testcase' and test_type == 'resource-usage'
type == 'testcase' and test_type == 'control-flow-analysis'
type == 'testcase' and test_type == 'data-flow-analysis'
type == 'testcase' and test_type == 'fault-injection'
type == 'testcase' and test_type == 'struct-func-cov'
type == 'testcase' and test_type == 'struct-call-cov'

.. grid-item::

.. needpie:: Derivation techniques
:labels: requirements-analysis, boundary-values, equivalence-classes, fuzz-testing, error-guessing, explorative-testing
:legend:

type == 'testcase' and derivation_technique == 'requirements-analysis'
type == 'testcase' and derivation_technique == 'boundary-values'
type == 'testcase' and derivation_technique == 'equivalence-classes'
type == 'testcase' and derivation_technique == 'fuzz-testing'
type == 'testcase' and derivation_technique == 'error-guessing'
type == 'testcase' and derivation_technique == 'explorative-testing'

.. needtable:: Passed tests
:filter: result == "passed"
:tags: TEST
:columns: name as "testcase";result;fully_verifies;partially_verifies;test_type;derivation_technique;id as "link"

.. needtable:: Failed tests
:filter: result == "failed"
:tags: TEST
:columns: name as "testcase";result;fully_verifies;partially_verifies;test_type;derivation_technique;id as "link"

.. needtable:: Skipped tests
:filter: result != "passed" and result != "failed"
:tags: TEST
:columns: name as "testcase";result;fully_verifies;partially_verifies;test_type;derivation_technique;id as "link"

**7. Test logs**
- Test log per test case with status passed/failed/not_run

@AlexanderLanin AlexanderLanin Aug 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once #566 is merged:

Suggested change
.. document:: Lifecycle Verification Report
:id: doc__lifecycle_verification_report
:status: draft
:version: 1
:safety: ASIL_B
:security: YES
:realizes: wp__verification_module_ver_report
This verification report is based on the :need:`gd_temp__verification_plan`.
It covers all the components of the above stated module.
**1. Verification Coverage**
**1.1. on Requirements**
- Lists of component requirements (incl. AoU satisfied by the component) tested by which test case, passed/failed and completeness verdict
(this shall be generated by tools and accompanied by progress charts to be usable also for project steering)
- For external component Assumptions of Use: coverage by platform safety manual or feature/components incl. test case, passed/failed and completeness verdict
- This is split in a list of QM requirements tested and a separate list of tests for ASIL rated requirements.
- List of component requirements (ASIL rated) linked to inspection checklist and verdict (derived from PR export)
.. grid:: 2
.. grid-item::
.. needpie:: Valid requirements
:labels: not valid, valid but not tested, valid and tested
:colors: red, orange, green
type == 'comp_req' and status == 'invalid'
type == 'comp_req' and testlink == '' and (status == 'valid' or status == 'invalid')
type == 'comp_req' and testlink != '' and (status == 'valid' or status == 'invalid')
.. grid-item::
.. needpie:: Code-linked requirements
:labels: not linked, linked
:colors: red, green
type == 'comp_req' and source_code_link == ''
type == 'comp_req' and source_code_link != ''
.. needtable:: QM requirements
:filter: (type == "aou_req" or type == "comp_req") and status == "valid" and safety == "QM"
:columns: id;title;satisfied_by as "Satisfied by";testlink as "Tested by"
:class: compact-needtable
.. needtable:: ASIL requirements
:filter: (type == "aou_req" or type == "comp_req") and status == "valid" and safety == "ASIL_B"
:columns: id;title;satisfied_by as "Satisfied by";testlink as "Tested by"
:class: compact-needtable
**1.2. on Architecture**
- List of component architecture tags tested by which test case, passed/failed and completeness verdict
(this shall be generated by tools and accompanied by progress charts to be usable also for project steering)
- This is split in a list of QM components tested and a separate list of tests for ASIL rated components.
- List of component architecture tags (ASIL rated) linked to inspection checklist and verdict (derived from PR export)
**1.3. on Detailed Design**
- List of detailed design tags tested by which test case, passed/failed and completeness verdict
(this shall be generated by tools and accompanied by progress charts to be usable also for project steering)
- This is split in a list of QM components tested and a separate list of tests for ASIL rated components.
- List of detailed design tags (ASIL rated) linked to inspection checklist and verdict (derived from PR export)
- The lists may also contain other verification methods like "Analysis" - process tbd
**2. DFA Report**
- List of the performed component DFA, pass/fail with open mitigations
**3. Safety Analysis Report**
- List of the performed component Safety Analysis, pass/fail with open mitigations
**4. Unit Verification Coverage**
**4.1. Structural Coverage**
- List of the units with C0(line) and C1(branch) coverage absolute and percentage
(this shall be generated by tools and accompanied by progress charts to be usable also for project steering)
- List includes a column for the safety rating of each unit.
**4.2. Static Code Analysis**
- List of the units with compiler warning numbers and coding rule violation numbers
(this shall be generated by tools and accompanied by progress charts to be usable also for project steering)
- List includes a column for the safety rating of each unit.
**4.3. Manual Code Inspection**
- List of components (ASIL rated) linked to inspection checklist and verdict (derived from PR export)
**5. Software component qualification verification report**
- Contains the needed verification results if for some components a qualification of pre-developed SW is performed.
**6. Test results**
- Test result per test case with status passed/failed/not_run
.. grid:: 2
.. grid-item::
.. needpie:: Test types
:labels: static-code-analysis, structural-statement-coverage, structural-branch-coverage, walkthrough, inspection, interface-test, requirements-based, resource-usage, control-flow-analysis, data-flow-analysis, fault-injection, struct-func-cov, struct-call-cov
:legend:
type == 'testcase' and test_type == 'static-code-analysis'
type == 'testcase' and test_type == 'structural-statement-coverage'
type == 'testcase' and test_type == 'structural-branch-coverage'
type == 'testcase' and test_type == 'walkthrough'
type == 'testcase' and test_type == 'inspection'
type == 'testcase' and test_type == 'interface-test'
type == 'testcase' and test_type == 'requirements-based'
type == 'testcase' and test_type == 'resource-usage'
type == 'testcase' and test_type == 'control-flow-analysis'
type == 'testcase' and test_type == 'data-flow-analysis'
type == 'testcase' and test_type == 'fault-injection'
type == 'testcase' and test_type == 'struct-func-cov'
type == 'testcase' and test_type == 'struct-call-cov'
.. grid-item::
.. needpie:: Derivation techniques
:labels: requirements-analysis, boundary-values, equivalence-classes, fuzz-testing, error-guessing, explorative-testing
:legend:
type == 'testcase' and derivation_technique == 'requirements-analysis'
type == 'testcase' and derivation_technique == 'boundary-values'
type == 'testcase' and derivation_technique == 'equivalence-classes'
type == 'testcase' and derivation_technique == 'fuzz-testing'
type == 'testcase' and derivation_technique == 'error-guessing'
type == 'testcase' and derivation_technique == 'explorative-testing'
.. needtable:: Passed tests
:filter: result == "passed"
:tags: TEST
:columns: name as "testcase";result;fully_verifies;partially_verifies;test_type;derivation_technique;id as "link"
.. needtable:: Failed tests
:filter: result == "failed"
:tags: TEST
:columns: name as "testcase";result;fully_verifies;partially_verifies;test_type;derivation_technique;id as "link"
.. needtable:: Skipped tests
:filter: result != "passed" and result != "failed"
:tags: TEST
:columns: name as "testcase";result;fully_verifies;partially_verifies;test_type;derivation_technique;id as "link"
**7. Test logs**
- Test log per test case with status passed/failed/not_run
.. document:: Lifecycle Verification Report
:id: doc__lifecycle_verification_report
:status: draft
:version: 1
:safety: ASIL_B
:security: YES
:realizes: wp__verification_module_ver_report
:post_template: module_verification_report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@AlexanderLanin AlexanderLanin AlexanderLanin left review comments
@NicolasFussberger NicolasFussberger Awaiting requested review from NicolasFussberger NicolasFussberger is a code owner
@pawelrutkaq pawelrutkaq Awaiting requested review from pawelrutkaq pawelrutkaq is a code owner
@ramceb ramceb Awaiting requested review from ramceb ramceb is a code owner
@MaciejKaszynski MaciejKaszynski Awaiting requested review from MaciejKaszynski MaciejKaszynski is a code owner
@FScholPer FScholPer Awaiting requested review from FScholPer FScholPer is a code owner
@antonkri antonkri Awaiting requested review from antonkri antonkri is a code owner
@anmittag anmittag Awaiting requested review from anmittag anmittag is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

LCM - Lifecycle & Health FT
Status: Backlog

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@danth @AlexanderLanin @MaciejKaszynski