feat: TC8 SOME/IP conformance test suite for someipd by jorgecasal · Pull Request #60 · eclipse-score/inc_someip_gateway

jorgecasal · 2026-03-30T13:13:43Z

Summary

Adds a TC8 SOME/IP conformance test suite for someipd, covering the OA TC8 ECU Test Specification. The suite runs under the Integration Test Framework (ITF): pytest runs on the host and drives someipd inside a QEMU guest (Linux or QNX8) over SSH on a TAP bridge, while the host sends and receives raw SOME/IP UDP/TCP to the device under test. A --tc8-standalone flag lets someipd offer services directly without the full gateway stack.

Scope: 185 test cases across 10 test modules, plus documentation and CI integration.

Scope of --tc8-standalone: These conformance tests validate someipd wire-level SOME/IP behavior only. They do not exercise the full mw::com to gatewayd to someipd middleware path. End-to-end verification of that path is out of scope for this suite.

What's included

Tests (tests/tc8_conformance/)

Service Discovery, SOME/IP message format, event notification, field conformance, and multi service coverage
Per target SOME/IP packet capture written to someip_capture.pcap via tcpdump on the host TAP interface
3 tests marked xfail for known vsomeip 3.6.1 limitations (2 in message format, 1 in service discovery)

Documentation (docs/)

Architecture, requirements, test specification, and requirement to test traceability for the conformance suite

someipd (src/someipd/main.cpp)

--tc8-standalone flag offers test services directly without gatewayd IPC
Uses ET_FIELD so vsomeip sends the initial field value on subscribe

Build and CI

Runs via bazel test --config=tc8-itf //tests/tc8_conformance/... on Linux and bazel test --config=tc8-itf-qnx //tests/tc8_conformance/... on QNX8
New //tests/common:capture helper library and @score_itf development dependency
json_schema_validator upgraded to 2.4.0; unused jsonschema Python dependency removed
tests/integration/BUILD.bazel migrated from py_pytest to score_py_pytest
.pre-commit-config.yaml large file limit raised from 50KB to 125KB for TC8 assets

Wire-level pytest tests verifying someipd against the OA TC8 ECU Test Specification v3.0 Chapter 5: service discovery (phases, reboot, timing), message format, event notification, field lifecycle, and TCP transport. - someipd --tc8-standalone mode: request/response, field GET/SET, UDP and TCP events; offer_event uses ET_FIELD so vsomeip delivers the cached field value to new subscribers immediately on subscribe (is_field in JSON config is not honoured in programmatic offer_event calls) - vsomeip config templates for SD timing and service/event/field/TCP tests, with JSON Schema validation at lint time - Architecture doc, requirements, test specification and OA spec traceability - Bazel port isolation via the env attribute: each target receives unique SD and service ports enabling medium targets to run in parallel; timing-sensitive and reboot lifecycle targets retain the exclusive tag for serial execution

github-actions · 2026-03-30T13:17:40Z

License Check Results

🚀 The license check job ran with the Bazel command:

bazel run //:license-check

Status: ⚠️ Needs Review

Click to expand output

[License Check Output]
Extracting Bazel installation...
Starting local Bazel server (8.6.0) and connecting to it...
INFO: Invocation ID: c027d17c-d9f8-435d-b92f-72cc460b730e
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Computing main repo mapping: 
Loading: 
Loading: 2 packages loaded
WARNING: Target pattern parsing failed.
ERROR: Skipping '//:license-check': no such target '//:license-check': target 'license-check' not declared in package '' defined by /home/runner/work/inc_someip_gateway/inc_someip_gateway/BUILD
ERROR: no such target '//:license-check': target 'license-check' not declared in package '' defined by /home/runner/work/inc_someip_gateway/inc_someip_gateway/BUILD
INFO: Elapsed time: 14.356s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target

github-actions · 2026-03-30T13:23:01Z

The created documentation from the pull request is available at: docu-html

- Raise check-added-large-files limit from 50 KB to 125 KB to accommodate large but legitimate TC8 test modules and the RST test specification document - Add REUSE.toml annotation for tests/tc8_conformance/config/*.json so the four vsomeip JSON config files pass reuse-lint-file - Prepend Apache-2.0 SPDX comment headers to the two TC8 README.md files that were missing copyright notices

TC8 conformance tests require special setup (multicast route, env vars) and already run in a dedicated step; avoid double-execution via -tc8 filter.

mikehaller · 2026-03-30T14:35:28Z

Documentation overall looks very good, reviewed the current state at https://eclipse-score.github.io/inc_someip_gateway/pr-60/architecture/tc8_conformance_testing.html - not in detail, but general sections and impression. Nice!

On my local checkout, running the tests works with the expected failures:

Executed 10 out of 10 tests: 10 fail locally
3 failed, 2 skipped in 59.50s

$ time bazel test --test_tag_filters=tc8 --test_env=TC8_HOST_IP=127.0.0.1 //tests/tc8_conformance/...

Result:

FAILED ../score_tooling+/python_basics/score_pytest::TestSDClientStopSubscribe::test_ets_084_stop_subscribe_ceases_events - AssertionError: ETS_084: Prerequisite failed — no SubscribeAck received
FAILED ../score_tooling+/python_basics/score_pytest::TestSDClientReboot::test_ets_081_reboot_flag_set_after_first_restart - AssertionError: ETS_081: No SD messages captured after restart
FAILED ../score_tooling+/python_basics/score_pytest::TestSDClientReboot::test_ets_082_reboot_flag_set_after_second_restart - AssertionError: ETS_082: No SD messages captured after second restart

real 6m16.073s

mikehaller · 2026-03-30T15:43:34Z

I missed setting up the multicast... now the tests run fine locally as well, can confirm.

# Set up the multicast route locally
$ sudo ip route add 224.0.0.0/4 dev lo

# Re-run the tests
$ time bazel test --test_tag_filters=tc8 --test_env=TC8_HOST_IP=127.0.0.1 //tests/tc8_conformance/...
INFO: Analyzed 13 targets (0 packages loaded, 0 targets configured).
INFO: Found 3 targets and 10 test targets...
INFO: Elapsed time: 214.786s, Critical Path: 182.28s
INFO: 11 processes: 2 action cache hit, 1 internal, 10 linux-sandbox.
INFO: Build completed successfully, 11 total actions
//tests/tc8_conformance:tc8_event_notification                           PASSED in 24.6s
//tests/tc8_conformance:tc8_field_conformance                            PASSED in 3.8s
//tests/tc8_conformance:tc8_message_format                               PASSED in 42.4s
//tests/tc8_conformance:tc8_multi_service                                PASSED in 10.0s
//tests/tc8_conformance:tc8_sd_client                                    PASSED in 19.0s
//tests/tc8_conformance:tc8_sd_format                                    PASSED in 49.3s
//tests/tc8_conformance:tc8_sd_phases_timing                             PASSED in 7.2s
//tests/tc8_conformance:tc8_sd_reboot                                    PASSED in 4.1s
//tests/tc8_conformance:tc8_sd_robustness                                PASSED in 5.6s
//tests/tc8_conformance:tc8_service_discovery                            PASSED in 181.6s

Executed 10 out of 10 tests: 10 tests pass.
There were tests whose specified size is too big. Use the --test_verbose_timeout_warnings command line option to see which ones these are.

real    3m34.876s
user    0m0.203s
sys     0m0.305s

mikehaller

As the TC8 Conformity Tests is a complete new part for someip gateway, I'll approve the current state of the PR. It's a very good step forward.

There is one issue regarding the main.cpp where I didn't fully grasp the plans and hence would accept it if it's just temporary.

- Use local json_schema_validator rule instead of @score_communication - Add integrity hashes to download_archive deps for reproducible builds - Add component descriptions for gatewayd and someipd in architecture docs

…C8 suite (eclipse-score#213) First in a 9-PR stack splitting eclipse-score#60 into reviewable slices. No test code yet, pure Bazel infrastructure: - Add `@score_itf` dev dep and upgrade `json_schema_validator` to 2.4.0 in `MODULE.bazel` - Add TC8 Bazel configs to `.bazelrc` and new `bazel/tools/` macros - Add `third_party/json_schema_validator/` BUILD rule - Minor BUILD adjustments across `score/` and `tests/benchmarks` - Raise pre-commit large-file limit to 125 KB; add REUSE entries for TC8 assets ## Stack | # | Branch | Scope | |---|--------|-------| | **1 (this)** | `jorgecasal/tc8-build-infra` | Bazel infra + deps | | 2 | `jorgecasal/tc8-someipd-standalone` | someipd `--tc8-standalone` flag | | 3 | `jorgecasal/tc8-itf-migration` | tests/common + ITF migration | | 4 | `jorgecasal/tc8-infra` | TC8 helpers, configs, conftest | | 5 | `jorgecasal/tc8-sd-core` | SD core tests | | 6 | `jorgecasal/tc8-sd-compliance` | SD format/reboot/robustness tests | | 7 | `jorgecasal/tc8-msg-format` | SOME/IP message format tests | | 8 | `jorgecasal/tc8-event-field` | Event/field/multi-service tests | | 9 | `jorgecasal/tc8-ci-docs` | CI workflows + documentation |

Remove TC8 mode from production someipd; introduce score/someipd_tc8 as a dedicated QM binary used exclusively by the conformance test suite.

Point BUILD and lifecycle helper at tc8_dut; add tc8_dut_config.json; align vsomeip config IDs with ETS spec; trim conftest docstrings.

…ts.py Replace per-file private constants with shared imports aligned to ETS spec values. Remove special characters from comments and strings throughout tc8_conformance.

Add TC8_SERVICE_ID and TC8_INSTANCE_ID substitution to config rendering in tc8_itf_conftest and dut_lifecycle. Fix routing manager name in tc8_someipd_multi.json from 'someipd' to 'tc8_dut'.

# Conflicts: # .bazelrc # BUILD # docs/requirements/stakeholder.rst # docs/tc8_conformance/requirements.rst # score/someipd/main.cpp # tests/tc8_conformance/helpers/__init__.py

…tc8 requirements

github-actions · 2026-08-10T08:57:26Z

Documentation preview for this pull request is available at:
pr-60: https://eclipse-score.github.io/inc_someip_gateway/pr-60/

…dirs

…ore#242) This is PR 2 of 9 in the TC8 conformance test suite stack (split from eclipse-score#60). Depends on: eclipse-score#213 (jorgecasal/tc8-build-infra), now merged into main. Note: Replaces the previously proposed PR eclipse-score#214 (jorgecasal/tc8-someipd-standalone), which was closed. The score/someipd_tc8/ standalone binary approach was replaced by the ETS stub described below. ## Changes score/someipd/main.cpp: migrate logging from std::cout to score::mw::log. No logic change. tests/tc8_conformance/application/ -- new ETS stub DUT: - shared/tc8_ets_service.h: service constants and event descriptors used by the stub - ets_stub/tc8_ets_stub.cpp: binary that offers tc8_service via mw::com GenericSkeleton and sends periodic event notifications for TC8 test capture - ets_stub/config/tc8_ets_stub_mw_com_config.json: mw::com manifest for the stub The ETS stub is the DUT application for TC8 conformance testing. It offers tc8_service via mw::com GenericSkeleton. gatewayd and someipd still run in the test setup and bridge that service to the SOME/IP network so the TC8 ETS can connect to it. Compared to the previous approach (a standalone someipd_tc8 binary), the ETS stub sits at the application layer and has no direct dependency on vsomeip or the IPC bridge internals. ## Stack | PR | Branch | Status | |----|--------|--------| | 1 | jorgecasal/tc8-build-infra | merged | | 2 | jorgecasal/tc8-ets-stub-dut | this PR | | 3 | jorgecasal/tc8-itf-migration | pending | | 4 | jorgecasal/tc8-infra | pending | | 5 | jorgecasal/tc8-sd-core | pending | | 6 | jorgecasal/tc8-sd-compliance | pending | | 7 | jorgecasal/tc8-msg-format | pending | | 8 | jorgecasal/tc8-event-field | pending | | 9 | jorgecasal/tc8-ci-docs | pending | ## Review focus - tc8_ets_stub.cpp: does the mw::com GenericSkeleton offer/notify flow correctly implement the event patterns expected by TC8 tests? - tc8_ets_service.h: are the event descriptors and DataTypeMetaInfo correct? - tc8_ets_stub_mw_com_config.json: are service ID, instance ID, and event IDs consistent with the TC8 test configs? - score/someipd/main.cpp: logging migration only, verify no logic regression.

github-actions · 2026-09-10T06:41:06Z

Coverage report for this pull request:

Status: failure (workflow run)
Coverage data: coverage-report

The archive contains the HTML report, the raw LCOV data and the JUnit XML test results.

jorgecasal requested a review from mikehaller March 30, 2026 13:14

jorgecasal changed the title ~~Add TC8 SOME/IP conformance test suite for someipd~~ RFC: TC8 SOME/IP conformance test suite for someipd Mar 30, 2026

jorgecasal added documentation Improvements or additions to documentation github_actions Pull requests that update GitHub Actions code python Pull requests that update python code labels Mar 30, 2026

jorgecasal added 2 commits March 30, 2026 15:28

Exclude TC8 tests from general test sweep

a0e345b

TC8 conformance tests require special setup (multicast route, env vars) and already run in a dedicated step; avoid double-execution via -tc8 filter.

jorgecasal added the enhancement New feature or request label Mar 30, 2026

jorgecasal force-pushed the jorgecasal_tc8_conformance_test_infrastructure branch from c507e28 to 63d628d Compare March 30, 2026 14:16

Mark known vsomeip 3.6.1 TC8 failures as xfail to unblock CI.

8420326

jorgecasal force-pushed the jorgecasal_tc8_conformance_test_infrastructure branch from 63d628d to 8420326 Compare March 30, 2026 14:17

jorgecasal added 2 commits March 30, 2026 16:26

Update workflow step name for TC8 conformance tests.

82388be

Fix small inconsistency in docs/tc8_conformance/index.rst

7639458

jorgecasal requested review from NEOatNHNG, lurtz and mariuswbr March 30, 2026 14:36

jorgecasal commented Mar 30, 2026

View reviewed changes

Comment thread tests/tc8_conformance/BUILD.bazel Outdated

Update CI/CD Integration docs to reflect current workflow

a549727

mikehaller approved these changes Mar 30, 2026

View reviewed changes

Comment thread tests/tc8_conformance/README.md Outdated

Comment thread score/someipd/main.cpp

jorgecasal changed the title ~~RFC: TC8 SOME/IP conformance test suite for someipd~~ TC8 SOME/IP conformance test suite for someipd Mar 31, 2026

jorgecasal commented Mar 31, 2026

View reviewed changes

Comment thread MODULE.bazel Outdated

jorgecasal added 2 commits March 31, 2026 07:39

Clarify SOME/IP server/client terminology in TC8 test spec.

146c350

Update known limitations table to reflect xfail markers.

9f84116

jorgecasal self-assigned this Mar 31, 2026

Align TC8 conformance with upstream main after merge

2dfcff0

- Use local json_schema_validator rule instead of @score_communication - Add integrity hashes to download_archive deps for reproducible builds - Add component descriptions for gatewayd and someipd in architecture docs

jorgecasal added 10 commits August 4, 2026 11:38

feat(tc8): extract TC8 DUT as standalone someipd_tc8 binary

e23926c

Remove TC8 mode from production someipd; introduce score/someipd_tc8 as a dedicated QM binary used exclusively by the conformance test suite.

feat(tc8): update conformance test infra for standalone DUT binary

f94820e

Point BUILD and lifecycle helper at tc8_dut; add tc8_dut_config.json; align vsomeip config IDs with ETS spec; trim conftest docstrings.

refactor(tc8): centralize service/event/method IDs in helpers/constan…

35388d6

…ts.py Replace per-file private constants with shared imports aligned to ETS spec values. Remove special characters from comments and strings throughout tc8_conformance.

fix(tc8): fix config template substitution and routing manager name

8fc1735

Add TC8_SERVICE_ID and TC8_INSTANCE_ID substitution to config rendering in tc8_itf_conftest and dut_lifecycle. Fix routing manager name in tc8_someipd_multi.json from 'someipd' to 'tc8_dut'.

refactor(tc8): replace nlohmann/json with score::json in dut_config

050cce1

fix(tc8): update tc8 conformance test infrastructure

064391a

docs(tc8): update architecture doc for someipd_tc8 binary

c25b88d

Merge branch 'main' into jorgecasal_tc8_conformance_test_infrastructure

79f18db

# Conflicts: # .bazelrc # BUILD # docs/requirements/stakeholder.rst # docs/tc8_conformance/requirements.rst # score/someipd/main.cpp # tests/tc8_conformance/helpers/__init__.py

fix(docs): remove invalid satisfies field and add missing version in …

af1755c

…tc8 requirements

fix: add missing newline at end of tc8_dut_config.json

c529182

jorgecasal and others added 13 commits August 10, 2026 11:01

refactor(tc8): simplify and neutralize comments in dut_service.cpp

1db6dc7

Merge branch 'main' into jorgecasal_tc8_conformance_test_infrastructure

335f576

refactor(tc8): remove someipd_tc8 DUT binary and orphan config

6e28ae6

feat(someipd): add eventgroup_ids to event config and routing

8f08567

refactor(tc8): restructure application into shared/ and ets_stub/ sub…

cc059c0

…dirs

feat(tc8): add and update ETS stub config files

346ca05

feat(tc8): update test infrastructure for ETS stub DUT

e7fb7d3

docs(tc8): update architecture doc for ETS stub DUT

47a07e3

fix(reuse): merge tc8_conformance JSON paths into shared annotation

03206c8

Merge branch 'main' into jorgecasal_tc8_conformance_test_infrastructure

30b9192

refactor(tc8): rename and deduplicate DUT lifecycle helpers

cc43cd2

refactor(tc8): align InstanceSpecifier pattern with mw::com tutorial

0ce2f5e

refactor(someipd): remove orphaned section comments from main.cpp

f71baeb

jorgecasal mentioned this pull request Aug 13, 2026

feat(tc8): add ETS stub DUT and someipd logging migration #242

Merged

merge main into jorgecasal_tc8_conformance_test_infrastructure

ed44e4b

fix CI failures introduced by main merge

4875b42

Navigation Menu

Uh oh!

feat: TC8 SOME/IP conformance test suite for someipd - #60

feat: TC8 SOME/IP conformance test suite for someipd#60
jorgecasal wants to merge 79 commits into
eclipse-score:mainfrom
etas-contrib:jorgecasal_tc8_conformance_test_infrastructure

jorgecasal commented Mar 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Mar 30, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Mar 30, 2026

Uh oh!

mikehaller commented Mar 30, 2026

Uh oh!

Uh oh!

mikehaller commented Mar 30, 2026

Uh oh!

mikehaller left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 10, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

jorgecasal commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What's included

Uh oh!

github-actions Bot commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

License Check Results

Uh oh!

github-actions Bot commented Mar 30, 2026

Uh oh!

mikehaller commented Mar 30, 2026

Uh oh!

Uh oh!

mikehaller commented Mar 30, 2026

Uh oh!

mikehaller left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Aug 10, 2026

Uh oh!

github-actions Bot commented Sep 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jorgecasal commented Mar 30, 2026 •

edited

Loading

github-actions Bot commented Mar 30, 2026 •

edited

Loading