Protocol conformance testing with TTCN-3
Zephyr’s network protocols are covered from two directions. The tests under tests/net exercise the implementation from the inside, in C, built into the same image. Conformance suites written in TTCN-3 come at it from the outside: they speak the protocol over a real network interface, and check what Zephyr sends against what the standard requires.
The two catch different things. A test written against the implementation tends to encode what the implementation does. A suite written against the standard does not know what the implementation does, which is the point.
TTCN-3 is a language standardised by ETSI for writing tests. The suites here
are compiled with Eclipse Titan, an open source TTCN-3 compiler, and are
kept in the net-tools repository under ttcn3, alongside the other
host side tools used for network testing.
How it fits together
The Zephyr side of a conformance test is only the system under test: an ordinary application, configured to enable the protocol being tested. Nothing about the test is compiled into it, there is no control channel, and the suite drives it entirely over the network.
Those applications, and the harness that runs a suite against them, live under tests/net/conformance. Twister builds and starts the application, and a small pytest harness builds the suite with Titan, runs it, and turns Titan’s verdict into a test result.
Each test skips itself when Titan, the third party TTCN-3 modules or the network interface is missing, so the suites are harmless in a run that has not been set up for them. See Running the conformance suites for what a run needs, and How the conformance tests are put together for how the parts are put together.
The suites
Which interface a suite uses, and whether it has to be run as root, follow from what it does: a suite that works below the IP layer reads frames from a packet socket on a link of its own.
Suite |
System under test |
Interface |
Runs as |
|---|---|---|---|
|
any user |
||
|
any user |
||
|
any user |
||
|
any user |
||
|
any user |
||
|
any user |
||
|
root |
||
|
root |
||
|
root |
||
|
root |
||
|
root |
Adding a suite is described in Adding a suite.
Known gaps
Where a suite asserts behaviour that does not match the standard, it says so at the point the assertion is made, so that the divergence is recorded rather than frozen in silently. What follows is the other kind of gap: ground no suite covers yet.
DNS-SD legacy unicast queries
The hostname side of the mDNS responder answers a legacy unicast query the way RFC 6762 section 6.7 asks. The service discovery side does not: it builds its own messages, sets the cache flush bit on the records that belong to one instance, uses the lifetimes it would have used for a multicast answer, and echoes neither the identifier nor the question. Fixing it means reworking name compression offsets that are all computed from a fixed header size.
The dnssd suite records this rather than asserting the standard, in
f_check_legacy_shape, so that a test does not sit failing until somebody
gets to it. Each check there says what would have to change with it.
MQTT 5.0, packet identifiers and re-sending
The mqtt suite covers MQTT 3.1.1. Zephyr also implements MQTT 5.0
(CONFIG_MQTT_VERSION_5_0), and the Titan project publishes no
protocol module for it, so covering it would mean writing the message types
before writing any test.
Packet identifiers and re-sending are not covered either. Zephyr’s client
leaves both to the application: mqtt_publish() sends the identifier and
the duplicate flag it is given, so a test of either would test the system under
test’s own counter rather than the client.
CoAP block transfer and observe
TD_COAP_BLOCK_01 and TD_COAP_OBS_01 are not run. They address
/large and /obs, and the application provides only /test; against
it the observe case waits for notifications that never arrive and the run does
not finish. Adding those two resources is the obvious next step.
Overlapping DNS queries
The resolver renews its source port before sending to a server that has nothing
outstanding, which with the default of one query at a time means every query.
Queries that overlap on one server still share a port, so the check in the
dns suite would not catch a regression in that case. See RFC 5452
section 9.2.
Continuous integration
These suites are not part of the ordinary Twister run: they need a Titan installation, a network interface facing the device, and a checkout of the suites, none of which a normal build has. They are also slow, and they cannot run at the same time as each other.
They run nightly instead, from
.github/workflows/net_conformance.yml, which can also be started
by hand from the Actions tab. The job installs the packaged Titan and sets
TTCN3_DIR=/usr, brings up both interfaces in a container holding
NET_ADMIN, and runs the whole directory as root so that no suite is
skipped. The Twister report and the harness logs are kept as artifacts.
Other TTCN-3 suites
The Eclipse Titan project publishes protocol modules and test ports for a large number of protocols, as separate repositories under gitlab.eclipse.org/eclipse/titan. The suites here build against those rather than defining their own message formats.
Some complete suites exist there too. titan.misc contains a CoAP
conformance suite that can be run against the CoAP service
sample; see CoAP for that one.
An older TTCN-3 suite for TCP, written at Intel for the TCP rewrite, informed
what the tcp suite covers, but none of its code is used. It drove Zephyr
through a JSON control channel and asserted the stack’s internal state names,
and the option that channel needed has been removed; see the 4.5 migration
guide.