Docs Test Architecture
Test Architecture
The test tiers, what each one asserts, how to regenerate its fixtures, and the self-enforcing gate tests.
On this page
The tests are of three kinds:
- Over two hundred integration-test binaries under
tests/.- Every
tests/*.rsfile is one. - That is because nothing turns Cargo’s auto-discovery off.
- Every
- Unit tests inside
src/. - Doctests.
The exact total is not repeated here on purpose.
- The homepage stat card carries it.
- A pre-commit gate holds it.
- So any number written on this page would be wrong within a week.
- The opening sentence read “fifty” until 2026-09-06.
- On that date a sweep of these pages counted 238.
What follows is the map:
- What each tier asserts.
- How to regenerate its artifacts.
- Most importantly, the roster of gate tests.
- These fail you for reasons that have nothing to do with the code you were writing.
Tiers
| Tier | Files | Asserts |
|---|---|---|
| CLI surface | cli_test, cli_options_test, cli_help_test, cli_defaults_test, cli_flag_behavior_test | Parsing, the default value of every parameter, --help grouping across ~160 flags, and behavior for flags that were once untested. |
| CLI golden | cli_goldens | trycmd goldens under tests/cli/ — exact stdout/stderr for a command line. tests/cli/cookbook/ pins the output of every cookbook command check-cookbook.py executes, one case per distinct command, run in a scratch directory because recipes write files. |
| Integration | integration_test, capture_test, rtp_integration_test, pipeline_test, parse_path_test, bootstrap_test, app_servers_test, config_test | Capture-to-output pipeline, and the library facades WS2 extracted from main.rs — bootstrap planning is a pure Cli + Config → RunPlan function precisely so these tests can reach it. config_test covers the step before planning: config discovery (-f, SIPNAB_CONFIG, --no-config, unknown-key warnings, the missing-file error) driven through the real binary’s --dump-config. |
| TUI | tui_snapshot_test, tui_state_test, tui_e2e_test | Rendered buffers via ratatui’s TestBackend + insta; state-machine transitions; and end-to-end drives of the real binary inside tmux. See TUI testing. |
| Servers | api_test, api_token_test, mcp_stdio_test, mcp_http_test, mcp_token_test, mcp_token_rotation_test, metrics_test, hep_test | REST, MCP (stdio and HTTP), signed-token auth and rotation, Prometheus scrapes, HEP ingestion — all end to end against a spawned process. |
| Security | security_test, privilege_drop_test, resource_bounds_test, crash_test | Audit regressions, the never-continue-as-root guarantee, attacker-keyed map caps, and crash-report handling with the real binary. |
| Property & fuzz | property_test, smoke_fuzz_test, fuzz_corpus_replay | proptest invariants; the always-on stable-toolchain fuzz floor; and — despite its name — a deterministic sweep over an adversarial seed set defined in the file itself, not a replay of fuzz/corpus/ (it opens no files). |
| Contract | json_schema_test, summary_consistency_test, output_behavior_test, error_types_test, api_guidelines_test, wasm_exports_test | The shapes other people’s code depends on: JSON schemas, cross-surface summary agreement, machine-readable output flags, typed errors, #[non_exhaustive] on growth-prone enums, WASM export list. |
| Docs enforcement | docs_drift_test, dev_docs_drift_test, link_integrity_test, doc_example_coverage_test, config_examples_test, site_journey_test, mockup_alignment_test | See the gate roster below. |
| Governance | flag_coverage_test, keybinding_drift_test, feature_gate_test, config_wiring_test | Rules about how the project grows, not about a specific behavior. |
| Meta | support_selftest | The shared test helpers have their own tests. |
tests/support/
Files in a subdirectory of tests/ are not compiled as their own test
binaries. So an explicit path attribute pulls shared helpers in:
#[path = "support/run.rs"]
mod run;
That idiom is why the same module appears in several test files
with no mod.rs chain. Each test binary compiles its own copy.
| Helper | Provides |
|---|---|
mod.rs | Shared normalization (normalize()) used to compare output across platforms; self-tested by support_selftest. |
run.rs | The canonical binary-spawn helper for CLI/output/config integration tests — one place that knows how to find and invoke the built binary. |
server.rs | REST API spawn harness: start the server on an ephemeral port, wait for readiness, tear down. |
mcp.rs | The same for HTTP MCP, including the JSON-RPC framing. |
teardown.rs | terminate(), the one way a harness stops the binary it spawned: SIGTERM, a bounded wait, and SIGKILL only for a child still running after it. Child::kill() is SIGKILL, and a process killed that way never writes its coverage profile. Pulled in with include! rather than #[path]. |
schema.rs | JSON-Schema validation against tests/schemas/. |
tui_fixtures.rs | SIP fixture builders shared by the TUI snapshot and state tests. |
executable.rs | write_executable(), the one way a test writes a file it then runs: a child process writes it, so this process never holds it open for writing and an exec cannot fail with ETXTBSY (“Text file busy”) while another test thread forks. executable_stub_test refuses a file written in-process and then made executable. |
fuzz.rs | A deterministic xorshift PRNG shared by the stable-toolchain fuzzers, so a failure reproduces from its seed. |
ports.rs | refused_tcp_port(), the one way a test gets a loopback TCP port where a connect fails with ConnectionRefused: a socket bound and not listening, held until the guard drops, so no test running alongside can bind the number. Binding :0, reading the port, and closing the socket leaves the number free for another test to listen on; released_port_scan_test refuses that pattern under tests/. |
Fixtures and corpora
Every command below ran against the current tree and left it unchanged.
- That is the point of documenting them.
- If one produces a diff, something has drifted.
| Artifact | Regenerate with |
|---|---|
tests/fixtures/ — capture fixtures | Every capture in the directory comes from tests/support/synthetic_captures.rs, and so do the two SIPp media files in harness/sipp/scenarios/ and the two fuzz seeds in fuzz/corpus/pcap_reader/ named truncated-sip and empty-classic: seventeen in all. cargo run --features native --bin gen_fixture writes them, relative to the CARGO_MANIFEST_DIR Cargo sets (the current directory when run outside Cargo), and synthetic_captures_test fails when a committed byte differs from what the generator builds. The directory’s other fixtures, the JSON goldens, ng replies and TLS keys, stay as checked in. tests/PROVENANCE.md records where every capture outside tests/pcap-samples/ came from, and every_committed_capture_is_public_or_synthetic refuses one with no entry, finding captures by their leading bytes rather than their names. |
tests/snapshots/ — TUI buffers | cargo insta test --features tui --accept (needs cargo install cargo-insta) |
tests/cli/ — trycmd goldens | TRYCMD=overwrite cargo test --features full --test cli_goldens |
tests/cli/cookbook/ — cookbook output goldens | python3 scripts/check-cookbook.py --bless writes a command-only case for each executed cookbook command that has none and deletes each case no command produces; then TRYCMD=overwrite cargo test --features full --test cli_goldens records the output. A case’s file name is a hash of its command, and it is also the directory the command’s output files go to. |
tests/schemas/ — JSON Schemas | Hand-maintained; json_schema_test validates output against them. |
tests/pcap-samples/ — capture fixtures | Thirteen are synthetic: python3 tests/gen-pcap-samples.py writes ten and python3 tests/gen-link-type-samples.py writes the three link-layer framings (DLT_LOOP, PPPoE inside Linux cooked capture v1 and v2); each script in check mode reports any of its own that have drifted from it. Nothing regenerates the rest; they stay as checked in. New ones arrive through harness/scripts/capture.sh then promote.sh, which record the run in PROVENANCE.md beside them — every_committed_capture_fixture_says_where_it_came_from refuses a fixture with no entry, and the 36 that predate that rule sit on an enumerated list which only shrinks. |
tests/install-sh/ — installer cases | Hand-maintained; exercised by the install-sh CI job. |
fuzz/corpus/ — fuzz seeds | Grown by cargo fuzz run (nightly). Note nothing on the stable toolchain reads this directory: fuzz_corpus_replay drives its own in-file seed set. To make a reproducer run in every cargo test, add it to smoke_fuzz_test as well. |
The public vCon datasets
The vcon-dev community publishes five vCon datasets as git repositories.
- They hold 17,443 containers in all.
- None of them carries SIP signaling.
tests/fixtures/vcon-datasets/PINS.tsvpins each one to a commit.- It records the license of its LICENSE file at that commit.
Two tests read them.
vcon_dataset_subset_testruns on every build with thevconfeature.- It reads the four containers committed
under
tests/fixtures/vcon-datasets/. - It pins the findings
vcon_schema::validatereports for each. - It also checks three more things:
- The
jsonschemareference engine agrees. - Each container is valid once the test repairs its recorded dataset issues.
- The fetcher (
--vcon-fetch-kind generic, against a server on127.0.0.1) saves each one unchanged.
- The
- Each dataset directory there carries the LICENSE file of its dataset.
- It also carries a README with the source path and SHA-256 of every file.
- It reads the four containers committed
under
vcon_dataset_corpus_testreads every container in the five datasets.- It runs only when
SIPNAB_VCON_DATASETSnames a directory thatscripts/fetch-vcon-datasets.pyfilled. - Without it, the binary prints one
NOTICEline saying its gates did not run, as the capture corpus does. - The pre-push corpus gate runs it, with the shell’s
SIPNAB_VCON_DATASETS, whenever that gate runs. The pre-commit test pool runs with the variable unset.
- It runs only when
Fetch the datasets and run the corpus test:
# Run all of these, in order.
python3 scripts/fetch-vcon-datasets.py "$HOME/vcon-datasets"
SIPNAB_VCON_DATASETS="$HOME/vcon-datasets" \
cargo test --features full --profile corpus --test vcon_dataset_corpus_test
The script runs only git.
- It fetches each pinned commit.
- It checks it out detached, with hooks and symbolic links turned off.
- It refuses a checkout whose HEAD is not the pinned commit.
- It refuses a checkout whose files have changed.
- A second run verifies the cache and moves a dataset whose pin changed.
- The five datasets take about 2.3 GB of disk.
- The corrected copy described below takes about 2.2 GB more.
The corpus test prints three things for each dataset:
- The number of containers.
- The number the schema accepts.
- Each finding, with the number of containers that carry it.
It asserts these things:
- Each dataset is at its pinned commit.
- sipnab and the reference agree on every container.
- They agree on the verdict.
- They agree on the instance paths of the findings.
- The per-dataset counts measured on 2026-10-09.
- These hold for as long as the pins do.
- Moving a pin in
PINS.tsvmeans measuring again and rewritingEXPECTEDin the test.
What the five datasets get, at their pins:
| Dataset | Containers | Valid | Findings, with the number of containers that carry each |
|---|---|---|---|
vcon-supreme-court-arguments | 8,503 | 0 | Attachment Object without mediatype: 8,503 |
ietf-meeting-vcons | 8,181 | 0 | Attachment Object without mediatype: 8,181. Dialog Object with url and no content_hash: 4,079 |
vcon-dataset-city-of-newport-ri | 115 | 0 | Attachment Object without mediatype: 115. Dialog Object with url and no content_hash: 115 |
fake-vcons | 601 | 0 | Attachment Object without mediatype: 601. Attachment Object without start: 601. Dialog Object with a body and no encoding: 264 |
tadhack-2025 | 43 | 0 | Attachment Object without mediatype: 43. Attachment Object without start: 43 |
Each finding is a rule that draft-ietf-vcon-vcon-core-04 states in its prose and in its schema:
- section 4.4.5
for an inline attachment’s
mediatype. - section 4.4.2
for
start. - section 2.4
for
content_hashbesideurl. - section 2.3.2
for
encoding.
The reference engine reports the same findings at the same paths. With those four repaired, every one of the 17,443 containers is valid to sipnab and to the reference engine.
The corrected copy
tests/fixtures/vcon-datasets/CORRECTED.tsv
pins NormB/vcon-datasets-core04.
- It is a copy of the first three datasets.
- It adds two kinds of key and changes nothing else:
mediatypeon every inline attachment.content_hashon the 425 IETF audio recordings whose files it downloaded, with an IETF login, and hashed.
- Its CHANGES.md lists what it leaves open and why.
- fake-vcons and tadhack-2025 are not in it.
- The reason is the contact data described in the subset README.
- The fetch script fetches both pins files unless
--pinsnames others. - It accepts
--pinsmore than once.
The corpus test holds the corrected copy to three things. These are beside the tests above that pin the upstream findings.
- It is at its pinned commit.
- Each corrected dataset gets the answer measured on 2026-10-11.
- sipnab and the reference agree on every container.
- Each corrected container is its upstream container with keys added.
- The test counts the added
content_hashvalues (425). - It requires each to be a SHA-512 token.
- With them set aside, repairing both containers with the test’s own repair gives the same document.
- The corrected one needs no
mediatyperepair. - That last check compares this repository’s repair rule with the corrected repository’s own, on all 16,799 containers.
- The test counts the added
| Dataset | Containers | Valid | Findings, with the number of containers that carry each |
|---|---|---|---|
vcon-supreme-court-arguments | 8,503 | 8,503 | None |
ietf-meeting-vcons | 8,181 | 4,527 | Dialog Object with url and no content_hash: 3,654 |
vcon-dataset-city-of-newport-ri | 115 | 0 | Dialog Object with url and no content_hash: 115 |
The remaining finding needs a hash of each recording. No fetch returns these recordings to hash.
- The IETF ones are YouTube pages.
- The Newport video host refused the download.
Accepting a snapshot or overwriting a golden is a decision, not a fix.
- Read the diff first.
- These files are the record of what the tool promised its users.
The gate-test roster
These fail on changes you thought had nothing to do with them. Each one exists because the thing it guards silently rotted at least once.
| Gate | Trips when |
|---|---|
docs_drift_test | A --flag named on a published page does not exist in the CLI. That corpus derives from git ls-files rather than a hand-kept list, and covers every tracked docs/ page except design/ and research/, plus website/content/, README.md, SECURITY.md and CONTRIBUTING.md — these pages included, so a phantom flag here fails too. A flag belonging to another tool goes in FOREIGN_FLAGS, scoped to the label of each page naming it. Also: a version marker in the docs or man page disagrees with Cargo.toml; the README feature table misses a Cargo feature; a [theme] slot in ThemeConfig has no documentation, or the slot count quoted in either config reference is wrong. Also the benchmark reproducibility contract: the bench/ harness the benchmarks page tells readers to run must exist and be executable, bench/carrier.py must still produce the corpus composition the page quotes (checked at 1/100 scale), and both doc trees must name the same measured artifact and date. |
dev_docs_drift_test | A page under docs/internals/ links to a path that no longer exists, names a fn that no longer exists, uses an absolute GitHub URL instead of a relative one, is not registered in build-wiki.py, or breaks a mermaid convention. It also builds the wiki and fails if any relative link survives into the output — the wiki is flat and has no repo tree, so such a link publishes dead. That check runs the generator rather than reading it: the version that only asserted CODE_LINK_RE appeared in the script passed while ](../bench/) was shipping broken. |
link_integrity_test | Any relative link or heading anchor in either doc tree does not resolve; Zola content uses a plain relative .md link that would render as a dead URL. On the wiki-source side the scan is the top-level docs/*.md plus docs/internals/ — docs/design/ and docs/research/ are planning material outside the published journey and are deliberately not walked, though a link into them from a scanned page still has to resolve. A third scope covers the root community files (README.md, SUPPORT.md, MAINTAINERS.md, CONTRIBUTING.md, SECURITY.md, CODE_OF_CONDUCT.md), which neither doc tree walks — these are what GitHub renders in the sidebar, so a rename that broke their cross-references used to go unnoticed. |
doc_example_coverage_test | A user-facing CLI flag appears in fewer than two documented examples. A ratchet — the exemption list may only shrink. |
flag_coverage_test | A new long flag ships with no test referencing it. Also a ratchet: adding a test for a grandfathered flag fails until you remove it from the baseline list. |
keybinding_drift_test | A controller handles a key that the F1 help never mentions. |
config_wiring_test | A config key exists but is never read, or a CLI flag has no config fallback where its peers do. |
feature_gate_test | A flag whose subsystem is not compiled in fails late or silently instead of fast and clearly. |
api_guidelines_test | A growth-prone public enum loses #[non_exhaustive], or a shared store stops being Debug. |
summary_consistency_test | Two serializing surfaces disagree about a dialog or stream field. |
wasm_exports_test | The WASM binding loses an export the browser analyzer calls. |
site_journey_test / mockup_alignment_test | A website journey breaks, or a terminal mockup on the site drifts from real output. Also the homepage’s advertised numbers: the binary-size ceiling, the glibc floor, the throughput tiles (which must quote a figure that appears on the benchmarks page they link to), and the automated-test count, which the template must read from the file pages.yml generates rather than state as a number. It holds the standards cards to a canonical table naming the code symbol behind every item, the field the program emits for every quality metric, and a citation in docs/ and the built site mirror, one card per standard. It also holds the homepage’s four MCP examples to the files gen-mcp-examples.sh generated, and holds each of those to the claim the surrounding copy makes about it. That covers the second half of live binary -> website/data/mcp-examples/*.json -> index.html; demos/gen-mcp-examples.sh --check covers the first and needs a built binary, so CI cannot run it. The homepage’s voice-stack section must link exactly the guides in the “Your SIP server” and “Add to your voice stack” groups of the docs navigation, so a guide added to one reaches the other. |
config_examples_test | A config sample in the docs no longer parses. |
doc_commands_run_test | A command shown in docs/, the README or a hand-written page under website/content/ is one sipnab refuses. It extracts every line that starts with sipnab from every shell block, including a fence indented under a list item, through the same CommonMark fence rule as scripts/lib_markdown.py (it checks the two parsers agree on every page), and RUNS 447 of the 496 against a capture that ships with the repository, several at a time, each in a directory and a home of its own, failing on a usage error rather than on a non-zero exit – plenty of documented commands exit non-zero for honest reasons. A device or root example runs with a device name that cannot exist substituted for the one the page names, so its flags still parse in full; a server example runs under a shared wall-clock bound, because clap refuses in milliseconds and a process still alive after it necessarily parsed its arguments. Only two kinds never run, and it names both: a shell program rather than one invocation, and a command that would exec something of its own or attach to the kernel. A command matching neither fails the test. It prints its own coverage, because the difference between “352 ran” and “3 ran, 380 skipped” is the difference between a gate and a decoration. |
examples_run_test | A program in examples/ no longer runs the way its own comment says, or prints something other than the output the comment shows. It runs every cargo run command the comments document (12, across all seven programs), except one under sudo, and holds docs/library.md’s count of the programs to the directory. |
release_surface_test | A release adds a flag, MCP tool or REST route that no release note names, or a guide’s Tested on names a sipnab release older than a flag its commands use. It reads the three lists at every release tag, with extractors it first holds to clap and the MCP router. |
homepage_claim_truth_test | A claim the homepage states in PROSE stops being true: the MCP tool counts and their read-only split, which released builds carry the bpf feature, what the installer actually puts on a host, a capability the static musl build does not have, the filter language’s field and operator counts, a TUI key the page names that nothing binds, and a CLI flag the page names that does not exist, plus the unsafe count and the safety claim resting on it — the page said “memory-safe by construction” beside 88 unsafe blocks and no forbid(unsafe_code). site_journey_test gates the tiles; this gates the sentences, because the two disagreed on one page for months. |
cookbook_recipes_test | A command in docs/examples.md no longer runs or names a flag --help lacks; or a command it executes has no output golden in tests/cli/cookbook/ and no reason in the checker’s OUTPUT_UNPINNED table; or a golden there is one no cookbook command produces. |
support_selftest | The shared normalization helper changes behavior under the tests that depend on it. |
released_port_scan_test | A test under tests/ binds a loopback port 0, reads the port, and releases the socket before using the number, which a test running alongside can take in between. |
The rule for all of them: the gate is not the problem.
- If
flag_coverage_testfails, the flag needs a test. - If
dev_docs_drift_testfails, a page now lies. - Adding an exemption is the last resort.
- Every exemption list in this repo works as a ratchet, so it cannot quietly grow.
The client example programs
Three checks cover the programs under clients/ that the user pages quote.
client_snippets_testholds every block marked with a comment such as<!-- snippet: clients/python/mcp_probe.py#sse-data-lines -->ondocs/rest-api.md,docs/client-examples.md, the metrics and MCP pages and the hand-writtenwebsite/content/docs/api-clients.mdto the marked region of its program, byte for byte. It also requiresscripts/smoke-clients.shto run the site’s whole clients, the capability examples and the operator tasks.example_parity_testholds the lifecycle table ondocs/client-examples.mdto the programs inclients/.scripts/smoke-clients.shruns every program end to end on loopback, against sipnab servers replaying captures in the repository. CI runs it after the build step that builds the binaries and examples. Each program must print its exact output lines and must fail on a failure case: a wrong token, a missing call or customer, a capture with nothing to judge, or no TFPS.
What the script does that a reader of the user page does not need to:
- It waits for evidence, never a fixed sleep:
source_exhaustedin/v1/statsbefore it queries a replaying server, andpackets_admittedin/v1/hep/sendersbefore it reads the HEP roster. sipnab discards input it has not processed when it stops. - TFPS bans by writing a pinned BPF map as root, which no CI runner allows.
The ops server runs with
--tfps-ctlnamingclients/python/tests/fake_tfps_ctl.py, which answers in the documentstfps_ctl --jsonprints.clients/python/tests/test_fake_tfps_ctl.pyholds it to the fixtures pinned against the real binary. The script checks that the stand-in’s block list holds the two banned addresses. evidence_handoff.pyruns twice over one capture into two file roots, and the script requires the same output and byte-identical files. The absolute path each answer carries is the one difference, and the program prints it relative to its root. The README and manifest name the sipnab version, so the script compares two runs rather than pinning hashes.agent_triage.pyruns over stdio and then over HTTP, against a server started on loopback port 0 with a signing key generated at run time and--mcp-tools core. The signed token must print the same lines as the stdio run. A token signed with another key, a token minted from the same key for the REST API, and a request with no token must each get401.customer_export.pyreads three files cut fromsip-problem-call.pcapthe way a wrappedtcpdump -C -Wring leaves them, andcapinfosandtsharkread the export.- The BPF half of
--uprobe-tlsneeds root and a kernel with BTF, so no CI runner can run it.examples/tls_plaintext_records.rsruns the decode half on records laid out as the BPF program publishes them.
The development loop
Logging. SIPNAB_LOG is a tracing EnvFilter.
- So it takes levels and per-module directives
(
SIPNAB_LOG=debug,SIPNAB_LOG=sipnab::rtp=trace). - In TUI mode sipnab suppresses logging unless
SIPNAB_LOGsays otherwise.- The alternative is log lines painting over the interface.
-qlowers the default in CLI mode.- Configured by
init_logging().
Benches. cargo bench.
- The release profile unwinds, as bench units do, so the
cdylibcrate type needed for the WASM build and the benches share one panic strategy. --profile profilingadds full symbols, for a profiler.- The
profilingprofile is release codegen withpanic = "unwind"and debug symbols kept.- That is also what callgrind wants.
- Baselines live in
benches/BASELINES.md.
nextest. .config/nextest.toml defines three profiles:
default(no retries, 30s slow timeout).ci(no retries, no fail-fast, immediate-final output).e2e(2 retries, 60s timeout) for the timing-sensitive tmux tests.
Worth knowing: no workflow currently invokes nextest.
- CI runs plain
cargo test. - The config is there for local use.
- It is also there for the day the e2e shim goes away.
The docker lab. harness/ is a docker-compose stack
for generating real traffic.
- It holds OpenSIPS, rtpengine and SIPp.
make upin that directory builds and starts it.make downtears it down.
WASM. One pre-commit gate covers the browser analyzer.
It checks that website/static/wasm/sipnab.js
still exports every function the site calls.
A second gate once demanded a freshly built bundle alongside any staged
src/wasm.rs.
- It no longer exists, nor does the binary it guarded.
- The published analyzer went eleven releases stale while that gate stayed green.
- The reason:
src/wasm.rshad no commits in the window. - The interface held still while the implementation behind it moved.
- The reason:
- The Pages workflow builds the bundle at deploy time now.
- So the build produces what ships, rather than the tree carrying it.
- It cannot go stale.
Feature matrices. A green cargo test --features full is not proof.
- CI also builds reduced feature sets.
- Code behind
#[cfg(not(feature = "tui"))], or any othercfg(not(feature)), is invisible to the full build. - For exactly this reason, the pre-push hook runs two checks before pushing:
cargo clippy --workspace --all-features --all-targets -- -D warnings.- The
fuzzworkspace check.
Failure-path coverage
The tests and audits behind the operator fault model. That page says how sipnab behaves under each fault class. This section records what pins each behavior.
Parser panic surface
Two harnesses exercise every parser reachable from packet or file bytes:
- Coverage-guided fuzzing:
fuzz/fuzz_targets/(cargo-fuzz / libFuzzer).- The targets run weekly (and on demand) via
.github/workflows/fuzz.yml. - Crash reproducers upload as artifacts.
- The targets run weekly (and on demand) via
- Continuous fuzzing: the same targets run under ClusterFuzzLite from
.clusterfuzzlite/.- Batch fuzzing and corpus pruning run daily. A coverage report runs weekly.
- The difference from the weekly run above is the corpus. It persists between runs instead of restarting from the tracked seeds. So what libFuzzer learns accumulates instead of starting over.
- Always-on smoke fuzz:
tests/smoke_fuzz_test.rsruns incargo test(no nightly needed).- It runs ~40k random + structurally mutated inputs per entry point under
catch_unwind. - It covers the same parser set plus:
- the full link-layer decap chain (
parse_packetacross several link types), - the pcap file reader.
- the full link-layer decap chain (
- A caught panic fails the test. The offending input is hex-dumped for a repro seed.
- Calling every fuzz entry point also compile-checks their signatures. So a target that drifts out of step with its parser breaks the test build. It does not drop out of the fuzz suite unnoticed.
- Example:
fuzz/fuzz_targets/sip_parser.rshad passed a&strtransport argument whereparse_siptakesTransportProto. The whole suite stopped compiling, with nothing incargo testto say so.
- It runs ~40k random + structurally mutated inputs per entry point under
The smoke layer is the regression floor. It found a UTF-8 char-boundary
panic in the TLS keylog hex decoder
(src/capture/tls.rs).
Nobody had ever run the committed fuzz target against that decoder.
- The decoder checked byte-length parity, then sliced
&hex[i..i+2]as astr. - So a multi-byte UTF-8 char (for example
€, 3 bytes) split by the 2-char window panicked. The message was “byte index is not a char boundary”. - That was a remote DoS via a crafted
SSLKEYLOGFILEline. - The decoder now works on raw bytes with explicit nibble validation instead. Any non-ASCII or non-hex byte gives a clean parse error naming its offset.
- The regression test
decode_hex_multibyte_utf8_does_not_panicand the corpus seeds infuzz/corpus/keylog_line/pin it there.
Contained panics: the fuzzers look for panics. These tests hold what happens when one gets through anyway (A parser that panics). No known real input makes these parsers panic, so each test hands its reader a parser that does:
a_frame_whose_parser_panics_is_a_decode_error(src/capture/mod.rs),a_record_reader_that_panics_is_an_error_for_the_file(src/capture/frames.rs),a_member_whose_reader_panics_is_broken_not_fatal(src/capture/archive/mod.rs),a_container_whose_opener_panics_is_refused(src/capture/vcon_input.rs),a_packet_whose_parser_panics_is_refused_as_malformed(src/capture/hep.rs),a_sip_message_whose_parse_panics_is_dropped(src/pipeline.rs),a_frame_whose_decoder_panics_is_a_decode_error(src/rtp/audio_export.rs),contain_counts_a_panic_without_sipnabs_hook(src/crash.rs).
tests/contained_entry_points_test.rs
holds the wiring: each reader’s entry point reaches its contained half, and only
one place under src/capture calls crash::contain. The SIP and RTP readers
outside it reach that place through crate::capture::contained.
Property tests: tests/property_test.rs (proptest) asserts the
semantic invariants the fuzzers cannot:
- a SIP message built from generated fields parses back to those fields,
- SDP survives a build→parse→rebuild round trip,
- the filter DSL is a total function on arbitrary text (parse then evaluate never panic).
Verified bounded in code (confirmed by audit, exercised by
fuzz): SIP header count (≤200) and fold size (≤8 KB); websocket payload (≤64 KB); TLS record length (≤18432); IP/GRE encapsulation depth (≤5); RTP CSRC / RTCP report counts (5-bit fields); all length-driven
Vec::with_capacity sites bounded by the actual slice.
- RTP/RTCP length-field arithmetic (
u16 * 4) cannot overflowusize. - Each multiply carries a bounds
ensure!after it.
Resource bounds
| Store | Bound proven by |
|---|---|
DialogStore | tests/resource_bounds_test.rs (50k unique Call-IDs, both modes, len() ≤ cap at every step) |
StreamStore | same test (50k unique SSRCs) |
| per-dialog messages | dialog_store.rs unit tests |
| per-stream audio frames | stream_store.rs |
| IP/TCP reassembly | reassembly.rs |
Capture, sink and concurrency audit
Audited, found sound (true-positive findings: none):
- Capture file (
pcap_reader.rs):- Magic/length/block-truncation all return clean errors.
- Tests cover them (
too_short_file,invalid_magic,truncated_epb_block_no_panic). The smoke fuzz also exercises them. - The FILE path implements the same exit-status rule as the live path. It did not always.
- Reading a set deliberately continues past a truncated member and returns
Ok. - So the join never saw it, and a truncated pcap exited 0 with a whole-looking report.
- Reading a set deliberately continues past a truncated member and returns
ReadTally::reportinsrc/capture/file.rsis the one place both the single-threaded and--coresreaders converge on.- It records the same
lostpredicate that decides its log severity into the run-integrity record insrc/output/run_integrity.rs. - The exit-code sites and the
--json-dialogs/--reportmarkers read that record. - One place emits both the human sentence on stderr and the machine record. So they cannot disagree.
- It records the same
- A
--pluginthat would not load is the same class and lands the same way (exit1,plugins.failedin the record). - The same record also carries idle compaction. Idle compaction deliberately does NOT move the exit status. A retention policy doing what you configured it to do is not a failed run.
- Live loop (
capture/live.rs):- Device-open and BPF-compile failures return clean errors via the ready channel.
- Receiver-dropped breaks cleanly.
- A transient
recv()error is fatal to the capture thread, and untested. - The thread ends by returning the error rather than panicking.
- The exit status is the only place the partial-read distinction survives.
Downgrade that join to a warning and exit 0, and an incomplete run reads exactly
like a whole one to anything checking
$?.
- event_exec: sipnab reaps every child, and a test covers the escaping.
- Log/JSON sinks:
serdeescaping.\r\nstripped from alert log values (log-injection guard, tested).- Writes use
?, nounwrap.
- Shutdown:
- Atomic-flag signal handlers (
signals.rs, tested). parking_lotlocks cannot poison.- Closed-channel sends checked.
- Atomic-flag signal handlers (
unsafe: libc and other FFI calls, with RAII/Drop-guarded fd ops.- Every block states its own soundness argument.
Cargo.tomlsets clippy’sundocumented_unsafe_blockstowarn.- Both the pre-push hook and CI run clippy with
-D warnings. So a block with no adjacent// SAFETY:comment fails the build.
Untested failure paths
- Live-capture transient recv error: needs a fault-injecting capture source.
- Interface-down mid-capture: funnels into the same fatal recv path.
- Opus decode (
rtp/opus_decode.rs) delegates to the externallibopus. Its FFI input is length-checked. The codec itself is not fuzzed here. - No disk-full test for the pcap/wav writers. Writes propagate
?, so there is no panic, but no test covers the error path.