Docs Fail a build on what a capture shows

Fail a build on what a capture shows

On this page

What you need
Write a suite
Run it
Run it in a pipeline
Check conformance in the same suite
Ask from an agent
When it does not work

A change to a dialplan, a proxy script or a trunk can break calls that nobody places until a customer does. An expectation suite states what a test call capture must show: how many calls, what share answered, how many conformance errors, how good the audio was. sipnab --expect judges a capture against it and exits non-zero when a rule fails, so the step that runs it fails the build.

What you need

A sipnab build with the default features. Judging a capture file needs no privileges.
A capture of the traffic you want to judge. In a pipeline that is usually a test run: SIPp or a test phone places calls through a staging server while sipnab or tcpdump records them.
A suite file in your repository, sipnab.expect.toml by convention.
Nothing on your SIP servers. The suite reads the capture only.

Write a suite

A suite is a list of rules. Each rule measures one figure over the capture’s calls and compares it with a threshold:

[[rules]]
name = "calls captured"
metric = "count"
op = ">="
value = 5

[[rules]]
name = "answer-seizure ratio"
metric = "asr"
op = ">="
value = 50

[[rules]]
name = "no busy answers"
metric = "count"
op = "=="
value = 0
scope = "filter:response_code == 486"

This is tests/fixtures/expect/sip-problem-call.expect.toml in a source checkout.

Key	Required	What it takes
`metric`	yes	`count`: the calls in scope. `asr`: the percentage of INVITEs with a final response that the far end answered. `lint_errors`: the conformance findings at or above a severity. `mos_p<N>`: the Nth percentile of the streams’ MOS, from `mos_p0` (the worst stream) to `mos_p100` (the best)
`op`	yes	`>=`, `>`, `<=`, `<`, `==` or `!=`
`value`	yes	The threshold, a number
`name`	no	A label for the output. Without it, the output names the rule by its position, such as `#1`
`scope`	no	`filter:<expression>` narrows the rule to the calls a filter selects; an alias such as `filter:problems` works too. `severity:<info, notice, warning or error>` sets the lowest severity a `lint_errors` rule counts. Without it, the rule reads the whole capture
`min_sample`	no	The fewest calls (or streams, for MOS) the rule needs. Below it, the rule is `skipped` instead of judged
`grounded_only`	no	MOS rules only. `true` (the default) reads only streams whose codec has a published impairment value. `false` admits the others, and the result says so

An asr or MOS rule that measured nothing fails: an asr rule whose scope holds no INVITE with a final response, or a MOS rule over a capture with no RTP, does not pass by default. Give it min_sample when an empty population is acceptable. A count rule always measures something: a scope that selects no call counts 0, so count == 0 passes.

Run it

sipnab -I tests/pcap-samples/sip-problem-call.pcap --expect tests/fixtures/expect/sip-problem-call.expect.toml --no-cli-print --quiet

Expectations: fail, 1 of 3 rules passed, 2 failed
Rule                  Metric  Op  Threshold  Observed  Sample  Verdict
----------------------------------------------------------------------
calls captured        count   >=  5          5         5       pass
answer-seizure ratio  asr     >=  50         20        5       fail
no busy answers       count   ==  0          1         5       fail
answer-seizure ratio: 20 is NOT >= 50 over 5 seizure(s)
no busy answers: 1 is NOT == 0 over 5 dialog(s)

The first line is the suite’s verdict. Each rule’s row gives what it observed and over how many calls, and each failed rule repeats on a line of its own.

The run exits with the verdict:

Exit	Meaning
`0`	Every rule that ran passed
`3`	At least one rule failed
`2`	No rule ran, because each one’s population was below its `min_sample`. Or sipnab could not read the suite, or the suite holds a rule sipnab refuses
`1`	sipnab could not read the capture

sipnab reads and checks the suite before it reads the capture, so a mistake in the file stops the run at once.

The rules judge every call in the capture. --filter and --since do not narrow them. Narrow a rule with its own scope.

Run it in a pipeline

Run the suite as its own step, after the step that records the capture. The step fails when sipnab exits non-zero:

sipnab -I test-calls.pcap --expect sipnab.expect.toml --no-cli-print --quiet

To keep a machine-readable record of the verdict, write the JSON report as well. exit_code is the run’s exit status:

sipnab -I tests/pcap-samples/sip-problem-call.pcap --expect tests/fixtures/expect/sip-problem-call.expect.toml --json --no-cli-print --quiet | jq -c '{verdict, exit_code, failed: [.results[] | select(.verdict == "fail") | .name]}'

{"verdict":"fail","exit_code":3,"failed":["answer-seizure ratio","no busy answers"]}

A pipe hands the shell jq’s exit status, not sipnab’s. In a script, read exit_code from the JSON, or use set -o pipefail.

Check conformance in the same suite

A lint_errors rule counts the conformance linter’s findings, so one suite can fail a build on non-conformant SIP as well as on failed calls. The scope sets the lowest severity counted:

[[rules]]
name = "no conformance errors"
metric = "lint_errors"
op = "=="
value = 0
scope = "severity:error"

With that rule in lint.expect.toml, against the capture that ships with sipnab to show what the linter finds:

sipnab -I tests/pcap-samples/sip-lint-findings.pcap --expect lint.expect.toml --no-cli-print --quiet

Expectations: fail, 0 of 1 rule passed, 1 failed
Rule                   Metric       Op  Threshold  Observed  Sample  Verdict
----------------------------------------------------------------------------
no conformance errors  lint_errors  ==  0          2         3       fail
no conformance errors: 2 is NOT == 0 over 3 dialog(s)

The rule honors the same suppressions as --lint: a .sipnablint file beside the capture, or the file --lint-suppress-file names. See Suppress a rule.

Ask from an agent

The MCP tool evaluate_expectations judges the capture the server holds. It takes the rules inline as JSON, or a suite file’s text in rules_toml, and returns the same report. See evaluate_expectations.

When it does not work

A metric sipnab does not know stops the run before it reads the capture, with exit 2. The message names the suite file you gave, here sipnab.expect.toml, and lists the metrics sipnab takes:

error: invalid value 'sipnab.expect.toml' for '--expect <FILE>': sipnab.expect.toml: rule 0: unknown metric 'asr2'. Known metrics: count, asr, lint_errors, mos_p<N>. A percentile is spelled mos_p10, mos_p50, mos_p90 and so on, from mos_p0 (worst stream) to mos_p100 (best)

A rule that measured nothing fails, and says why. Here a MOS rule over sip-problem-call.pcap, which holds no RTP:

Expectations: fail, 0 of 1 rule passed, 1 failed
Rule  Metric   Op  Threshold  Observed  Sample  Verdict
-------------------------------------------------------
#1    mos_p10  >=  3.50       -         0       fail
#1: unevaluable: no scored stream was in scope, so 'mos_p10 >= 3.5' rests on nothing. Declare a min_sample of 1 or more to accept an empty population in writing; a gate does not pass on data it never judged

With min_sample = 1 sipnab skips the same rule instead, and a suite in which sipnab skipped every rule exits 2:

Expectations: not evaluated, 0 of 0 rules passed, 1 skipped
Rule  Metric   Op  Threshold  Observed  Sample  Verdict
-------------------------------------------------------
#1    mos_p10  >=  3.50       -         0       skipped
#1: skipped: 0 scored stream(s) is below the declared min_sample of 1

What you see	Why	What to do
Exit 2 before sipnab reads the capture, naming a rule	The suite has a metric, operator, scope or key sipnab does not take, or a filter that does not parse	Fix the rule the message names
Exit 2 and `not evaluated`	Every rule was below its `min_sample`	Capture more calls, or lower `min_sample`
A rule fails with `unevaluable` and `Sample` 0	An `asr` or MOS rule measured nothing: its scope held no INVITE with a final response, or no stream	Check the filter with `--report --filter`. Give `min_sample` if an empty capture is acceptable
A MOS rule fails on a capture with audio	`grounded_only` excluded streams whose codec has no published impairment value	See MOS and codecs, or set `grounded_only = false` and read the caveat
The step passes although a rule failed	A pipe returned the exit status of the last command	Run sipnab on its own, or use `set -o pipefail`