Docs Manage many sipnab instances

Manage many sipnab instances

One table of every sipnab instance's build, binary and configuration digests, privilege and uptime, the instances that did not answer, and HEP senders nobody listed.

On this page

How it works
Before you start
Step 1: run each instance with the REST API over TLS
Step 2: mint an inventory token on each instance
Step 3: write the hosts file
Step 4: read the fleet
Verify
Troubleshoot
States and findings
Check a binary against its release
What it cannot tell you

sipnab fleet inventory asks every sipnab instance you list for its inventory, and prints one table of the fleet. Use it to answer “which build and which configuration does each instance run, and which ones are down?” before and after an upgrade or a configuration change.

The table shows, for each instance:

the build it runs, and the SHA-256 of its binary,
the digest of its effective configuration: the file it loaded and its command line,
the user it runs as, and how long it has been up.

Below the table it names the instances that did not answer, the instances on a different build from the rest, and the HEP senders a collector receives from that are not in your list. It reports. It changes nothing on any instance.

Some upgrades need every sender and its collector on the same build, such as the HEP HMAC version 2 change in Upgrade a fleet that uses HMAC authentication. The report shows which instances are still on the old build.

The commands and output on this page come from a run on 2026-11-03: four instances in containers on the network 192.0.2.0/24, and the controller on the host. The instances ran a development build of 0.5.232 with this feature, and one ran release 0.5.130.

How it works

Each instance serves GET /v1/inventory on its REST API (--api). See GET /v1/inventory for every field.
sipnab fleet inventory runs on one machine, the controller. It reads a hosts file and asks every instance in it at once. The instances never connect to the controller.
Each request carries a bearer token with scope inventory. That token reads the inventory and nothing else: no dialogs, no messages, no audio.
The answer holds no configuration value and no command-line argument, only their SHA-256 digests.
The command reads at most 16 KiB of each answer’s status line and headers (MAX_HEAD_BYTES) and at most 4 MiB of its body (MAX_ANSWER_BYTES). The inventories in this page’s run were under 2 KB each.

Before you start

sipnab on each instance. A build whose sipnab --version lists the api feature, and that serves GET /v1/inventory. An older build answers 404 there, and the report marks it refused.
sipnab on the controller. Any build with the api feature. It needs no capture privilege: it opens no capture.
TLS for each instance’s API. A certificate whose subject alternative name is the address or name in the instance’s URL, its key readable by the sipnab user only, and the certificate of the issuing CA on the controller. Plain http:// works too, and sends the token in clear text.
A signing key on each instance, in a file only the sipnab user can read. Set up authentication shows how to create one.
Network. The controller reaches each instance’s API port over TCP, 8443 in this page’s examples. Open that port to the controller’s address only. The instances need no route to the controller.
Other systems. No change. OpenSIPS, Kamailio, Homer and a HEP collector keep their configuration. A collector that runs sipnab joins the list like any other instance.

Step 1: run each instance with the REST API over TLS

[each instance] Add --api, the TLS certificate and key, the signing key and a node name to the instance’s command line. This is the instance edge-a from the run, which reads a capture file and sends HEP to the collector hub:

sipnab -N -I /var/lib/sipnab/sample.pcap --config /etc/sipnab/edge-a.toml \
  --hep-send 192.0.2.10:9060 --hep-auth-mode hmac --hep-auth-file /etc/sipnab/hep.key \
  --api 192.0.2.11:8443 --api-tls-cert /etc/sipnab/edge-a.crt --api-tls-key /etc/sipnab/edge-a.key \
  --api-signing-key-file /etc/sipnab/edge-a.signing --node-name edge-a

The collector ran with the same API flags and its HEP listener:

sipnab -N --config /etc/sipnab/hub.toml \
  --hep-listen 192.0.2.10:9060 --hep-auth-mode hmac --hep-auth-file /etc/sipnab/hep.key \
  --api 192.0.2.10:8443 --api-tls-cert /etc/sipnab/hub.crt --api-tls-key /etc/sipnab/hub.key \
  --api-signing-key-file /etc/sipnab/hub.signing --node-name hub

Each instance logs REST API listening on <address> when its API is up.

Step 2: mint an inventory token on each instance

[each instance] Mint a token with that instance’s signing key:
```
sipnab --mint-token --token-scope inventory --token-id fleet-controller \
  --api-signing-key-file /etc/sipnab/edge-a.signing --api-token-ttl 2592000
```
It prints one token, starting s2., and exits. --api-token-ttl 2592000 makes it valid for 30 days.
[controller] Save each token in its own file, readable by the controller’s user only:
```
install -m 600 /dev/null /etc/sipnab/fleet/edge-a.token
```
Then paste the token into it. sipnab refuses a token file that other users can read.

Step 3: write the hosts file

[controller] Write /etc/sipnab/fleet/fleet.toml, one [[host]] table per instance. This is the file from the run:

[[host]]
name = "hub"
url = "https://192.0.2.10:8443"
token_file = "/etc/sipnab/fleet/hub.token"
ca_file = "/etc/sipnab/fleet/internal-ca.pem"

[[host]]
name = "edge-a"
url = "https://192.0.2.11:8443"
token_file = "/etc/sipnab/fleet/edge-a.token"
ca_file = "/etc/sipnab/fleet/internal-ca.pem"

[[host]]
name = "edge-b"
url = "https://192.0.2.12:8443"
token_file = "/etc/sipnab/fleet/edge-b.token"
ca_file = "/etc/sipnab/fleet/internal-ca.pem"

name is the name the report shows. Each name appears once.
url is the instance’s REST API, https:// or http://.
token_file holds the instance’s inventory token on one line.
ca_file is optional: a PEM CA file to trust instead of the host’s CA bundle.

sipnab refuses an unknown key, an empty name, an empty URL and a name used twice, with exit 2.

Step 4: read the fleet

[controller] Run:

sipnab fleet inventory --hosts /etc/sipnab/fleet/fleet.toml

Output from the run:

NAME             STATE        VERSION      COMMIT       BINARY       CONFIG          UID   UPTIME_S
hub              ok           0.5.232      29204db7     6bba53867ecd ec09522b49b4   1000         54
edge-a           ok           0.5.232      29204db7     6bba53867ecd 6a434df0c5e2   1000         40
edge-b           ok           0.5.232      29204db7     6bba53867ecd f63dfdce2c9d   1000         39

It exited 0.

BINARY and CONFIG are the first 12 hex digits of binary.sha256 and config.effective_sha256. The three instances run one binary and three configurations: each has its own config file and node name.
--json prints the whole report, with every instance’s full inventory.
--timeout <SECS> sets how long each instance has to answer. Default 10.

Verify

Every instance is ok, and there is no finding line. That is exit 0.
Two instances you expect to match show one CONFIG. Two instances with the same CONFIG loaded the same configuration values and run the same command line. A different --node-name alone gives a different CONFIG.

The exit status says the same for a script:

Code	Meaning
`0`	Every instance answered and there is no finding.
`1`	An instance did not answer, refused the token, or sent an inventory this build cannot read; or there is a finding.
`2`	sipnab cannot read or use the hosts file.

Troubleshoot

Each output below comes from the same run, with one thing changed.

unreachable and connection timed out. Nothing answered at that address within --timeout: the instance is down, the URL names the wrong address or port, or a firewall drops the controller’s packets. The other instances are still listed:
```
edge-c           unreachable  192.0.2.14:8443: connection timed out
```
Connection refused instead means the host answered and nothing listens on that port: check --api on the instance.
unreachable and invalid peer certificate: UnknownIssuer. The controller does not trust the CA that issued the instance’s certificate. Set ca_file for that host:
```
hub              unreachable  sending: invalid peer certificate: UnknownIssuer
```
refused with HTTP 401. The instance did not accept the token: it was minted with another instance’s signing key, it expired, someone revoked it, or its scope is not inventory, full, actions or admin. Mint a new one on that instance (step 2). This run gave edge-b the hub’s token:
```
edge-b           refused      the instance answered HTTP 401
```
refused with HTTP 404. The instance runs a build without GET /v1/inventory. Upgrade it. Release 0.5.130 answered:
```
edge-old         refused      the instance answered HTTP 404
```
Release 0.5.130 also refuses --token-scope inventory at mint time, with invalid value 'inventory' for '--token-scope <SCOPE>'.

unreachable and can be read or written by other users. The token file’s mode lets other users read it. Run chmod 600 on it:

edge-a           unreachable  token_file '/etc/sipnab/fleet/open.token' can be read or written by other users (mode 0644); chmod 600 it

A mixed HEP HMAC version. A sender on 0.5.130 or earlier signs HEP with HMAC version 1, and a collector on 0.5.131 or later drops every packet it sends. In the run, edge-old (0.5.130) sent to hub. Listed in the hosts file, edge-old’s row is refused with HTTP 404, as above. The collector logs HEP HMAC auth is REFUSING version-1 tokens once, and counts the dropped packets. Read the count from the JSON report:
```
sipnab fleet inventory --hosts /etc/sipnab/fleet/fleet.toml --json \
  | jq -c '.hosts[] | {name, refused: .inventory.hep_listener.packets_refused, senders: [.inventory.hep_listener.senders[].peer]}'
```
```
{"name":"hub","refused":10,"senders":["192.0.2.11","192.0.2.12"]}
{"name":"edge-a","refused":0,"senders":[]}
{"name":"edge-b","refused":0,"senders":[]}
```
The hub refused 10 packets, and 192.0.2.13 is not among its senders. The report itself is ok when the old sender is not in the hosts file: a refused packet is not a finding. Upgrade the sender, as Upgrade a fleet that uses HMAC authentication describes.
finding unlisted_hep_sender. A collector in the list receives HEP from an address that no listed instance’s URL resolves to. Add that sender to the hosts file, or find out why it sends. With edge-b left out of the file:
```
finding unlisted_hep_sender: hub receives HEP from 192.0.2.12, which is no instance in the hosts file
```
The check compares addresses only. A sender whose packets leave from a different address than its API URL names shows up as unlisted.
incompatible and a schema_version. The instance answered an inventory format newer than the controller reads. Upgrade the controller’s sipnab. No release answers a newer format yet, so this output comes from a test server that answered schema_version 2:
```
edge-next        incompatible the instance answered inventory schema_version 2; this sipnab reads up to 1
```

States and findings

State of each instance:

State	Meaning
`ok`	It answered with an inventory.
`refused`	It answered with an HTTP status other than 200, such as `401` for a wrong, expired or narrower token, or `404` for a build without the inventory.
`unreachable`	No answer: no connection, a timeout, a TLS failure, a token file sipnab would not read, or an answer that is not JSON.
`incompatible`	Its inventory has a `schema_version` newer than this build reads, or none. Upgrade the controller’s sipnab.

Findings across the fleet:

Finding	Meaning
`version_mismatch`	The instances that answered do not all run the same build. The finding names each build and the instances on it.
`unlisted_hep_sender`	A HEP collector in the list receives from an address that no listed instance’s URL resolves to.

Check a binary against its release

Every instance reports signature.state as unverified. A process cannot prove which binary it is, and one that someone had tampered with would report “verified” too.

Download SHA256SUMS.txt from the release the instance claims, and verify it:
```
gh attestation verify SHA256SUMS.txt --repo NormB/sipnab
```
Compare the instance’s binary.sha256 with the line for its artifact. On Linux the instance hashes /proc/self/exe, so after an upgrade replaces the file the digest still names the binary that is running.

What it cannot tell you

Every field is the instance’s own claim. An attacker who controls a host can make it report anything.
For a secret held in a file, the digests cover its path only. The configuration and the command line name key, token and password files by path, so replacing the contents of such a file, for example rotating the key in --api-signing-key-file, changes no digest. A secret given as a value, such as --api-signing-key or a ${NAME} the configuration expands, changes config.effective_sha256.
It reads only the instances in your hosts file. An instance nobody listed is invisible to it, unless it sends HEP to a listed collector.
It changes nothing. Pushing a configuration, upgrading and restarting are not part of it.