Threading Model

The topology below is the reality on main today.

• The core principle is design decision D2.
• The packet path is synchronous threads + channels.
• tokio exists only inside the optional servers.

Topology

                       ┌──────────────────────────────────────────────────┐
                       │                 shared state                     │
                       │  Arc<RwLock<DialogStore>>  Arc<RwLock<StreamStore>>
                       │        (parking_lot, single writer)              │
                       └───────▲──────────────────────────▲───────────────┘
                        write  │                     read │ (try_read in TUI)
capture thread(s)              │                          │
  live device / file / HEP     │                          │
  (capture::start_multi_capture)                          │
        │ Packet               │                          │
        ▼                      │                          │
  capture::channel  ────► processing thread ──────► TUI event loop (main thread)
  (capped channel)        (TUI mode: pipeline::            crossterm events, render
                           process_packet;                 TUI MODE ONLY
                           batch mode: main.rs loop
                           on the main thread)

  auxiliary threads, by the RUN MODE that starts each — none of them is a
  child of the TUI event loop:

  both modes
    ├── server runtime thread          app::servers::start_servers
    │     one shared current-thread tokio runtime hosting every enabled
    │     async server as a task: api (axum), mcp (rmcp), mcp-http
    ├── metrics-server thread          app::servers::start_servers
    │     raw std::net::TcpListener accept loop + one short-lived
    │     metrics-conn thread per scrape
    ├── DNS resolver thread            app::build_resolver
    │     std::mpsc queue; spawned only when reverse DNS is enabled
    └── rtpengine reconciler           app::relay_reconciler::spawn
          bounded hand-off queue; starts only when --rtpengine-control
          names a relay on a live run

  batch mode only
    └── scanner-kill worker            process_isolation::spawn_scanner_kill_worker
          a separate PROCESS (this binary, re-executed), started by
          bootstrap::launch before the chroot and privilege drop; two
          threads here serve its pipes; --kill-scanner has no TUI path

The mode column above is the part to keep true. It is worth saying why.

• An earlier revision of this diagram drew every auxiliary thread as a child of the TUI event loop.
• Not one of them is.
  • Both run modes start four of them.
  • Batch alone starts the scanner-kill worker.

That is not a cosmetic error.

• --metrics really did ship wired to the TUI path alone.
• So every headless -N deployment got no metrics at all.
  • That is every container and systemd unit.
• This page said nothing that would have contradicted it.
• A diagram can group threads by who spawns them rather than by which mode runs them.
  • Such a diagram reads as reassurance.
  • It hides exactly that class of defect.

The four spawn sites, none of which is the TUI event loop:

• start_servers() for the server runtime and the metrics listener.
• build_resolver() for the reverse-DNS worker.
• spawn_scanner_kill_worker() for the kill worker.
• relay_reconciler::spawn() for the rtpengine reconciler.

The Prometheus listener is not a task on the shared tokio runtime. The distinction matters when reasoning about blocking.

• It is a raw TcpListener accept loop on its own thread.
• It is deliberately independent of tokio and axum.
  • So metrics stay scrapable when the async servers are not compiled in at all.

Each accepted scrape runs on a short-lived metrics-conn thread.

• A ConnGate bounds those to 16 in flight.
• Beyond that, new connections get an immediate 503 rather than a thread (SN-02, CWE-770).
• start_metrics_server() spawns the thread itself.
• start_servers calls it for both run modes.

Named threads

Every long-lived thread and its spawn site. A name in a backtrace maps straight back to a row here.

Modes is the column to trust when asking “does this run headless?”

• That is the question --metrics got wrong.
• both means the thread exists under -N as well as in the TUI.
• TUI and batch mean it does not.

Thread name	Modes	Spawned by	Role
capture-<device>	both	capture/native.rs	One per live device: pcap loop producing Packets.
capture-stop:<device>	both	capture/breaker.rs	Beside each live capture: calls pcap_breakloop when a stop is due, for reads that never return by themselves (libpcap’s netmap module on a silent link). Joined when its capture ends.
capture-file	both	capture/native.rs	Offline pcap reader feeding the same channel as live capture.
capture-hep	both	capture/native.rs	HEP/EEP receiver on the transport --hep-listen-transport names; packets carry asserted addresses and carry a flag HEP-origin. On udp this thread reads the datagrams itself. On tcp and tls it is the accept loop, and the reading happens in the per-connection threads below.
(unnamed per-connection readers)	both	capture/hep.rs	One scoped thread per accepted HEP stream connection under --hep-listen-transport tcp or tls, up to HEP_MAX_STREAM_CONNECTIONS, so the listener serves several agents at once and one silent peer cannot stall the rest. Scoped, so no reader outlives the listener; each polls the stop flag on its read timeout, which is what bounds the scope’s join. A TLS connection completes its handshake here rather than in the accept loop, for the same reason.
capture-multi	both	capture/native.rs	Supervisor for a multi-device capture set.
tui-processor	TUI	app/tui_mode.rs	The single store writer in TUI mode: drains the channel and runs the pipeline.
(unnamed workers)	batch	parallel.rs	--cores N reconstruction workers, spawned with bare thread::spawn — they own thread-local stores, so they show as unnamed in a backtrace.
(unnamed per-file readers)	batch	parallel.rs	--cores N over a multi-file -I set: one scoped thread per reader, each reading whole files and handing per-shard batches to the dispatcher, which releases them to the workers in file order. A single-file -I has one file and therefore one reader, so it gets none of these — the calling thread reads it, exactly as before.
servers	both	app/servers.rs	One current-thread tokio runtime hosting api, mcp and mcp-http as JoinSet tasks.
metrics-server	both	output/prometheus_server.rs	Raw TCP accept loop for Prometheus scrapes.
metrics-conn	both	output/prometheus_server.rs	One short-lived thread per accepted scrape, capped at 16 concurrent.
sipnab-dns	both	names.rs	Reverse-DNS resolver draining an std::sync::mpsc queue so the render path never blocks on a lookup. Starts only when the run turns reverse DNS on.
scanner-kill-fwd, scanner-kill-rx	batch	process_isolation.rs	The parent’s half of the scanner-kill worker, which is a process of its own: -fwd writes queued kill requests into the worker’s request pipe (the one write that can block, so it is never the capture thread’s), -rx reads its outcomes into the tally. The worker process runs the decision loop on its main thread between two pumps, kill-worker-in and kill-worker-out; it is the only process that answers an address the capture supplied, and the process parsing captured traffic holds none of its send sockets. --kill-scanner has no TUI path.
rtpengine-reconcile	both	app/relay_reconciler.rs	Asks an rtpengine relay which calls it holds, over the relay’s read-only list and query, so a stream the signaling does not explain still gets a Call-ID. Starts only when --rtpengine-control names a relay on a live run, and keeps the round trip off the packet path.
pcap-load	TUI	tui/controllers/file_open.rs	Loads a pcap chosen from inside the TUI, writing the live stores.
clipboard	TUI	tui/clipboard.rs	Holds the X11/Wayland selection alive after a copy without stalling the UI.
crash-probe	tests	crash.rs	Not a production thread. Only mod tests spawns this name, to prove the panic hook records a thread name. It appears in no shipped backtrace. This row exists so the next reader who greps the name does not re-add it as one.

How the capture thread ends

Every fatal exit taken after start_capture() has to go through stop_and_join(). There is no second correct way out.

The reason is where the thread starts.

• bootstrap::launch() spawns it before these steps:
  • The readiness hand-shake.
  • The chroot.
  • The privilege drop.
• The reason: the capture source must be open while the process still holds CAP_NET_RAW.
• Every failure from that point on happens with a capture thread already running and holding its source. Such failures include:
  • An unopenable source.
  • An unusable --chroot.
  • A refused privilege drop.
  • A hardening step the kernel rejects.
  • A companion server that cannot start.
• std::process::exit joins nothing and runs no destructors.
  • So exiting directly abandons it.

stop_and_join does three things:

• It sets the global shutdown flag.
• It drops the packet receiver.
• It joins.

Both signals matter, and neither is redundant.

• Dropping the receiver ends a thread blocked on a send.
• The flag ends one blocked elsewhere in its loop.
  • A live capture waiting out its read timeout reaches the flag check first.
• One read never times out.
  • libpcap’s netmap module waits inside pcap_next_ex until a frame arrives.
  • So on a silent link the flag check is never reached.
  • For that read, the capture’s capture-stop:<device> thread sees the same flag and calls pcap_breakloop.
  • The loop takes the broken read as a stop.
• The join has no timeout.
  • That matches the one the batch receive loop already performs at end of capture.
• A HEP listener blocked on its socket returns in milliseconds rather than hanging it.

One consequence applies to callers that cannot reach the handle.

• BatchRunner::new() does not own it.
• So its four fatal paths return a PlanError instead of exiting.
• batch::run holds both the handle and the receiver. It does the teardown.
• A function that cannot clean up must hand the failure to one that can.

How a gate holds this.

• ThreadSanitizer treats thread leak as a fatal finding, not a warning.
• cli_flag_behavior_test exercises both shapes:
  • A source that never opens.
  • A failure after it opened.
• So a regression fails there, rather than waiting for the weekly sanitizer run.
• Before this rule, sipnab -I /nonexistent.pcap (a mistyped filename) leaked a thread.

The second writer: pcap-load

Opening a pcap from inside the TUI spawns a second writer.

• The pcap-load worker writes the same stores the render thread is reading.
• That is why every render-side access is try_read() and never read().

sequenceDiagram
    autonumber
    participant UI as TUI event loop
    participant Load as pcap-load worker
    participant DS as DialogStore
    participant SS as StreamStore

    UI->>Load: spawn on O (file open dialog)
    UI->>DS: try_read for the frame
    DS-->>UI: guard, or skip the frame
    Load->>DS: write lock
    DS-->>Load: release
    Load->>SS: write lock
    SS-->>Load: release
    Load-->>UI: progress via async_messages
    Note over UI,SS: the render pass never blocks — a contended try_read skips a frame

--cores N offline mode replaces the single processing thread with a dispatcher + N workers.

• The dispatcher does a cheap host-pair peek.
• It shards raw packets over bounded channels.
• Each worker owns a private PacketProcessor + thread-local DialogStore/StreamStore.
  • There are no locks on the hot path.
• The stores merge at EOF.
• Flow correctness holds because a flow’s packets share a host pair.
  • Therefore they share a worker.
• A call’s SIP and its RTP can run between different hosts, and so on different workers.
  • The dispatcher also sends a copy of each packet that can carry SDP to every other worker, at the same place in that worker’s sequence.
  • A worker reads a copy only for the media sockets its SDP names, and counts nothing from it.
  • So each worker’s StreamStore sees a later call take over a socket and an SSRC at the point in the capture where it did, and starts that call’s stream there, as the single-threaded path does.

Lock discipline

• Parse outside the lock.
  • SIP/RTP/SDP parsing happens before any store lock appears.
  • classify_packet() touches no store at all.
  • Each store is then locked for writing once per packet.
  • That makes writes the most frequent lock operation in the process, not the rarest.
• Lock ordering:
  • When a path needs both stores, dialog store first, then stream store.
  • If it also needs the alert engine, that comes last.
  • The ordering is what carries the safety. The guards are not always disjoint.
  • Two paths each take one store at a time and release it before the other. “briefly” is accurate there.
    • process_packet() on the live path.
    • run_pcap_load() for the file-open worker above.
  • The batch applier (batch.rs) does not work that way.
    • It holds both write guards across the entire per-packet body.
  • This page previously said both write locks were never held at once. That was never true of the batch path.
• The guards queue side effects rather than performing them.
  • These accumulate while the code holds the guards:
    • Alert findings.
    • Per-message output.
    • The --alert-exec / --on-dialog-exec / --on-quality-exec spawns.
  • DeferredEffects::drain replays them after both drop.
  • So none of these happens inside the critical section:
    • A fork/exec.
    • A stdout write.
    • An AlertEngine lock.
  • Until that change, the batch loop took the alert engine’s write lock nested inside both store guards.
    • That was a third lock deep in the hot path, with a posix_spawn beside it.
  • The full rule and what would re-create that edge are in Invariants, “Dialog before stream, then alerts — one consistent order”.
• The TUI never blocks: all render-side store access is try_read().
  • On contention the frame renders with the previous data.
  • Counts may be one frame stale. This is deliberate.
  • An adaptive 10 fps active / 2 fps idle tick bounds staleness.
• Pause is an AtomicBool checked by the processing thread. No lock.
• mcp/ denies clippy::await_holding_lock.
  • parking_lot guards must never live across an .await.

Channels in use

Edge	Flavor
capture → processing	capture::channel — a count-capped permit pool over an unbounded crossbeam channel, so idle memory returns to ~0 while backpressure still blocks the sender. A live source pays one channel item per packet (the kill path reacts to single packets); a file source sends batches of 128 through the same channel, one item per batch, with the slot pool divided to keep the same in-flight packet cap
batch main loop	the same capture::channel wrapper, not a bare bounded channel: batch.rs receives on a capture::channel::PacketRx exactly as the TUI processor does
--cores reader → workers, file input	crossbeam bounded::<Vec<Packet>>(64) carrying batches of 128 (run_offline_parallel_file()) — same ~8192 in-flight packet cap, one channel hop per 128 packets instead of per packet
--cores per-file reader → dispatcher, multi-file input	crossbeam bounded carrying one batch per item (shard_set_parallel()). The queue depth is not the bound that matters: READ_AHEAD_BYTES caps in bytes how far a LATER file’s reader may run ahead of the file the dispatcher holds, because a batch is 128 packets of any size and the default snaplen is 65535
scanner-kill request / response	crossbeam bounded(256) from the capture thread to scanner-kill-fwd, then pipes to and from the worker process, then bounded(256) for the outcome stream (process_isolation.rs)
packet path → rtpengine reconciler	std::sync::mpsc::sync_channel(1024) (orphan_channel()) — the capture path offers a relay-side socket and never waits on it: a full queue drops the offer and counts it, because stalling packet processing on a relay round trip is the trade --rtpengine-control exists to avoid
DNS resolve queue	std::sync::mpsc (names.rs)
inside api/mcp servers	tokio (axum/rmcp internals)