Performance

sipr’s hot-path rules (docs/ARCHITECTURE.md §3 — no allocations beyond template slot filling, no synchronous I/O, no per-send string scanning) were designed from the start but never measured against the tool they exist to match. This page is that measurement.

Reproduce it with:

make bench-vs-sipp            # SIPP_BIN=... if sipp is not in the sibling checkout

On Debian and Ubuntu that needs the time package: the script reads CPU and peak RSS from /usr/bin/time, not the shell’s time keyword.

The same comparison runs nightly on Linux from .github/workflows/bench.yml, which is where to look for numbers off this laptop — see the caveats below.

Method

scripts/bench-vs-sipp.sh runs both tools against themselves — sipr-UAC to sipr-UAS, sipp-UAC to sipp-UAS — on loopback, at each rate, for a fixed window of seconds rather than a fixed call count, so every rate spends the same time under load. Both sides use each tool’s embedded uac/uas scenarios, which are the same call flow (INVITE / 200 / ACK / pause / BYE / 200), with -d 10 for a 10 ms pause.

Everything measured comes from the same two sources for both tools:

• CPU and peak RSS from /usr/bin/time around each process (BSD -l on macOS, GNU -v on Linux). Neither tool gets a background flag: sipp’s -bg forks and the parent exits immediately, leaving nothing to measure, and sipr skips its TUI on its own when stdout is not a terminal — so both run in the foreground with output redirected, and both pay for the same once-a-second screen refresh.
• Calls, retransmissions and concurrency from each tool’s own -trace_stat CSV, parsed by column name. sipr writes SIPp’s columns byte-for-byte (M40), so one parser reads both files.

Results

Two hosts. The Linux figures come from .github/workflows/bench.yml and are the ones worth quoting; the macOS ones are the development host, kept because they show how much the platform matters.

Linux (GitHub Actions, Linux 6.17 x86_64, 4 cores), sipr 0.27.1 release, sipp v3.7.7, 10 s per rate:

macOS (MacBook, Darwin 27.0.0, arm64), same versions and window:

Both tools placed and completed every call at every rate on both hosts, with no failures. CPU and RSS reproduce within a few percent across runs on the same host.

What the numbers say

The two sides of a run behave differently, and the honest summary needs them apart. Per call:

Generating load: sipr costs a fraction of sipp. sipp’s UAC burns ~10 s of CPU whatever the rate — 500 cps costs it exactly what 5000 does, which is a pacing loop spinning, not work done per call. sipr’s scales with traffic, so it is 2.6x cheaper per call at 5000 cps and 20x cheaper at 500. At low rates sipr is genuinely idle where sipp is not, which is the difference between a generator you can leave running next to the thing you are testing and one you cannot.

Answering load: sipp was slightly cheaper, up to 0.32. In the table above sipp’s UAS spends about 20% less CPU per call than sipr’s (0.065 ms against 0.084 ms at 5000 cps). That was not sipr’s code but its threads waking one another, and it is fixed — see Where the answering side’s time went. sipr also carries more calls concurrently at the same rate (101 against 70 at 5000 cps), i.e. its calls live a little longer, which is consistent with spending a little more per call on the answering side.

Beware the macOS UAS column. There sipp’s UAS looks catastrophic — a flat ~13.9 s — and that is an artifact, not a property of sipp. A macOS build of sipp reports Looking for sys/epoll.h - not found and falls back to poll/select; on Linux the same code is efficient. Anyone comparing receive-side cost should use the Linux table.

Memory is sipr’s clearest win, and it grows with the load. Peak RSS is lower everywhere, and on the UAS at 5000 cps it is 83 MiB against 421 MiB — a fifth. Both tools grow with concurrency, as they must when every live call holds state, but sipp grows far faster.

Retransmissions. sipr sent none at any rate on either host. sipp retransmitted 50 times at 5000 cps on Linux (24 on macOS): under its own load it dropped or delayed enough inbound messages to trip the UDP timers.

Where the per-call time actually goes

cargo bench --bench hot_path measures the three primitives §3 makes rules about, in isolation (numbers in benches/BASELINES.md): on the Linux runner, filling the INVITE template costs 531 ns, parsing a 200 OK and reading the four fields needed to route it 842 ns, arming and cancelling a retransmission timer 54 ns.

A call in this scenario carries about six messages, so those primitives account for roughly 8 µs of the ~79 µs of CPU a call costs on the UAC at 5000 cps — a tenth. The rest is syscalls, call-table bookkeeping and stats.

Two CI runs of the same commit put parse_and_route at 842 ns and 973 ns, a 15% spread on identical code. That is why the >10% regression rule is a prompt to investigate by hand and not a gate: on a shared runner the noise reaches the threshold on its own.

That is worth knowing before anyone optimises: the hot path the rules protect is already cheap, and the remaining cost is in the machinery around it. The one primitive with obvious headroom is message routing, where the accessors (call_id, cseq, top_via_branch) re-scan headers the parse already walked, costing about as much again as the parse itself.

Where the answering side’s time went

Profiling the one column sipr lost — the UAS on Linux — showed its own code was not the cost. On a 6-core Linux VM (OrbStack, aarch64; 5000 cps × 10 s, median of three), sipr’s UAS spent less user time than sipp’s (2.86 s against 3.54 s) and five times the kernel time (1.83 s against 0.36 s), with 180k voluntary context switches against sipp’s 9k. strace -f -c put 13.8 futex calls on every call; sipp makes none — it waits in epoll_pwait. Attributed per thread, two of sipr’s threads existed mostly to be woken:

• The timer thread was notified on every arm and every cancel. It only needs waking for a timer due before the one it already sleeps toward, and never for a cancel: a cancelled timer is a tombstone it skips when it wakes anyway. TimerService now records when its driver next looks at the queue and notifies only for a sooner deadline. This alone took the UAS from 4.68 s to 3.23 s, and the UAC from 4.25 s to 2.55 s — the UAC arms more timers per call.
• A relay thread (sipr-net-bridge) re-sent every inbound message from the transport’s channel into the engine’s: a second hop, one more wake and switch per message. Transports now deliver straight into the engine’s channel through sipr_net::NetSink. It halved the remaining context switches (104k to 58k) and took the UAS to 3.08 s.

Cheaper call starts then exposed a pacing flaw. The pacer ticked every 20 ms, so at 5000 cps it started 100 calls back to back; the faster those went out, the more of the burst — and of the 180/200 and BYE bursts that follow it — overflowed a socket receive buffer. The kernel counted it (RcvbufErrors in /proc/net/snmp): 240–760 drops a run and 50–80 failed calls, against none with -buff_size 1048576. The old build overflowed too, just less (16–69 drops). The tick now spans about five calls, within 1–20 ms, which is what SIPp’s generator does — it wakes when the next call is due and runs at most 1 ms at a time, opening about five a millisecond at 5000 cps. Arrivals that no longer come in bursts cost the receiver a little: each message now tends to wake its thread instead of sharing a wake with its neighbours.

make bench-vs-sipp on the same VM, one run each, CPU of UAS / UAC:

sipr failed no call at any rate; sipp failed 40 and retransmitted 859 times at 5000 cps. Repeat runs move these figures by about 10%, so the columns compare within a row, not across sessions.

What remains is one hop: the socket’s reader thread hands each message to the engine’s loop, where sipp reads it on its loop thread. Closing that means the loop waiting on sockets itself (epoll/kqueue, i.e. a new dependency or unsafe), which these numbers do not call for.

Caveats, and what these numbers are not

• One machine, both processes. UAC and UAS contend for the same cores, so absolute cps is not a ceiling — it is what this laptop sustains with both ends of the call on it. Neither tool was pushed to failure; 5000 cps was clean for both, and the ceiling was not probed.
• A macOS laptop, with shared cores and thermal throttling. The numbers are meaningful relative to each other — same host, same run, same flow — and not portable as absolute figures. benches/BASELINES.md records sipr-only loopback runs from a Linux host for that purpose.
• Peak concurrency is sampled once a second, so it is coarse and the noisiest column here; treat it as an order of magnitude. It is read from the UAC side on purpose: on the UAS side both tools keep finished calls counted for a while (dead-call retention), so that figure measures retention rather than live dialogs. Verified identical in both tools.
• sipp was built without SCTP (USE_GSL=1 USE_PCAP=1 USE_SCTP=0); that does not affect a UDP run.