Benchmark

From mnema-talk v3.3.13, rendered on this site.

A capacity measurement of Mnema Talk 2.2.0 on a small home server, repeated on 3.3.0 (see Repeat on 3.3.0). It answers one question: how much of a low-power machine do chat and voice use at a given number of members? It is a single run on a single host, so read the figures as orders of magnitude, not as a guarantee.

Test system

NotesNotes
Server version2.2.0 (the published container image)
CPUIntel N100, 4 cores, no hyper-threading
Memory16 GB
StorageNVMe SSD (btrfs cache pool); PostgreSQL and the application data live on it
Operating systemUnraid 7.3.4, Linux 6.18
Container runtimeDocker 29.8.2, the repository's docker-compose.yml (application, PostgreSQL 18, SeaweedFS)
Application memory limitthe default APP_MEM_LIMIT=1g
Networkloopback: the load generator ran on the same machine
Other loadone real member was online during the run; the instance was otherwise idle

Method

The tools are those of loadtest/: static binaries run by loadtest/host-run.sh on the server host. No browsers are involved. Members are synthetic accounts that cannot sign in; they are created before and deleted after the run, and a fingerprint of every database table before and after must match.

The load generator ran on the same host as the server. Its CPU and memory are part of what the machine spends, and it shares the four cores with the server and the database. The tables list the server's own CPU; the generator's CPU is given where it was significant. Numbers measured from a separate machine would be somewhat better, and on a network they would also include the network.

CPU is the application process in cores (process_cpu_seconds_total), averaged over the steady "hold" phase of each scenario; 4.0 would be the whole machine. Memory is the peak resident set of the application process. The database ran in its own container; its share is not included (it stayed below 0.1 core and about 260 MB in all scenarios).

ScenarioWhat it measures
Chat, N sessionsN WebSocket sessions in one text channel. 10 % of the sessions write 0.2 messages per second each, with typing indicators; 5 % of the messages get a reaction and 3 % an edit. Everyone reads. Latency is measured from the start of the send request to the moment each session reads the event. Loss is the share of expected deliveries that never arrived.
Voice, directOne shared screen (synthetic VP8, 1500 kbit/s, 15 fps) watched by 99 members, over direct UDP.
Voice, through the relayThe same screen and 3 speakers, 100 members; every member may use relayed candidates only, so all media goes through the built-in TURN relay.
Voice, mixedOne screen, 10 cameras (300 kbit/s, 24 fps) and 3 speakers; all 100 members watch the screen and all cameras.

Each scenario held for two minutes after connecting. Voice media is synthetic (valid packets sized to the bitrate, keyframes every three seconds); nothing is captured, encoded or decoded.

Results

Chat

Delivery latency is over all receiving sessions, so the 1,000-session run is 2.36 million measured deliveries. No message, edit or reaction was lost or duplicated in any run, and no send request failed.

SessionsWritersMessages p50 / p95 / p99Edits p95Reactions p95LossServer CPU (hold / peak)Server memory (peak)
1001012 / 18 / 26 ms30 ms13 ms0 %0.02 / 0.02 cores47 MB
3003014 / 21 / 44 ms29 ms15 ms0 %0.09 / 0.10 cores62 MB
1,00010033 / 84 / 115 ms147 ms65 ms0 %0.67 / 0.70 cores102 MB

Connecting took 20 ms at most per session. Goroutines returned to their starting value (about 60) after every run.

Voice, direct

99 members received one screen at 1500 kbit/s.

MembersReceived per memberLossFreezesConnect p50 / p95First video p50 / p95Server CPU (hold / peak)Server memory (peak)
1001585 kbit/s (1532 to 1631)0.00 %031 / 55 ms66 / 374 ms0.56 / 0.73 cores312 MB

Longest gap in any member's video: 67 ms. The generator used 0.54 cores.

Voice, through the built-in TURN relay

MembersThrough the relayScreen per memberAudio per memberLossFreezesConnect p50 / p95Server CPU (hold / peak)Server memory (peak)
100100 of 1001601 kbit/s119 kbit/s (3 speakers)0.00 %1 member, one 12.6 s gap1038 / 1058 ms1.61 / 1.67 cores312 MB

One member lost its connection once (a "flap") and was frozen for 12.6 seconds before it recovered; the scenario's threshold allows no flaps, so it is recorded as failed although the other 99 members saw no freeze. The generator used 0.82 cores. About 3.3 GB left the server in the two minutes.

Voice, mixed (does not hold)

One screen, 10 cameras and 3 speakers, every member watching everything: roughly 4.5 Mbit/s of video per member, 450 Mbit/s in total out of one machine that also runs the generator.

MembersScreen receivedCamera received (per track)Screen lossCamera lossAudio lossServer CPU (hold / peak)Server memory (peak)
100200 kbit/s of 1500134 kbit/s of 30081.6 %33.5 %0.3 %1.87 / 2.06 cores1044 MB

All members connected and none timed out, but most of the video did not arrive. The server used about two cores and reached the 1 GB memory limit, and the generator used 0.97 cores, so on this machine the run is limited by the shared host. We did not determine whether the server, the generator or the loopback path was the first bottleneck. Goroutines and heap returned to their starting values afterwards, so there is no leak.

What this shows

  • Chat is cheap: 1,000 concurrent sessions used about two thirds of a core and 100 MB, and delivered every event within 190 ms.
  • One screen share to about 100 members used about half a core directly and about 1.6 cores through the relay.
  • A Talk in which 100 members watch ten cameras and a screen at the same time was beyond this machine up to 3.3; since the SFU sends from all goroutines at once it holds with no loss at about 1.9 cores (the socket's write lock).

Repeat on 3.3.0

The six scenarios were run again on server 3.3.0 (the published container image, pinned by digest), with the same settings and the same host as above. This run used a throwaway stack on the same machine: its own application, PostgreSQL and SeaweedFS containers with fresh secrets and volumes, other ports, and no real members, no push, no update check and no other load from the instance. The production instance on the host was not involved. The load generator was built from the repository at the time of the run, so it is newer than the one used for 2.2.0. The announced address was the host's LAN address instead of loopback. It is one run, like the first; differences of a few percent are noise.

CPU is the average over the hold phase, memory the peak resident set of the application process. Egress is what the server's media counters sent in the run (not recorded for 2.2.0 except for the relay).

ScenarioCPU 2.2.0CPU 3.3.0Memory 2.2.0Memory 3.3.0Egress 3.3.0Loss 2.2.0Loss 3.3.0
Chat, 100 sessions0.020.0247 MB60 MBn/a0 %0 %
Chat, 300 sessions0.090.0862 MB65 MBn/a0 %0 %
Chat, 1,000 sessions0.670.65102 MB107 MBn/a0 %0 %
Voice, direct0.560.56312 MB181 MB2.7 GB0.00 %0.00 %
Voice, relay1.611.34312 MB192 MB3.2 GB (3.3 GB through the relay)0.00 %0.00 %
Voice, mixed1.871.371044 MB423 MB3.8 GBscreen 81.6 %, camera 33.5 %screen 76.9 %, camera 12.9 %
  • Chat is unchanged in CPU and a few MB larger. Delivery latency at 1,000 sessions is higher: messages p50 / p95 / p99 were 33 / 84 / 115 ms and are now 38 / 126 / 328 ms (maximum 1.3 s). Nothing was lost or duplicated, and the 100 and 300 session runs match the first run within 2 ms at p95.
  • Voice, direct costs the same CPU with 42 % less memory (heap 116 MB). No loss, no freezes.
  • Relay: CPU is 17 % lower and memory 38 % lower. As in the first run one member failed the scenario's thresholds: this time one member's signalling failed, it connected without the relay and was frozen for 43 s (before: one member, 12.6 s). The other 99 members saw no freeze. Both runs show one flap in 100 members; we did not investigate the cause.
  • Mixed still does not hold on this host: the screen loses 77 % of its packets and the cameras 13 %, with the generator at 1.17 cores. The loss is again at the server's ingress (generator sent 90,143 packets, the server counted 42,506), none after the server. CPU is 27 % lower and memory 59 % lower than on 2.2.0 and the server stays far from its 1 GB limit (peak heap 320 MB). This matches the analysis in Where the packet loss comes from: the generator and the shared host limit this shape, not the server.
  • No goroutine leak: goroutines returned to 40 to 48 after every scenario, and the database fingerprint matched before and after.

Same generator, both versions

The chat run with 1,000 sessions was repeated on 2.2.0 and on 3.3.3 back to back, with the same (current) load generator, on a throwaway stack on the same host (fresh database for each; one run each, so differences of a few percent are noise). This separates the change in the server from the change in the generator.

VersionMessages p50 / p95 / p99 / maxServer CPU (hold)Memory (peak)Loss
2.2.031 / 75 / 98 / 153 ms0.63 cores90 MB0 %
3.3.338 / 120 / 196 / 345 ms0.64 cores99 MB0 %

With the same generator, 3.3.3 delivers about 60 % slower at p95 and twice as slow at p99 than 2.2.0, at the same CPU and about 10 % more memory; nothing is lost. All deliveries still arrive within 350 ms. The 2.2.0 figures with the newer generator (75 / 98 ms) are slightly better than those of the first run (84 / 115 ms), so part of the gap in the 3.3.0 table above comes from the generator and the shared host.

The cause of the remaining difference was the channel's lock. Since 3.0.10 (migration 0104) a post holds its channel's row from the number it draws to its commit, so that GET /api/read-state can say exactly which messages its counts include. Every post in the busy channel queued for it, and the lock was held for 18 round trips to the database and through the WAL flush (about 5 ms on this host's disk). Timing spans in the server on this host showed a median hold of 13 to 15 ms and a wait for the lock of up to 110 ms (p99); the fan-out itself took under 20 ms (p99). The fix commits the post asynchronously and waits for the flush afterwards, before anybody hears of the message (concurrent posts share it), and holds the lock for 7 round trips. The same run, two of each back to back on fresh stacks:

VersionMessages p50 / p95 / p99 / maxServer CPU (hold)Memory (peak)Loss
3.3.337 / 122 / 213 / 443 ms, 38 / 116 / 179 / 333 ms0.65 cores98 to 102 MB0 %
with the fix33 / 79 / 113 / 186 ms, 32 / 83 / 117 / 201 ms0.63 to 0.64 cores100 to 101 MB0 %

p95 and p99 drop by about a third and come within 10 to 20 % of 2.2.0; the lock's ordering guarantee is kept.

The mixed Talk holds: the socket's write lock

The loss of the mixed shape at the server's ingress had one cause: the server's own UDP socket. Every member's media leaves through the SFU's one shared port, and Go lets one write at a time go through a socket for the whole system call. On this host a send costs about 14 µs, so the socket carried at most about 45,000 packets a second; the forwarding goroutines of all tracks queued on its lock (all 14 of them, in every goroutine dump of the run), read their publishers more slowly than they sent, and the tracks' receive buffers overflowed. Those packets were dropped before the server counted them, which is why the loss showed at the ingress while a third of the CPU was idle and the kernel counted no drops. A counter for these drops was added in the same change: mnema_sfu_ingress_dropped_total{reason="buffer"} (35,265 in the run before the fix).

Since the fix the SFU's port and the built-in relay's port send from all goroutines at once. This direct path exists on Linux only (the container image); other platforms keep Go's one-at-a-time path (see internal/fastudp). Same host, same throwaway stack and generator, server built from the same commit with and without the fix, one run each with a 120 s hold:

Shape (100 members)VersionScreen lossCamera lossEgress in the windowServer CPU (hold)Generator
screen, 10 cameras at 300 kbit/s, 3 speakersbefore76.5 %11.8 %4.35 M packets1.38 cores1.19 cores
after0.00 %0.00 %8.92 M packets1.91 cores1.42 cores
screen, 4 cameras at 1 Mbit/s, 3 speakersbefore64.4 %49.5 %4.48 M packets1.38 cores1.17 cores
after0.00 %0.00 %8.99 M packets1.91 cores1.40 cores
screen and 3 speakers, all through the relaybefore0.00 %3.56 M packets1.31 cores0.80 cores
after0.00 %3.55 M packets1.24 cores0.71 cores

The server now sends twice the packets for 40 % more CPU: about 24 µs per packet instead of 35. Audio lost nothing in any run. The relay shape was not limited by the lock and is unchanged within noise; its flap of the earlier runs did not occur in either run.

The next limit is the CPU itself. With 120 members (screen, 10 cameras, 3 speakers) the server used 2.23 cores, the generator 1.46 and the host was 88 % busy; the screen then lost 22 % of its packets, again in the tracks' receive buffers. About half of the server's CPU in these runs is the kernel's send path (the system call itself), the rest mostly Pion's per-packet interceptors; encryption is about 5 %.

Sizing guide

The question this section answers: how much hardware does a Talk of a given size need? It comes from a second series on server 2.4.0 on the same host as above: 27 shapes (3 speakers; plus a 1.5 Mbit/s screen share; plus 4 or 10 cameras at 300 kbit/s, or 4 cameras at 1 Mbit/s; 10, 25, 50 and 100 members; direct and through the relay for the first two shapes), each run three times with a 120 s hold (81 runs). The shapes with cameras were then measured again on server 2.5.0, which changed how video memory and cameras are handled (two runs of each; see 2.4.0 and 2.5.0). The generator ran on the same host, as above.

The model

Server CPU is well described by the number of packets the server handles. A least-squares fit over 22 shapes that ran without packet loss (the shapes without cameras from the 2.4.0 series, the shapes with cameras from the 2.5.0 series, repeats averaged) gives:

TermCost
each packet sent to a member (egress)22 µs of one core
each packet received from a publisher (ingress)72 µs
each connected member0.57 ms per second (0.00057 cores)
each relayed egress packet (built-in TURN)+12 µs

The fit has R² = 0.997; the largest difference to a measurement is 0.04 cores, and the standard deviation over the repeats of a shape is at most 0.03 cores. (The per-packet costs of the first fit on 2.4.0 alone were 24 and 57 µs; the two are not separable beyond that, because ingress and egress grow together.) The packet rates follow from the shape, with K speakers (50 packets/s each at 20 ms), C cameras and S screen shares of bitrate b, and N members:

  • ingress packets/s = 50·K + (C·b_camera + S·b_screen) / 6.5 kbit
  • egress packets/s = (N − 1) · ingress packets/s
  • upload ≈ 1.3 · (N − 1) · (K · 0.04 + C·b_camera + S·b_screen) Mbit/s (the factor covers packet headers, keyframes and retransmissions; measured 1.2 to 1.3)

6.5 kbit is the measured average video packet (about 800 bytes) of the synthetic publishers. Real browsers send 720p cameras at up to 2.5 Mbit/s, so use the real bitrate of your cameras.

Memory used to be the second limit: in 2.4.0 Pion kept a retransmission history per video stream and member, and the live heap grew to 380 to 720 MB with 4 to 10 cameras and 25 to 50 members. Server 2.5.0 sizes that history by bytes and by the current load. In the same shapes the peak live heap was 82 to 246 MB, and 203 to 323 MB with 100 members (see the comparison below). The process needs about 250 MB beyond the heap for the runtime and connections, so the default APP_MEM_LIMIT of 1 GB was enough in every shape that was measured on 2.5.0, with 1.3 times the heap plus 250 MB as the working rule. Shapes with only a screen share were not measured again on 2.5.0; their peak heap on 2.4.0 was 51 to 232 MB.

Sizing table

CPU is in cores of this host's Intel N100. "To plan for" is twice the fit, because bursts lose packets well before 100 % CPU. A modern server vCPU is usually at least as fast as an N100 core, but we did not calibrate this on another CPU, so treat it as an assumption. Upload is the connection the server needs. The default APP_MEM_LIMIT of 1 GB covers every shape in the table (see above; the application sets its Go memory limit to 85 % of it). The last column says whether the shape itself was run on the test host and kept all of its packets.

ShapeMembersEgress packets/sCPU, fitted (cores)CPU to plan for (x2)Upload (Mbit/s)On the test host
3 speakers101,3500.050.11measured, no loss
3 speakers253,6000.110.24measured, no loss
3 speakers507,3500.200.48measured, no loss
3 speakers10014,8500.400.815measured, no loss
3 speakers + 1 screen share (1.5 Mbit/s)103,4270.110.219measured, no loss
3 speakers + 1 screen share (1.5 Mbit/s)259,1380.250.551measured, no loss
3 speakers + 1 screen share (1.5 Mbit/s)5018,6580.470.9103measured, no loss
3 speakers + 1 screen share (1.5 Mbit/s)10037,6960.921.8208measured, no loss
+ 4 cameras (300 kbit/s each)105,0880.160.333measured, no loss
+ 4 cameras (300 kbit/s each)2513,5690.360.788measured, no loss
+ 4 cameras (300 kbit/s each)5027,7040.691.4180measured, no loss
+ 4 cameras (300 kbit/s each)10055,9731.342.7363measured, packet loss
+ 4 cameras (1 Mbit/s each)108,9650.280.666measured, no loss
+ 4 cameras (1 Mbit/s each)2523,9080.621.2175measured, no loss
+ 4 cameras (1 Mbit/s each)5048,8121.192.4358measured, no loss
+ 4 cameras (1 Mbit/s each)10098,6192.334.7 (more than 4)723measured, packet loss
+ 10 cameras (300 kbit/s each)2520,2150.531.1144measured, no loss
+ 10 cameras (300 kbit/s each)5041,2731.012.0294measured, no loss
+ 10 cameras (300 kbit/s each)10083,3881.984.0595measured, packet loss

Read the rows marked "packet loss" as "does not hold on this host", not as the cost of the shape: the numbers in them are the model's, which is fitted only on shapes without loss. They hold at most up to about 1.5 times the largest shape that ran cleanly (the clean shapes reach 1.2 cores of server CPU), and only for one Talk at a time: several Talks at once were not measured. The relay adds 60 to 70 % CPU for the screen-share shape.

What was measured

Per shape, mean of three runs (server CPU is the average over the hold phase; heap is the peak live heap; RSS is not shown because the process keeps memory from earlier runs). The generator CPU column is the load generator on the same host.

ShapeMembersPathServer CPU, cores (mean of 3, max spread)Generator CPUHeap peak MBEgress packets/sEgress Mbit/sWorst media lossContainer RcvbufErrorsmemory.events max
3 speakers10direct0.06 (±0.00)0.062571,44810.0 %00
3 speakers10relay0.09 (±0.00)0.061031,48410.0 %00
3 speakers + screen10direct0.10 (±0.00)0.10513,146160.0 %00
3 speakers + screen10relay0.17 (±0.00)0.10523,225170.0 %00
+ 4 cameras @ 300 kbit/s10direct0.15 (±0.00)0.152675,058290.0 %00
+ 4 cameras @ 1000 kbit/s10direct0.22 (±0.00)0.212617,987570.0 %00
3 speakers25direct0.12 (±0.00)0.11403,99740.0 %00
3 speakers25relay0.21 (±0.01)0.12354,14740.0 %00
3 speakers + screen25direct0.23 (±0.00)0.211038,693450.0 %00
3 speakers + screen25relay0.38 (±0.00)0.201039,065470.0 %00
+ 4 cameras @ 300 kbit/s25direct0.37 (±0.00)0.3437814,004790.0 %00
+ 4 cameras @ 1000 kbit/s25direct0.55 (±0.00)0.5137722,1451570.0 %00
+ 10 cameras @ 300 kbit/s25direct0.58 (±0.01)0.5471121,9711300.0 %02
3 speakers50direct0.23 (±0.00)0.20828,46580.0 %00
3 speakers50relay0.39 (±0.01)0.21568,99380.0 %00
3 speakers + screen50direct0.45 (±0.00)0.4015418,541970.0 %00
3 speakers + screen50relay0.77 (±0.01)0.4114919,8031030.0 %03
+ 4 cameras @ 300 kbit/s50direct0.79 (±0.01)0.6854229,8661700.0 %35
+ 4 cameras @ 1000 kbit/s50direct1.28 (±0.03)1.0454747,2043350.0 %495
+ 10 cameras @ 300 kbit/s50direct1.39 (±0.00)1.0672145,23126414.6 %314
3 speakers100direct0.46 (±0.01)0.3912818,168170.0 %00
3 speakers100relay0.75 (±0.01)0.389320,218190.0 %00
3 speakers + screen100direct1.00 (±0.01)0.8523240,1742110.0 %00
3 speakers + screen100relay1.62 (±0.00)0.8322945,0462370.0 %00
+ 4 cameras @ 300 kbit/s100direct1.35 (±0.01)1.0869349,62523964.0 %7617
+ 4 cameras @ 1000 kbit/s100direct1.41 (±0.01)1.1066448,71428568.1 %993
+ 10 cameras @ 300 kbit/s100direct3.11 (±0.00)0.7381528,27715581.4 %8115

2.4.0 and 2.5.0

Server 2.5.0 brought the memory-bound retransmission history, a retransmission budget per receiver, receiving only the cameras that are on screen (with a cap of 24 per member), and camera layers with a thumbnail. The shapes with cameras were run again on the same host under the same conditions (two runs each, mean shown; worst loss is the worst media class of any run). The load generator publishes each camera as one layer and subscribes to all cameras, so the thumbnail layer is not exercised; "4 watched" shows only the first four cameras (the others hidden through the existing opt-out), which is what a member whose grid shows four tiles receives with the visible-cameras behaviour.

ShapeMembers2.4.0 CPU2.5.0 CPU2.4.0 heap MB2.5.0 heap MB2.4.0 worst loss2.5.0 worst loss2.5.0 generator CPU
4 cameras @ 300 kbit/s250.370.35378820.0 %0.0 %0.33
4 cameras @ 1000 kbit/s250.550.543772460.0 %0.0 %0.51
10 cameras @ 300 kbit/s250.580.557111680.0 %0.0 %0.52
10 cameras @ 300 kbit/s , 4 watched250.580.377111630.0 %0.0 %0.37
4 cameras @ 300 kbit/s500.790.735421070.0 %0.0 %0.66
4 cameras @ 1000 kbit/s501.281.175471130.0 %0.0 %1.01
10 cameras @ 300 kbit/s501.391.1672117114.6 %0.0 %1.03
10 cameras @ 300 kbit/s , 4 watched501.390.7772112214.6 %0.0 %0.72
4 cameras @ 300 kbit/s1001.351.3369320364.0 %54.1 %1.10
4 cameras @ 1000 kbit/s1001.411.3966421068.1 %66.0 %1.13
10 cameras @ 300 kbit/s1003.111.3881532381.4 %77.2 %1.15
10 cameras @ 300 kbit/s , 4 watched1003.111.3581522781.4 %60.4 %1.15
  • Memory: the peak live heap fell by a factor of 1.5 to 5 in every shape (for example 721 MB to 171 MB at 50 members and 10 cameras).
  • CPU: equal within noise up to 25 members, 8 to 17 % lower at 50, and far lower at 100 members with 10 cameras (3.1 to 1.4 cores), where 2.4.0 was spending its time on garbage collection against the memory limit.
  • Loss: 50 members with 10 cameras no longer lose packets (15 % before). At 100 members the shapes with cameras still lose a large share of the screen's packets (54 to 77 %), with the generator at 1.1 cores; see below.
  • Only four cameras watched: at 50 members CPU falls from 1.16 to 0.77 cores and at 25 from 0.55 to 0.37, in line with the egress packets that are no longer sent.

Opus DTX (2.6.0)

Server 2.6.0 turns on Opus discontinuous transmission for the microphone, so a member who is not speaking sends about five small packets a second instead of fifty. The load generator can emulate this (AUDIO_DTX=1, with AUDIO_TALK_PCT the share of time each speaker talks); it sends a 16-byte frame every 200 ms while a speaker is silent, which is what Chrome sends. Two cells, 10 speakers and the rest of the members listening, each run twice with a 120 s hold, with speakers talking 30 % of the time:

MembersDTXEgress packets/sServer CPU (cores)Audio loss
100off58,6141.310 %
100on, 30 % talk time22,3720.580 %
50off28,2160.670 %
50on, 30 % talk time10,4430.280 %

Packets fall by 62 to 63 % and server CPU by 56 to 58 %. This is an emulation with a fixed talk pattern, not real people: real conversations have fewer speakers talking at once than ten, so the saving in an ordinary call is smaller in absolute terms, and music or a steady tone (which the encoder may treat as silence) is why DTX is off when the noise filter is set to Off and can be switched off in Audio settings. The sizing table above assumes continuous speakers (50 packets/s each), which is the conservative case.

The scenarios in Admin > Status

The capacity panel of Admin > Status (operations) turns the measurements above into one row per kind of Talk: the largest Talk of N people whose CPU, memory and upload fit into what is free. Nothing in it is measured anew; each row comes from the runs above, as follows (packets/s, with 6.5 kbit per video packet and 50 packets/s per speaker).

RowIngress packets/sEgress packets/sMemory per personSource
Text chat onlynonenone61 kB residentCPU fitted on the chat runs as 0.148 ms·N + 0.522 µs·N² per second (100 sessions 0.02 cores, 1,000 sessions 0.67; it predicts 0.09 for 300, as measured); memory from 47 MB at 100 to 102 MB at 1,000 sessions
Listening: 3 speakers150150·(N−1)1.3 × 1.28 MBthe "3 speakers" rows; heap 128 MB at 100 members
Conversation: everyone talks50·N50·N·(N−1)1.3 × 1.28 MBderived: N continuous speakers instead of 3 (with DTX real calls send fewer packets, see above)
Screen share + 3 speakers381381·(N−1)1.3 × 2.32 MBthe "3 speakers + screen" rows; heap 232 MB at 100 members
Video call: every camera onN·(50 + 485)N·(50·(N−1) + 77·min(N−1, 24))1.3 × 3.42 MBderived: each camera sends all three layers at their caps (2.5 Mbit/s, 500 and 150 kbit/s, 485 packets/s together) and each member receives the others at the 360p layer (500 kbit/s, 77 packets/s), at most 24 cameras; heap per member from 10 cameras at 50 members on 2.5.0 (171 MB)

CPU is the model above (22 µs per egress packet, 72 µs per ingress packet, 0.57 ms per member per second), scaled by the host's own factor when there is enough traffic, against half of the cores minus what the server uses now. Memory is 1.3 times the heap per person (the working rule above; the 250 MB base is already in the server's resident memory) against 85 % of the container limit minus the resident memory now. Upload is 1.3 times the media the server sends; it limits a row only when VOICE_UPLOAD_LIMIT_MBIT says how fast the line is. The first two rows and the screen share were run up to 100 members (chat up to 1,000); the panel says so when a figure goes beyond that. The conversation and the video call were never run as such: their rows say derived. On the N100 of this page, idle, with the default 1 GB memory limit and no upload limit, that gives about 1,770 people in text chat (CPU), 450 listening (memory), 40 in a conversation (CPU), 210 watching a screen share (CPU, needing about 440 Mbit/s of upload) and 20 in a video call (CPU, about 270 Mbit/s).

Where the packet loss comes from

The shapes with loss all have a screen share and several cameras with 50 to 100 members: 4 cameras at 300 or 1000 kbit/s with 100 members (64 % and 68 % loss in the worst media class) and 10 cameras with 50 and 100 members (15 % and 81 %). The packet counters of the report split the path into three stages. In these runs the packets are lost on the way into the server: for 100 members with 4 cameras at 1 Mbit/s the generator sent 90,769 packets in the window, of which the server counted 42,634 at its ingress (53 % missing), while 100 % of what the server forwarded arrived at the members. The kernel counters in the same runs show a few dozen to about a hundred receive-buffer errors in the container and 3 to 17 hits of the container memory limit (memory.events max), and the generator used 1.1 cores (the plan allowed at most 1.0), so the host was fully shared in these runs. We did not determine the first bottleneck. Two causes are plausible and untested: the generator cannot send as fast as it should while the server and database take the other cores, and the 1 GB memory limit (the 100-member shape with 10 cameras reached a live heap of 815 MB and a resident set of up to 986 MB, close to the 870 MB the application sets as its Go memory limit; its server CPU of 3.1 cores for 28,000 packets/s is four times what the fit predicts at that packet rate and fits garbage collection under that limit). A run with the generator on a second machine, and one with a higher APP_MEM_LIMIT, would tell them apart; neither was done on 2.4.0.

On 2.5.0 the 100-member shape with 10 cameras uses 1.4 cores instead of 3.1 and its heap is 323 MB instead of 815 MB, which supports the explanation that 2.4.0 was collecting garbage against its memory limit; but the loss at the server's ingress remains (77 % of the screen's packets), now with the server far from its limit and the generator on 1.15 cores. That points at the load generator and the shared host as the limit of these runs, not at the server, but we have not shown it: a run with the generator on a second machine is still missing. Later runs found the real cause, the server's socket: see the socket's write lock.

The 100-member shape of the first benchmark (screen, 10 cameras, 3 speakers) lost 81 % of the screen's packets on 2.4.0 and 77 % on 2.5.0. net.core.rmem_max and wmem_max on this host are already 4 MiB, which is what the server requests, so a larger kernel limit alone would not have changed anything.

Limits and caveats

  • One host, one run. There is no repeat, no variance and no other hardware. A different CPU, storage or kernel will differ.
  • Generator on the same host. See above. It inflates latency a little and takes CPU the server could have used.
  • Loopback network. No packet loss, jitter or bandwidth limit between the generator and the server. Real members on the internet add their own latency, and the relay figure does not include a real uplink.
  • Rate limits shape the setup, not the capacity. The server allows 120 WebSocket upgrades per minute per address, so the chat test connected 100 sessions a minute (1,000 sessions took ten minutes to connect), and the relay accepts 120 allocation requests per minute per member and holds 256 allocations in total. These are configured limits, not what the machine could do. The test also stays under the API limit of 600 requests per minute per account.
  • Synthetic media. Voice packets are not real encoded video, and there is no end-to-end encryption work on the generator side. A real browser decrypts and decodes every stream; the server only forwards packets, so server CPU is comparable but client CPU is not covered.
  • Short holds. Each scenario ran for two minutes. Memory growth over hours was not measured here.
  • Not tested: more than 1,000 chat sessions, more than 100 voice members, several Talks at once, mixed chat and voice at the same time, large attachments, and many members signing in at once.
  • Push notifications reached the one real member who was online while the messages were sent (24 deliveries in each chat run).

Reproduce

make load-test-host        # bin/loadtest-host/mnema-loadtest-linux-{amd64,arm64}.tar.gz
scp bin/loadtest-host/mnema-loadtest-linux-amd64.tar.gz <host>:
# on the host: tar xzf ... && cd mnema-loadtest-linux-amd64 && ./host-run.sh

Take a backup first and run it when nobody is using the instance. The full procedure, every setting and the thresholds are in loadtest/README.md. The capacity table in doc/architecture.md lists the same figures in short form.

Something wrong on this page? Tell us.