Monitoring and health
Mnema Talk tells you a lot about itself. Admin > Status is the first place to look; for alerts, it speaks Prometheus.
Admin > Status
The Owner and Administrators get one overview: version and updates, database size and migrations, object storage and free disk space, the last backup and its restore test, live activity (sessions, WebSocket connections, Talks, people in calls), and the setup checks. Every amber or red check names the setting to change.
The Capacity panel shows live CPU, memory and media traffic, and for each kind of Talk about how many people fit and whether CPU, memory or upload runs out first. Test my connection checks from your browser whether voice reaches the server.
Admin > System (Owner only) adds commit, runtime and an operator view with charts of the last hour: uptime, CPU, memory, traffic, errors and ICE failures.
From a terminal: doctor
docker compose exec app mnema-talk doctor
It prints the same checks, one line each with [ ok ], [warn] or [FAIL] and the fix under every amber or red one, and exits with status 1 when a check is red. That makes it fit for scripts.
Health endpoints
| Endpoint | Answers |
|---|---|
GET /api/health | 200 with status, version and features when the database answers and storage is reachable, else 503 naming what is down |
GET /api/health/live | 200 whenever the process serves requests; asks neither database nor storage |
The compose health check uses /api/health. An orchestrator that restarts on a failed probe must use /api/health/live instead: a restart does not bring storage back and would end every running Talk.
Metrics and alerts
Set METRICS_TOKEN (24+ characters, openssl rand -hex 24) in .env and run docker compose up -d. GET /api/metrics then answers in Prometheus format to that bearer token, and 404 without it.
# prometheus.yml
scrape_configs:
- job_name: mnema-talk
metrics_path: /api/metrics
scheme: https
scrape_interval: 15s
authorization:
credentials: <METRICS_TOKEN>
static_configs:
- targets: ["chat.example.com"]
rule_files:
- mnema-alerts.yml
Quick check by hand:
curl -H "Authorization: Bearer $METRICS_TOKEN" https://chat.example.com/api/metrics
Labels are route patterns and fixed states, never user or channel ids. Keep the token out of access logs and allow only your Prometheus host at the proxy if you can.
Ready-made alert rules are in deploy/monitoring/mnema-alerts.yml, checked in CI against the metrics the server exports:
| Alert | Fires when |
|---|---|
MnemaDown | The scrape fails for 2 minutes |
MnemaStorageDown | Object storage is unreachable for 5 minutes |
MnemaHealthFailing | The probe of /api/health fails for 3 minutes |
MnemaDatabasePoolExhausted | Every database connection is busy for 5 minutes |
MnemaMemoryHigh | Resident memory above 800 MB for 15 minutes |
MnemaGoroutineLeak | More than 5,000 goroutines for 15 minutes |
MnemaServerErrors | More than one 5xx every 10 seconds for 10 minutes |
MnemaSlowRequests | 95th percentile API latency above one second for 15 minutes |
MnemaIceFailures | More than 20 % of voice connections fail ICE over 30 minutes |
MnemaStorageErrors | More than 5 object storage errors in 15 minutes |
MnemaGoroutinePanic | The server recovered from a panic in the last 15 minutes |
MnemaBackupTooOld | No successful backup for 36 hours |
One Prometheus container is enough for graphs and the alert list; add Alertmanager or any receiver that speaks its webhook format (ntfy, Gotify, e-mail) to be notified. Without Prometheus, scripts/health-check.sh from cron covers "is it up", with Unraid or webhook notifications.
Logs
docker compose logs --since 10m app
Logs are JSON lines and rotate at 3 × 10 MB per service. For more detail set LOG_LEVEL=debug, run docker compose up -d, reproduce the problem, then remove the line again. On Synology without a shell: Container Manager > Container > mnema-talk > Log.