Monitoring and health

Mnema Talk tells you a lot about itself. Admin > Status is the first place to look; for alerts, it speaks Prometheus.

Admin > Status

The Owner and Administrators get one overview: version and updates, database size and migrations, object storage and free disk space, the last backup and its restore test, live activity (sessions, WebSocket connections, Talks, people in calls), and the setup checks. Every amber or red check names the setting to change.

The Capacity panel shows live CPU, memory and media traffic, and for each kind of Talk about how many people fit and whether CPU, memory or upload runs out first. Test my connection checks from your browser whether voice reaches the server.

Admin > System (Owner only) adds commit, runtime and an operator view with charts of the last hour: uptime, CPU, memory, traffic, errors and ICE failures.

From a terminal: doctor

docker compose exec app mnema-talk doctor

It prints the same checks, one line each with [ ok ], [warn] or [FAIL] and the fix under every amber or red one, and exits with status 1 when a check is red. That makes it fit for scripts.

Health endpoints

EndpointAnswers
GET /api/health200 with status, version and features when the database answers and storage is reachable, else 503 naming what is down
GET /api/health/live200 whenever the process serves requests; asks neither database nor storage

The compose health check uses /api/health. An orchestrator that restarts on a failed probe must use /api/health/live instead: a restart does not bring storage back and would end every running Talk.

Metrics and alerts

Set METRICS_TOKEN (24+ characters, openssl rand -hex 24) in .env and run docker compose up -d. GET /api/metrics then answers in Prometheus format to that bearer token, and 404 without it.

# prometheus.yml
scrape_configs:
  - job_name: mnema-talk
    metrics_path: /api/metrics
    scheme: https
    scrape_interval: 15s
    authorization:
      credentials: <METRICS_TOKEN>
    static_configs:
      - targets: ["chat.example.com"]
rule_files:
  - mnema-alerts.yml

Quick check by hand:

curl -H "Authorization: Bearer $METRICS_TOKEN" https://chat.example.com/api/metrics

Labels are route patterns and fixed states, never user or channel ids. Keep the token out of access logs and allow only your Prometheus host at the proxy if you can.

Ready-made alert rules are in deploy/monitoring/mnema-alerts.yml, checked in CI against the metrics the server exports:

AlertFires when
MnemaDownThe scrape fails for 2 minutes
MnemaStorageDownObject storage is unreachable for 5 minutes
MnemaHealthFailingThe probe of /api/health fails for 3 minutes
MnemaDatabasePoolExhaustedEvery database connection is busy for 5 minutes
MnemaMemoryHighResident memory above 800 MB for 15 minutes
MnemaGoroutineLeakMore than 5,000 goroutines for 15 minutes
MnemaServerErrorsMore than one 5xx every 10 seconds for 10 minutes
MnemaSlowRequests95th percentile API latency above one second for 15 minutes
MnemaIceFailuresMore than 20 % of voice connections fail ICE over 30 minutes
MnemaStorageErrorsMore than 5 object storage errors in 15 minutes
MnemaGoroutinePanicThe server recovered from a panic in the last 15 minutes
MnemaBackupTooOldNo successful backup for 36 hours

One Prometheus container is enough for graphs and the alert list; add Alertmanager or any receiver that speaks its webhook format (ntfy, Gotify, e-mail) to be notified. Without Prometheus, scripts/health-check.sh from cron covers "is it up", with Unraid or webhook notifications.

Logs

docker compose logs --since 10m app

Logs are JSON lines and rotate at 3 × 10 MB per service. For more detail set LOG_LEVEL=debug, run docker compose up -d, reproduce the problem, then remove the line again. On Synology without a shell: Container Manager > Container > mnema-talk > Log.

Something wrong on this page? Tell us.