Updating
From mnema-talk v3.3.13, rendered on this site.
The one page for updating a Mnema Talk server. Per-version changes you may have to act on are in the upgrade notes; going back is Rollback. Database migrations run by themselves at startup and are forward-only: an older version cannot use a schema a newer one has migrated. So take a backup first, every time.
1. Back up
Back up the database, the media and .env before every update
(details). In the folder with
docker-compose.yml and .env:
BACKUP_DIR=/path/to/backups ./scripts/backup.sh
A server made with the setup assistant has no scripts/ folder yet. Fetch the
scripts of the release you run once (replace X.Y.Z with the version in
Admin > System, or the newest on the
releases page):
base=https://raw.githubusercontent.com/mnema-xyz/talk/vX.Y.Z
mkdir -p scripts && cd scripts
for f in backup.sh backup-common.sh backup-online.sh restore.sh upgrade-postgres.sh health-check.sh; do curl -fsSLO "$base/scripts/$f"; done
chmod +x *.sh && cd ..
2. Choose how to update
Which way is available depends on the tag in MNEMA_IMAGE in .env:
MNEMA_IMAGE tag | Follows | One-click button | Command line |
|---|---|---|---|
:X.Y (for example :3.3) | all patch releases of that line | yes, until the next minor line | docker compose pull && docker compose up -d; a new minor line is a deliberate tag change |
:latest | every release, breaking ones included | always | docker compose pull && docker compose up -d |
:X.Y.Z or @sha256:… | nothing (pinned) | no; the System tab shows the steps | pinned procedure |
mnema-talk:local (built on the server) | your checkout | no | git fetch --tags && git checkout vX.Y.Z && docker compose up -d --build |
The setup assistant writes a tag for the version it came with. Pinned tags are
the most predictable: you choose each version and can go back by setting the
old one. :X.Y is the practical middle ground: a fix: release bumps the patch
and a feat: release the minor, so :X.Y receives only fixes. GHCR has no
:X tag: releases publish X.Y.Z, X.Y and latest there (the Docker Hub
mirror also carries :X).
Still on ghcr.io/msch128/mnema-talk? That path got its last release with
3.2.1 when the repository moved to the mnema-xyz organization, and GHCR does
not redirect: a pull there finds nothing new. Set MNEMA_IMAGE to
ghcr.io/mnema-xyz/mnema-talk with the same tag, then
docker compose pull && docker compose up -d. Admin > System says so while the
old path is set, and the "Update now" button refuses until it is changed.
The "Update now" button
The button belongs to the autoupdate Compose profile:
optional services that start only when enabled.
Admin > System shows the running version, the newest release with its notes
and, when a newer one exists, a dot on the server menu and a badge on "Admin
console" (the server asks GitHub every 30 minutes; UPDATE_CHECK_ENABLED=false
turns it off). Nothing updates automatically. Open tabs show "A new version is
available – click here to reload" after an update, and a call in progress
reconnects after the reload.
The optional updater sidecar adds the button. It verifies the new image's
signature and provenance, pulls that exact digest and restarts only the app;
everyone is disconnected for about 30 to 60 seconds, and it asks for the Owner's
password again. It is off by default. Enable it when the setup assistant asks, or
add two lines to .env and start the profile:
COMPOSE_PROFILES=autoupdate
UPDATER_TOKEN=<output of: openssl rand -hex 32> # at least 32 characters
docker compose up -d
It needs a released image (not mnema-talk:local), and outbound HTTPS to ghcr.io
and to Sigstore (Network ports). The sidecar
holds the Docker socket: read the threat model
first. How the check works: Verifying a release image.
Synology: platform notes.
Unattended updates (cron)
The "Update now" button never runs by itself. For a server that should follow
patch releases on its own, pin the minor line (MNEMA_IMAGE=...:X.Y, a fix:
release only bumps the patch) and let cron do what you would do by hand: back
up, pull, recreate the app, check health and, when the new version fails, go
back to the old one. Save this as /usr/local/sbin/mnema-auto-update,
chmod 700 it, and adjust the paths at the top:
#!/usr/bin/env bash
set -uo pipefail
exec 9>/run/mnema-auto-update.lock; flock -n 9 || exit 0 # never two runs at once
cd /opt/mnema-talk || exit 1 # your deployment folder
backups=/var/backups/mnema-talk # where this script's backups go
bad=/var/lib/mnema-bad-image # id of a release that failed its health check
halted=/var/lib/mnema-update-halted # written when a rollback fails: no updates until you remove it
HEALTH_WAIT=${HEALTH_WAIT:-180} # seconds, at least 5
MIGRATION_WAIT=${MIGRATION_WAIT:-3600} # longest wait while the new version still migrates
[[ $HEALTH_WAIT =~ ^[0-9]+$ ]] && [ "$HEALTH_WAIT" -ge 5 ] || { echo "HEALTH_WAIT must be a number >= 5" >&2; exit 1; }
[[ $MIGRATION_WAIT =~ ^[0-9]+$ ]] || { echo "MIGRATION_WAIT must be a number" >&2; exit 1; }
[ ! -e "$halted" ] || { echo "updates halted after a failed rollback: $(cat "$halted")" >&2; exit 1; }
img=$(sed -n 's/^MNEMA_IMAGE=//p' .env)
[[ ${img##*:} =~ ^[0-9]+\.[0-9]+$ ]] || { echo "MNEMA_IMAGE must be a pinned X.Y tag, got '$img'" >&2; exit 1; }
old=$(docker inspect -f '{{.Image}}' mnema-talk 2>/dev/null || true) # what runs now
docker pull -q "$img" >/dev/null || { echo "pull of $img failed" >&2; exit 1; }
new=$(docker image inspect -f '{{.Id}}' "$img")
[ "$new" != "$old" ] || exit 0 # nothing new
rollback() { [ -n "$old" ] && docker tag "$old" "$img"; } # the tag points at the old image again
if [ "$(cat "$bad" 2>/dev/null)" = "$new" ]; then rollback; exit 0; fi # failed before: do not retry
health() { for _ in $(seq 1 $((HEALTH_WAIT / 5))); do
[ "$(docker inspect -f '{{.State.Health.Status}}' mnema-talk 2>/dev/null)" = healthy ] && return 0; sleep 5; done; return 1; }
# The server holds this advisory lock while it applies migrations.
migrating() { [ "$(docker compose exec -T postgres sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -At -c "SELECT count(*) FROM pg_locks WHERE locktype='"'advisory'"' AND granted AND objsubid=1 AND ((classid::bigint << 32) | objid::bigint) = 470003576161"' </dev/null)" = 1 ]; }
halt() { docker compose stop app; echo "$(date -u +%FT%TZ) $1" > "$halted"
echo "ROLLBACK FAILED: $1. The app is stopped; restore $backup by hand (doc/upgrade.md, Rollback)" >&2; exit 1; }
out=$(BACKUP_MODE=online BACKUP_DIR=$backups KEEP_DAYS=14 ./scripts/backup.sh 2>&1); status=$?
echo "$out"
backup=$(echo "$out" | sed -n 's/^backup complete: \(.*\) (restore test: [a-z]*)$/\1/p') # this run's backup
if [ "$status" != 0 ] || [ ! -d "$backup" ]; then
echo "backup failed, not updating" >&2; rollback; exit 1 # retried at the next run
fi
docker compose up -d # recreates the services whose image changed (app and updater)
waited=0
while ! health && migrating; do # never decide while a migration still runs
[ "$waited" -lt "$MIGRATION_WAIT" ] || { echo "still migrating after ${MIGRATION_WAIT}s, not rolling back: check by hand" >&2; exit 1; }
echo "a migration is still running, waiting"; waited=$((waited + HEALTH_WAIT))
done
health && exit 0 # after a migration, the health check gets HEALTH_WAIT once more
echo "new version not healthy, going back" >&2
echo "$new" > "$bad"; rollback; docker compose up -d
health && exit 0
# The old version refuses a database the new one migrated: restore this run's backup.
[ "$(docker compose logs app | grep -c 'migrated by a newer version')" != 0 ] || halt "the old version is unhealthy too"
(cd "$backup" && sha256sum --quiet -c SHA256SUMS) || halt "backup $backup fails its checksums"
echo "the old version refuses the migrated database, restoring $backup" >&2
docker compose stop app
./scripts/restore.sh "$backup" --yes || halt "restore failed"
docker compose up -d
health || halt "the old version is unhealthy after the restore"
echo "rolled back and restored $backup: writes made since that backup are lost" >&2
exit 1
Run it hourly from /etc/cron.d/mnema-auto-update
(10 * * * * root /usr/local/sbin/mnema-auto-update >> /var/log/mnema-auto-update.log 2>&1)
and keep it clear of the nightly backup slot. Nothing alerts you: the log file
is the only report, so read it now and then (or add a mail or webhook call at the
echo lines). A release that failed its health check is remembered in
/var/lib/mnema-bad-image and not retried until a newer image appears; a failed
backup is retried at the next run. If old is empty (the app container was
missing) the script cannot roll back and only reports it.
When the new version does not become healthy, the script goes back in steps:
- It waits while the new version still applies migrations (up to
MIGRATION_WAIT), so it never rolls back in the middle of one. - It points the tag at the old image and starts it. Without schema changes this is enough, and no data is lost.
- Only if the old version refuses the database (the new one already migrated
it:
the database was migrated by a newer versionin the log) does it check the checksums of the backup it took minutes earlier, restore exactly that backup withrestore.shand start the old version again. Messages and uploads made between that backup and the rollback are lost. - If the restore or the old version fails as well, it stops the app, writes
/var/lib/mnema-update-haltedand makes no further attempts. Repair by hand (Rollback) and remove that file to resume updates.
Things to know:
- Members are disconnected for 30 to 60 seconds during an update, so pick the hours that suit your community (change the cron hour field).
- This path pulls by tag and does not verify the signature the way the "Update now" button does (Verifying a release image). Use a pinned digest and the button if that matters more than automation.
- A new minor line (a
feat:release) is never picked up; it can change the compose file, so move to it by hand as described above. - The backup is taken online, so the app keeps serving while it runs; writes made during the backup itself are not in it either.
- The script reads no secret and prints none:
restore.shandbackup.shread.envthemselves, and the database query runs inside the postgres container.
Command line within a line
For :X.Y and :latest:
cd /path/to/mnema-talk
docker compose pull && docker compose up -d
Compose recreates only the services whose image changed; the downtime is a few seconds. Then verify.
On Synology without SSH: Container Manager > Project > mnema-talk > Action >
Stop, then Clean (removes the project's containers and images, not your data)
and Build, which pulls the current image of the tag again. Menu wording differs
a little between DSM versions.
Pinned or new minor line
Run in the deployment folder. The first lines look up the newest release; set
NEW yourself to install another.
NEW=$(curl -fsSL https://api.github.com/repos/mnema-xyz/talk/releases/latest | sed -n 's/.*"tag_name": *"v\([^"]*\)".*/\1/p') # the newest release, or set a version
[ -n "$NEW" ] || { echo "could not read the latest version; set NEW=x.y.z yourself" >&2; false; }
# Note what runs now, for a rollback
grep '^MNEMA_IMAGE=' .env
curl -fsS http://127.0.0.1:8080/api/health
cp docker-compose.yml docker-compose.yml.bak
# Fetch the compose file (and scripts) of the new release
base=https://raw.githubusercontent.com/mnema-xyz/talk/v$NEW
curl -fsSLo docker-compose.yml "$base/docker-compose.yml"
curl -fsSLo .env.example.new "$base/.env.example"
for f in backup.sh backup-common.sh backup-online.sh restore.sh upgrade-postgres.sh health-check.sh; do
curl -fsSLo "scripts/$f" "$base/scripts/$f"
done
chmod +x scripts/*.sh
# New or changed variables since your .env was written
diff <(grep -o '^[A-Z_0-9]*=' .env | sort) <(grep -o '^[A-Z_0-9]*=' .env.example.new | sort)
# Point the app at the new image
if grep -q '^MNEMA_IMAGE=' .env; then
sed -i "s|^MNEMA_IMAGE=.*|MNEMA_IMAGE=ghcr.io/mnema-xyz/mnema-talk:$NEW|" .env
else
echo "MNEMA_IMAGE=ghcr.io/mnema-xyz/mnema-talk:$NEW" >> .env
fi
# Check the configuration, pull and start
docker compose config -q
docker compose pull app
docker compose up -d # add --profile turn / --profile autoupdate if you use them
(sed -i as written is for Linux; on macOS use sed -i ''.) The same image is
also published on Docker Hub as docker.io/mnemaxyz/mnema-talk (same digest and
signature); GHCR stays the primary reference. With
MNEMA_IMAGE=mnemaxyz/mnema-talk:<tag> "Update now" works too and pulls from
Docker Hub.
3. Read the notes
For every release you skip over, read its entry in the upgrade notes:
some need a setting changed before you update, for example a narrower
TRUSTED_PROXY_CIDRS (Broad trusted proxies).
Check the changelog before updating across a release that changed the schema.
4. Verify
docker compose ps # app "healthy" after about 15-30 seconds
curl -fsS http://127.0.0.1:8080/api/health
docker compose logs --since 10m app | grep -E 'migration applied|"level":"(WARN|ERROR)"'
Open the app: other open tabs show the reload notice, and Admin > System shows the new version. Then look at Admin > Status for new amber or red items.
If it goes wrong
Stop the app and follow Rollback. Because migrations are
forward-only, going back across a release that changed the database means
restoring the backup from step 1. Keep that backup and the saved
docker-compose.yml.bak until the new version has passed your own checks.