Production Operations

Persistent state

All durable application state is stored in /data/conductor.db: projects, users, password hashes, sessions, publications, encrypted credentials, and execution history. SQLite runs in WAL mode.

The database alone is not a complete recovery set. A usable backup must include the exact CONDUCTOR_SECRET_KEY that encrypted stored credentials. Production backup archives therefore include the database, .env, Compose metadata, a manifest, and checksums. Treat archives as secrets and copy them off-host.

Backup

From a source/release directory:

scripts/production/backup-conductor.sh \
  --compose-file compose.production.yml \
  --env-file .env \
  --backup-dir ./backups \
  --retention-days 7

The script invokes the application-owned SQLite online-backup command inside the running container. It does not copy a live WAL database. It then creates a 0600 archive with:

database/conductor.db
configuration/.env
configuration/compose.production.yml
manifest.json
SHA256SUMS

It writes progress to stderr and a machine-readable result to stdout and backup-result.json.

SkeletonWorks uses:

sudo /opt/skeletonworks/scripts/backup-conductor.sh \
  --fqdn conductor.example.com --retention-days 7

Its archive additionally contains .secrets and defaults to /opt/skeletonworks/backups/conductor/<fqdn>/.

Restore

Restore replaces live state. Verify that the chosen archive and the intended target match.

scripts/production/restore-conductor.sh --list --backup-dir ./backups
scripts/production/restore-conductor.sh \
  --backup-file ./backups/conductor_<timestamp>.tar.gz \
  --compose-file compose.production.yml \
  --env-file .env

Interactive restore requires typing RESTORE. Automation must explicitly pass --force. By default, restore creates a current-state safety backup, validates paths/checksums/format/keys, stops Conductor, replaces the database, restores UID/GID 10001 and mode 0600, restarts, and waits for health.

SkeletonWorks:

sudo /opt/skeletonworks/scripts/restore-conductor.sh --fqdn conductor.example.com --list
sudo /opt/skeletonworks/scripts/restore-conductor.sh \
  --fqdn conductor.example.com --backup-file <archive>

Never combine a database with a different encryption key. A checksum-valid but mismatched key cannot decrypt stored REST credentials.

Upgrade

Read release notes and compatibility warnings.
Create and copy a verified backup off-host.
Change CONDUCTOR_VERSION or the pinned SkeletonWorks image.
Pull and recreate.
Wait for health and test login, project load, and a published deep link.

scripts/production/backup-conductor.sh
docker compose --env-file .env -f compose.production.yml pull
docker compose --env-file .env -f compose.production.yml up -d
curl --fail http://127.0.0.1:8080/api/health

SkeletonWorks upgrades are idempotent reruns of setup-conductor.sh --image <new pinned ref>.

Rollback

Application images and database schema must be treated as a pair. For a failed compatible deployment, return to the prior pinned image. If the release changed the database incompatibly, restore the pre-upgrade archive as well.

Preserve logs and failed-state data.
Set the previous image tag/digest.
Restore the matching pre-upgrade backup.
Verify health and application smoke tests.

Health and logs

Liveness/ready endpoint: GET /api/health
Container health: docker compose ... ps
Logs: docker compose ... logs --tail=200 conductor

The health response includes the packaged version when built with release metadata.

Recovery objectives

The default scripts retain seven daily archives, but retention is not a substitute for off-host storage. Recovery point and recovery time objectives are operator decisions. At least one periodic clean-host restore drill should be scheduled and recorded.

3.9 KiB Raw Blame History