Victor Wiebe
ade2b10033
Some checks failed
Release production image / production-image (push) Has been cancelled
101 lines
3.9 KiB
Markdown
101 lines
3.9 KiB
Markdown
# Production Operations
|
|
|
|
## Persistent state
|
|
|
|
All durable application state is stored in `/data/conductor.db`: projects, users, password hashes, sessions, publications, encrypted credentials, and execution history. SQLite runs in WAL mode.
|
|
|
|
The database alone is not a complete recovery set. A usable backup must include the exact `CONDUCTOR_SECRET_KEY` that encrypted stored credentials. Production backup archives therefore include the database, `.env`, Compose metadata, a manifest, and checksums. Treat archives as secrets and copy them off-host.
|
|
|
|
## Backup
|
|
|
|
From a source/release directory:
|
|
|
|
```bash
|
|
scripts/production/backup-conductor.sh \
|
|
--compose-file compose.production.yml \
|
|
--env-file .env \
|
|
--backup-dir ./backups \
|
|
--retention-days 7
|
|
```
|
|
|
|
The script invokes the application-owned SQLite online-backup command inside the running container. It does not copy a live WAL database. It then creates a `0600` archive with:
|
|
|
|
- `database/conductor.db`
|
|
- `configuration/.env`
|
|
- `configuration/compose.production.yml`
|
|
- `manifest.json`
|
|
- `SHA256SUMS`
|
|
|
|
It writes progress to stderr and a machine-readable result to stdout and `backup-result.json`.
|
|
|
|
SkeletonWorks uses:
|
|
|
|
```bash
|
|
sudo /opt/skeletonworks/scripts/backup-conductor.sh \
|
|
--fqdn conductor.example.com --retention-days 7
|
|
```
|
|
|
|
Its archive additionally contains `.secrets` and defaults to `/opt/skeletonworks/backups/conductor/<fqdn>/`.
|
|
|
|
## Restore
|
|
|
|
Restore replaces live state. Verify that the chosen archive and the intended target match.
|
|
|
|
```bash
|
|
scripts/production/restore-conductor.sh --list --backup-dir ./backups
|
|
scripts/production/restore-conductor.sh \
|
|
--backup-file ./backups/conductor_<timestamp>.tar.gz \
|
|
--compose-file compose.production.yml \
|
|
--env-file .env
|
|
```
|
|
|
|
Interactive restore requires typing `RESTORE`. Automation must explicitly pass `--force`. By default, restore creates a current-state safety backup, validates paths/checksums/format/keys, stops Conductor, replaces the database, restores UID/GID 10001 and mode `0600`, restarts, and waits for health.
|
|
|
|
SkeletonWorks:
|
|
|
|
```bash
|
|
sudo /opt/skeletonworks/scripts/restore-conductor.sh --fqdn conductor.example.com --list
|
|
sudo /opt/skeletonworks/scripts/restore-conductor.sh \
|
|
--fqdn conductor.example.com --backup-file <archive>
|
|
```
|
|
|
|
Never combine a database with a different encryption key. A checksum-valid but mismatched key cannot decrypt stored REST credentials.
|
|
|
|
## Upgrade
|
|
|
|
1. Read release notes and compatibility warnings.
|
|
2. Create and copy a verified backup off-host.
|
|
3. Change `CONDUCTOR_VERSION` or the pinned SkeletonWorks image.
|
|
4. Pull and recreate.
|
|
5. Wait for health and test login, project load, and a published deep link.
|
|
|
|
```bash
|
|
scripts/production/backup-conductor.sh
|
|
docker compose --env-file .env -f compose.production.yml pull
|
|
docker compose --env-file .env -f compose.production.yml up -d
|
|
curl --fail http://127.0.0.1:8080/api/health
|
|
```
|
|
|
|
SkeletonWorks upgrades are idempotent reruns of `setup-conductor.sh --image <new pinned ref>`.
|
|
|
|
## Rollback
|
|
|
|
Application images and database schema must be treated as a pair. For a failed compatible deployment, return to the prior pinned image. If the release changed the database incompatibly, restore the pre-upgrade archive as well.
|
|
|
|
1. Preserve logs and failed-state data.
|
|
2. Set the previous image tag/digest.
|
|
3. Restore the matching pre-upgrade backup.
|
|
4. Verify health and application smoke tests.
|
|
|
|
## Health and logs
|
|
|
|
- Liveness/ready endpoint: `GET /api/health`
|
|
- Container health: `docker compose ... ps`
|
|
- Logs: `docker compose ... logs --tail=200 conductor`
|
|
|
|
The health response includes the packaged version when built with release metadata.
|
|
|
|
## Recovery objectives
|
|
|
|
The default scripts retain seven daily archives, but retention is not a substitute for off-host storage. Recovery point and recovery time objectives are operator decisions. At least one periodic clean-host restore drill should be scheduled and recorded.
|