# Production Operations

## Persistent state

All durable application state is stored in `/data/conductor.db`: projects, users, password hashes, sessions, publications, encrypted credentials, and execution history. SQLite runs in WAL mode.

The database alone is not a complete recovery set. A usable backup must include the exact `CONDUCTOR_SECRET_KEY` that encrypted stored credentials. Production backup archives therefore include the database, `.env`, Compose metadata, a manifest, and checksums. Treat archives as secrets and copy them off-host.

## Backup

From a source/release directory:

```bash
scripts/production/backup-conductor.sh \
  --compose-file compose.production.yml \
  --env-file .env \
  --backup-dir ./backups \
  --retention-days 7
```

The script invokes the application-owned SQLite online-backup command inside the running container. It does not copy a live WAL database. It then creates a `0600` archive with:

- `database/conductor.db`
- `configuration/.env`
- `configuration/compose.production.yml`
- `manifest.json`
- `SHA256SUMS`

It writes progress to stderr and a machine-readable result to stdout and `backup-result.json`.

SkeletonWorks uses:

```bash
sudo /opt/skeletonworks/scripts/backup-conductor.sh \
  --fqdn conductor.example.com --retention-days 7
```

Its archive additionally contains `.secrets` and defaults to `/opt/skeletonworks/backups/conductor/<fqdn>/`.

## Restore

Restore replaces live state. Verify that the chosen archive and the intended target match.

```bash
scripts/production/restore-conductor.sh --list --backup-dir ./backups
scripts/production/restore-conductor.sh \
  --backup-file ./backups/conductor_<timestamp>.tar.gz \
  --compose-file compose.production.yml \
  --env-file .env
```

Interactive restore requires typing `RESTORE`. Automation must explicitly pass `--force`. By default, restore creates a current-state safety backup, validates paths/checksums/format/keys, stops Conductor, replaces the database, restores UID/GID 10001 and mode `0600`, restarts, and waits for health.

SkeletonWorks:

```bash
sudo /opt/skeletonworks/scripts/restore-conductor.sh --fqdn conductor.example.com --list
sudo /opt/skeletonworks/scripts/restore-conductor.sh \
  --fqdn conductor.example.com --backup-file <archive>
```

Never combine a database with a different encryption key. A checksum-valid but mismatched key cannot decrypt stored REST credentials.

## Upgrade

1. Read release notes and compatibility warnings.
2. Create and copy a verified backup off-host.
3. Change `CONDUCTOR_VERSION` or the pinned SkeletonWorks image.
4. Pull and recreate.
5. Wait for health and test login, project load, and a published deep link.

```bash
scripts/production/backup-conductor.sh
docker compose --env-file .env -f compose.production.yml pull
docker compose --env-file .env -f compose.production.yml up -d
curl --fail http://127.0.0.1:8080/api/health
```

SkeletonWorks upgrades are idempotent reruns of `setup-conductor.sh --image <new pinned ref>`.

## Rollback

Application images and database schema must be treated as a pair. For a failed compatible deployment, return to the prior pinned image. If the release changed the database incompatibly, restore the pre-upgrade archive as well.

1. Preserve logs and failed-state data.
2. Set the previous image tag/digest.
3. Restore the matching pre-upgrade backup.
4. Verify health and application smoke tests.

## Health and logs

- Liveness/ready endpoint: `GET /api/health`
- Container health: `docker compose ... ps`
- Logs: `docker compose ... logs --tail=200 conductor`

The health response includes the packaged version when built with release metadata.

## Recovery objectives

The default scripts retain seven daily archives, but retention is not a substitute for off-host storage. Recovery point and recovery time objectives are operator decisions. At least one periodic clean-host restore drill should be scheduled and recorded.