conductor/docs/OPERATIONS.md

# Production Operations

## Persistent state

All durable application state is stored in `/data/conductor.db`: projects, users, password hashes, sessions, publications, encrypted credentials, and execution history. SQLite runs in WAL mode.

The database alone is not a complete recovery set. A usable backup must include the exact `CONDUCTOR_SECRET_KEY` that encrypted stored credentials. Production backup archives therefore include the database, `.env`, Compose metadata, a manifest, and checksums. Treat archives as secrets and copy them off-host.

## Production configuration

The public Compose installation reads `.env`. Keep it mode `0600`, exclude it from source control, and retain it with the disaster-recovery set.

| Variable | Purpose and operating rule |
|---|---|
| `CONDUCTOR_SECRET_KEY` | Required 32-byte encryption key encoded as 64 hexadecimal characters or base64. Generate once and preserve it with the database. Changing or losing it makes existing stored credentials unreadable. |
| `CONDUCTOR_SESSION_KEY` | Required value of at least 32 bytes used to protect sessions. Generate independently from the encryption key. Changing it signs everyone out. |
| `CONDUCTOR_IMAGE` | Public registry/repository name without a tag. |
| `CONDUCTOR_VERSION` | Pinned published image tag. Do not use a moving `latest` tag for a controlled installation. |
| `CONDUCTOR_BIND_ADDRESS` | Use `127.0.0.1` when the reverse proxy runs on the same host. |
| `CONDUCTOR_PORT` | Host loopback port forwarded to container port 8080. |
| `CONDUCTOR_MEM_LIMIT`, `CONDUCTOR_CPUS` | Container resource ceilings. Increase deliberately after measuring demand. |
| `CONDUCTOR_PROXY_INTERNAL_ORIGINS` | Optional comma-separated exact origins approved to reach private/internal networks. It accepts schemes, hostnames, and effective ports—not wildcards, paths, credentials, or IP literals. |

Production refuses to start if either required key is absent or invalid. Generate them independently with `openssl rand -hex 32`; never paste their output into documentation, tickets, or screenshots.

After changing a non-secret configuration value, recreate the service and verify health:

```bash
docker compose --env-file .env -f compose.production.yml up -d
curl --fail http://127.0.0.1:8080/api/health
```

For a SkeletonWorks deployment, edit `/opt/skeletonworks/conductor/<fqdn>/.env`, recreate that stack, and check its HTTPS health endpoint:

```bash
sudoedit /opt/skeletonworks/conductor/conductor.example.com/.env
sudo docker compose \
  -f /opt/skeletonworks/conductor/conductor.example.com/docker-compose.yml \
  up -d
curl --fail https://conductor.example.com/api/health
```

The current SkeletonWorks setup script preserves the two keys in `.secrets`, but rewrites `CONDUCTOR_PROXY_INTERNAL_ORIGINS` to an empty value when it is rerun. Until that installer accepts and preserves an internal-origin setting, review and reapply the intended value after every setup or upgrade rerun before testing affected applications.

Do not casually rotate `CONDUCTOR_SECRET_KEY`; v1.0.0 has no online credential re-encryption procedure. Read [Secrets](SECRETS.md) and [Proxy Security](PROXY_SECURITY.md) before changing key or internal-origin policy.

## Backup

From a source/release directory:

```bash
scripts/production/backup-conductor.sh \
  --compose-file compose.production.yml \
  --env-file .env \
  --backup-dir ./backups \
  --retention-days 7
```

The script invokes the application-owned SQLite online-backup command inside the running container. It does not copy a live WAL database. It then creates a `0600` archive with:

- `database/conductor.db`
- `configuration/.env`
- `configuration/compose.production.yml`
- `manifest.json`
- `SHA256SUMS`

It writes progress to stderr and a machine-readable result to stdout and `backup-result.json`.

SkeletonWorks uses:

```bash
sudo /opt/skeletonworks/scripts/backup-conductor.sh \
  --fqdn conductor.example.com --retention-days 7
```

Its archive additionally contains `.secrets` and defaults to `/opt/skeletonworks/backups/conductor/<fqdn>/`.

## Restore

Restore replaces live state. Verify that the chosen archive and the intended target match.

```bash
scripts/production/restore-conductor.sh --list --backup-dir ./backups
scripts/production/restore-conductor.sh \
  --backup-file ./backups/conductor_<timestamp>.tar.gz \
  --compose-file compose.production.yml \
  --env-file .env
```

Interactive restore requires typing `RESTORE`. Automation must explicitly pass `--force`. By default, restore creates a current-state safety backup, validates paths/checksums/format/keys, stops Conductor, replaces the database, restores UID/GID 10001 and mode `0600`, restarts, and waits for health.

SkeletonWorks:

```bash
sudo /opt/skeletonworks/scripts/restore-conductor.sh --fqdn conductor.example.com --list
sudo /opt/skeletonworks/scripts/restore-conductor.sh \
  --fqdn conductor.example.com --backup-file <archive>
```

Never combine a database with a different encryption key. A checksum-valid but mismatched key cannot decrypt stored REST credentials.

## Administrative bootstrap and recovery

Browser setup is the normal way to create the first administrator. The following fallback works only while the database contains no users. It invokes the compiled command inside the running production service:

```bash
read -r -p 'New administrator username: ' CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME
read -r -s -p 'New administrator password (12+ characters): ' CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD
printf '\n'
export CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD

docker compose --env-file .env -f compose.production.yml exec -T \
  -e CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME \
  -e CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD \
  conductor node dist/scripts/bootstrapAdmin.js

unset CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD
```

If every administrator is inaccessible, recover one existing local administrator. Recovery cannot create an account or elevate a user. It re-enables the named administrator, replaces the password hash, and revokes that account's sessions:

```bash
read -r -p 'Existing administrator username: ' CONDUCTOR_RECOVERY_ADMIN_USERNAME
read -r -s -p 'New administrator password (12+ characters): ' CONDUCTOR_RECOVERY_ADMIN_PASSWORD
printf '\n'
export CONDUCTOR_RECOVERY_ADMIN_USERNAME CONDUCTOR_RECOVERY_ADMIN_PASSWORD
export CONDUCTOR_ALLOW_ADMIN_RECOVERY=I_UNDERSTAND

docker compose --env-file .env -f compose.production.yml exec -T \
  -e CONDUCTOR_ALLOW_ADMIN_RECOVERY \
  -e CONDUCTOR_RECOVERY_ADMIN_USERNAME \
  -e CONDUCTOR_RECOVERY_ADMIN_PASSWORD \
  conductor node dist/scripts/bootstrapAdmin.js

unset CONDUCTOR_ALLOW_ADMIN_RECOVERY CONDUCTOR_RECOVERY_ADMIN_USERNAME CONDUCTOR_RECOVERY_ADMIN_PASSWORD
```

Run recovery from a private operator terminal, confirm the target carefully, and clear any terminal capture governed by local policy. Create and retain a second enabled administrator so routine recovery can remain in the browser.

For SkeletonWorks, run the corresponding command from the stack directory in a root shell so the password variables are available to Compose. For an empty-installation bootstrap:

```bash
sudo -i
cd /opt/skeletonworks/conductor/conductor.example.com

read -r -p 'New administrator username: ' CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME
read -r -s -p 'New administrator password (12+ characters): ' CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD
printf '\n'
export CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD

docker compose -f docker-compose.yml exec -T \
  -e CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME \
  -e CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD \
  conductor node dist/scripts/bootstrapAdmin.js

unset CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD
exit
```

For guarded recovery of an existing administrator:

```bash
sudo -i
cd /opt/skeletonworks/conductor/conductor.example.com

read -r -p 'Existing administrator username: ' CONDUCTOR_RECOVERY_ADMIN_USERNAME
read -r -s -p 'New administrator password (12+ characters): ' CONDUCTOR_RECOVERY_ADMIN_PASSWORD
printf '\n'
export CONDUCTOR_RECOVERY_ADMIN_USERNAME CONDUCTOR_RECOVERY_ADMIN_PASSWORD
export CONDUCTOR_ALLOW_ADMIN_RECOVERY=I_UNDERSTAND

docker compose -f docker-compose.yml exec -T \
  -e CONDUCTOR_ALLOW_ADMIN_RECOVERY \
  -e CONDUCTOR_RECOVERY_ADMIN_USERNAME \
  -e CONDUCTOR_RECOVERY_ADMIN_PASSWORD \
  conductor node dist/scripts/bootstrapAdmin.js

unset CONDUCTOR_ALLOW_ADMIN_RECOVERY CONDUCTOR_RECOVERY_ADMIN_USERNAME CONDUCTOR_RECOVERY_ADMIN_PASSWORD
exit
```

Replace `conductor.example.com` with the deployed FQDN. Browser setup and another enabled administrator remain the preferred paths.

## Upgrade

1. Read release notes and compatibility warnings.
2. Create and copy a verified backup off-host.
3. Change `CONDUCTOR_VERSION` or the pinned SkeletonWorks image.
4. Pull and recreate.
5. Wait for health and test login, project load, and a published deep link.

```bash
scripts/production/backup-conductor.sh
docker compose --env-file .env -f compose.production.yml pull
docker compose --env-file .env -f compose.production.yml up -d
curl --fail http://127.0.0.1:8080/api/health
```

SkeletonWorks upgrades are idempotent reruns of `setup-conductor.sh --image <new pinned ref>`.

## Rollback

Application images and database schema must be treated as a pair. For a failed compatible deployment, return to the prior pinned image. If the release changed the database incompatibly, restore the pre-upgrade archive as well.

1. Preserve logs and failed-state data.
2. Set the previous image tag/digest.
3. Restore the matching pre-upgrade backup.
4. Verify health and application smoke tests.

## Health and logs

- Liveness/ready endpoint: `GET /api/health`
- Container health: `docker compose ... ps`
- Logs: `docker compose ... logs --tail=200 conductor`

The health response includes the packaged version when built with release metadata.

## Recovery objectives

The default scripts retain seven daily archives, but retention is not a substitute for off-host storage. Recovery point and recovery time objectives are operator decisions. At least one periodic clean-host restore drill should be scheduled and recorded.