conductor/docs/OPERATIONS.md

# Production Operations

## Persistent state

All durable application state is stored in `/data/conductor.db`: projects, users, password hashes, sessions, publications, encrypted credentials, and execution history. SQLite runs in WAL mode.

The database alone is not a complete recovery set. A usable backup must include the exact `CONDUCTOR_SECRET_KEY` that encrypted stored credentials. Production backup archives therefore include the database, `.env`, Compose metadata, a manifest, and checksums. Treat archives as secrets and copy them off-host.

## Production configuration

The public Compose installation reads `.env`. Keep it mode `0600`, exclude it from source control, and retain it with the disaster-recovery set.

| Variable | Purpose and operating rule |
|---|---|
| `CONDUCTOR_SECRET_KEY` | Required 32-byte encryption key encoded as 64 hexadecimal characters or base64. Generate once and preserve it with the database. Changing or losing it makes existing stored credentials unreadable. |
| `CONDUCTOR_SESSION_KEY` | Required value of at least 32 bytes used to protect sessions. Generate independently from the encryption key. Changing it signs everyone out. |
| `CONDUCTOR_IMAGE` | Public registry/repository name without a tag. |
| `CONDUCTOR_VERSION` | Pinned published image tag. Do not use a moving `latest` tag for a controlled installation. |
| `CONDUCTOR_BIND_ADDRESS` | Use `127.0.0.1` when the reverse proxy runs on the same host. |
| `CONDUCTOR_PORT` | Host loopback port forwarded to container port 8080. |
| `CONDUCTOR_MEM_LIMIT`, `CONDUCTOR_CPUS` | Container resource ceilings. Increase deliberately after measuring demand. |
| `CONDUCTOR_PROXY_INTERNAL_ORIGINS` | Optional comma-separated exact origins approved to reach private/internal networks. It accepts schemes, hostnames, and effective ports—not wildcards, paths, credentials, or IP literals. |

Production refuses to start if either required key is absent or invalid. Generate them independently with `openssl rand -hex 32`; never paste their output into documentation, tickets, or screenshots.

After changing a non-secret configuration value, recreate the service and verify health:

```bash
docker compose --env-file .env -f compose.production.yml up -d
curl --fail http://127.0.0.1:8080/api/health
```

For a SkeletonWorks deployment, edit `/opt/skeletonworks/conductor/<fqdn>/.env`, recreate that stack, and check its HTTPS health endpoint:

```bash
sudoedit /opt/skeletonworks/conductor/conductor.example.com/.env
sudo docker compose \
  -f /opt/skeletonworks/conductor/conductor.example.com/docker-compose.yml \
  up -d
curl --fail https://conductor.example.com/api/health
```

The current SkeletonWorks setup script preserves the two keys in `.secrets`, but rewrites `CONDUCTOR_PROXY_INTERNAL_ORIGINS` to an empty value when it is rerun. Until that installer accepts and preserves an internal-origin setting, review and reapply the intended value after every setup or upgrade rerun before testing affected applications.

Do not casually rotate `CONDUCTOR_SECRET_KEY`; v1.1.0 has no online credential re-encryption procedure. Losing or changing it makes existing stored credentials unrecoverable. Changing `CONDUCTOR_SESSION_KEY` is safe when an intentional global sign-out is acceptable.

### Outbound proxy policy

Treat project URLs, DNS answers, redirects, upstream data, and errors as untrusted.

- Only `http` and `https` URLs without embedded credentials are accepted.
- DNS must return only publicly routable addresses unless the exact hostname origin is operator-approved. Loopback, private, link-local, metadata, multicast, reserved, and mixed safe/unsafe answers fail closed.
- Every redirect is revalidated. At most three are followed, and authentication or sensitive author headers are stripped when the origin changes.
- Internal exceptions are comma-separated exact origins such as `https://api.internal.example:8443`. Schemes and effective ports must match; paths, wildcards, credentials, and IP literals are invalid.
- Author request bodies are limited to 1 MiB, proxy API JSON to 2 MiB, responses to 5 MiB, and execution-history previews to 2 KiB. The fixed total timeout is 30 seconds.
- Permit required private destinations narrowly and constrain container egress at the network layer when stronger isolation is needed.

Encryption at rest protects against disclosure of SQLite alone. It does not protect against compromise of the running backend, a hostile host administrator, memory inspection, a compromised REST destination, or disclosure of both the database and encryption key. Prefer header-based API keys over query-string keys when an upstream supports both.

## Backup

From a source/release directory:

```bash
scripts/production/backup-conductor.sh \
  --compose-file compose.production.yml \
  --env-file .env \
  --backup-dir ./backups \
  --retention-days 7
```

The script invokes the application-owned SQLite online-backup command inside the running container. It does not copy a live WAL database. It then creates a `0600` archive with:

- `database/conductor.db`
- `configuration/.env`
- `configuration/compose.production.yml`
- `manifest.json`
- `SHA256SUMS`

It writes progress to stderr and a machine-readable result to stdout and `backup-result.json`.

SkeletonWorks uses:

```bash
sudo /opt/skeletonworks/scripts/backup-conductor.sh \
  --fqdn conductor.example.com --retention-days 7
```

Its archive additionally contains `.secrets` and defaults to `/opt/skeletonworks/backups/conductor/<fqdn>/`.

## Restore

Restore replaces live state. Verify that the chosen archive and the intended target match.

```bash
scripts/production/restore-conductor.sh --list --backup-dir ./backups
scripts/production/restore-conductor.sh \
  --backup-file ./backups/conductor_<timestamp>.tar.gz \
  --compose-file compose.production.yml \
  --env-file .env
```

Interactive restore requires typing `RESTORE`. Automation must explicitly pass `--force`. By default, restore creates a current-state safety backup, validates paths/checksums/format/keys, stops Conductor, replaces the database, restores UID/GID 10001 and mode `0600`, restarts, and waits for health.

SkeletonWorks:

```bash
sudo /opt/skeletonworks/scripts/restore-conductor.sh --fqdn conductor.example.com --list
sudo /opt/skeletonworks/scripts/restore-conductor.sh \
  --fqdn conductor.example.com --backup-file <archive>
```

Never combine a database with a different encryption key. A checksum-valid but mismatched key cannot decrypt stored REST credentials.

## Administrative bootstrap and recovery

Browser setup is the normal way to create the first administrator. The following fallback works only while the database contains no users. It invokes the compiled command inside the running production service:

```bash
read -r -p 'New administrator username: ' CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME
read -r -s -p 'New administrator password (12+ characters): ' CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD
printf '\n'
export CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD

docker compose --env-file .env -f compose.production.yml exec -T \
  -e CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME \
  -e CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD \
  conductor node dist/scripts/bootstrapAdmin.js

unset CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD
```

If every administrator is inaccessible, recover one existing local administrator. Recovery cannot create an account or elevate a user. It re-enables the named administrator, replaces the password hash, and revokes that account's sessions:

```bash
read -r -p 'Existing administrator username: ' CONDUCTOR_RECOVERY_ADMIN_USERNAME
read -r -s -p 'New administrator password (12+ characters): ' CONDUCTOR_RECOVERY_ADMIN_PASSWORD
printf '\n'
export CONDUCTOR_RECOVERY_ADMIN_USERNAME CONDUCTOR_RECOVERY_ADMIN_PASSWORD
export CONDUCTOR_ALLOW_ADMIN_RECOVERY=I_UNDERSTAND

docker compose --env-file .env -f compose.production.yml exec -T \
  -e CONDUCTOR_ALLOW_ADMIN_RECOVERY \
  -e CONDUCTOR_RECOVERY_ADMIN_USERNAME \
  -e CONDUCTOR_RECOVERY_ADMIN_PASSWORD \
  conductor node dist/scripts/bootstrapAdmin.js

unset CONDUCTOR_ALLOW_ADMIN_RECOVERY CONDUCTOR_RECOVERY_ADMIN_USERNAME CONDUCTOR_RECOVERY_ADMIN_PASSWORD
```

Run recovery from a private operator terminal, confirm the target carefully, and clear any terminal capture governed by local policy. Create and retain a second enabled administrator so routine recovery can remain in the browser.

For SkeletonWorks, run the corresponding command from the stack directory in a root shell so the password variables are available to Compose. For an empty-installation bootstrap:

```bash
sudo -i
cd /opt/skeletonworks/conductor/conductor.example.com

read -r -p 'New administrator username: ' CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME
read -r -s -p 'New administrator password (12+ characters): ' CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD
printf '\n'
export CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD

docker compose -f docker-compose.yml exec -T \
  -e CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME \
  -e CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD \
  conductor node dist/scripts/bootstrapAdmin.js

unset CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD
exit
```

For guarded recovery of an existing administrator:

```bash
sudo -i
cd /opt/skeletonworks/conductor/conductor.example.com

read -r -p 'Existing administrator username: ' CONDUCTOR_RECOVERY_ADMIN_USERNAME
read -r -s -p 'New administrator password (12+ characters): ' CONDUCTOR_RECOVERY_ADMIN_PASSWORD
printf '\n'
export CONDUCTOR_RECOVERY_ADMIN_USERNAME CONDUCTOR_RECOVERY_ADMIN_PASSWORD
export CONDUCTOR_ALLOW_ADMIN_RECOVERY=I_UNDERSTAND

docker compose -f docker-compose.yml exec -T \
  -e CONDUCTOR_ALLOW_ADMIN_RECOVERY \
  -e CONDUCTOR_RECOVERY_ADMIN_USERNAME \
  -e CONDUCTOR_RECOVERY_ADMIN_PASSWORD \
  conductor node dist/scripts/bootstrapAdmin.js

unset CONDUCTOR_ALLOW_ADMIN_RECOVERY CONDUCTOR_RECOVERY_ADMIN_USERNAME CONDUCTOR_RECOVERY_ADMIN_PASSWORD
exit
```

Replace `conductor.example.com` with the deployed FQDN. Browser setup and another enabled administrator remain the preferred paths.

## Upgrade

1. Read release notes and compatibility warnings.
2. Create and copy a verified backup off-host.
3. Change `CONDUCTOR_VERSION` or the pinned SkeletonWorks image.
4. Pull and recreate.
5. Wait for health and test login, project load, and a published deep link.

```bash
scripts/production/backup-conductor.sh
docker compose --env-file .env -f compose.production.yml pull
docker compose --env-file .env -f compose.production.yml up -d
curl --fail http://127.0.0.1:8080/api/health
```

SkeletonWorks upgrades are idempotent reruns of `setup-conductor.sh --image <new pinned ref>`.

## Rollback

Application images and database schema must be treated as a pair. For a failed compatible deployment, return to the prior pinned image. If the release changed the database incompatibly, restore the pre-upgrade archive as well.

1. Preserve logs and failed-state data.
2. Set the previous image tag/digest.
3. Restore the matching pre-upgrade backup.
4. Verify health and application smoke tests.

## Health and logs

- Liveness/ready endpoint: `GET /api/health`
- Container health: `docker compose ... ps`
- Logs: `docker compose ... logs --tail=200 conductor`

The health response includes the packaged version when built with release metadata.

## Recovery objectives

The default scripts retain seven daily archives, but retention is not a substitute for off-host storage. Recovery point and recovery time objectives are operator decisions. At least one periodic clean-host restore drill should be scheduled and recorded.

## Application icon storage (v1.1.0)

Conductor creates its app_icons table and adds a nullable icon_path column to publication metadata during normal database initialization. Repeating initialization is safe; older publications use the default icon. The canonical project-document schema remains 0.1.0.

Uploaded PNGs are decoded, re-encoded and stored in SQLite under a content hash. Requests are limited to 256 KiB, dimensions to square 16–512 pixels, and decompression is bounded. Matching images share one stored asset. Icons use immutable public URLs under /app-icons/; they are decorative assets rather than confidential documents.

Existing Conductor database backups cover both the icon choices and uploaded bytes. Keep normal database backup/restore procedures, persistent keys and matching application code. Installer-supplied static icons belong to that application's package and must be backed up with its code. Resetting an icon removes the publication's selection; it does not delete the shared image. Automatic removal of unused image assets is not implemented.

The v1.1.0 release adds independently installed apps. Fresh setup changes and the supported upgrade process are tracked in [SLICE12](../SLICE12.md) and [SLICE13](../SLICE13.md). Do not use a fresh setup run as an upgrade of the current combined Social Scheduler image.

## Installed application recovery in v1.1.0

The backup tools now write format conductor-backup-v2, including the exact application files referenced by the SQLite snapshot. The restore tool accepts v1 and v2. Old v1 backups have no independent app assets. Persistent keys decrypt both stored REST credentials and installed-app backend connections. Restore the matching app files as well as the database and keys; see [Application packages](APP_PACKAGES.md).