conductor/docs/OPERATIONS.md

Production Operations

Persistent state

All durable application state is stored in /data/conductor.db: projects, users, password hashes, sessions, publications, encrypted credentials, and execution history. SQLite runs in WAL mode.

The database alone is not a complete recovery set. A usable backup must include the exact CONDUCTOR_SECRET_KEY that encrypted stored credentials. Production backup archives therefore include the database, .env, Compose metadata, a manifest, and checksums. Treat archives as secrets and copy them off-host.

Production configuration

The public Compose installation reads .env. Keep it mode 0600, exclude it from source control, and retain it with the disaster-recovery set.

Variable	Purpose and operating rule
CONDUCTOR_SECRET_KEY	Required 32-byte encryption key encoded as 64 hexadecimal characters or base64. Generate once and preserve it with the database. Changing or losing it makes existing stored credentials unreadable.
CONDUCTOR_SESSION_KEY	Required value of at least 32 bytes used to protect sessions. Generate independently from the encryption key. Changing it signs everyone out.
CONDUCTOR_IMAGE	Public registry/repository name without a tag.
CONDUCTOR_VERSION	Pinned published image tag. Do not use a moving latest tag for a controlled installation.
CONDUCTOR_BIND_ADDRESS	Use 127.0.0.1 when the reverse proxy runs on the same host.
CONDUCTOR_PORT	Host loopback port forwarded to container port 8080.
CONDUCTOR_MEM_LIMIT, CONDUCTOR_CPUS	Container resource ceilings. Increase deliberately after measuring demand.
CONDUCTOR_PROXY_INTERNAL_ORIGINS	Optional comma-separated exact origins approved to reach private/internal networks. It accepts schemes, hostnames, and effective ports—not wildcards, paths, credentials, or IP literals.

Production refuses to start if either required key is absent or invalid. Generate them independently with openssl rand -hex 32; never paste their output into documentation, tickets, or screenshots.

After changing a non-secret configuration value, recreate the service and verify health:

docker compose --env-file .env -f compose.production.yml up -d
curl --fail http://127.0.0.1:8080/api/health

For a SkeletonWorks deployment, edit /opt/skeletonworks/conductor/<fqdn>/.env, recreate that stack, and check its HTTPS health endpoint:

sudoedit /opt/skeletonworks/conductor/conductor.example.com/.env
sudo docker compose \
  -f /opt/skeletonworks/conductor/conductor.example.com/docker-compose.yml \
  up -d
curl --fail https://conductor.example.com/api/health

The current SkeletonWorks setup script preserves the two keys in .secrets, but rewrites CONDUCTOR_PROXY_INTERNAL_ORIGINS to an empty value when it is rerun. Until that installer accepts and preserves an internal-origin setting, review and reapply the intended value after every setup or upgrade rerun before testing affected applications.

Do not casually rotate CONDUCTOR_SECRET_KEY; v1.0.0 has no online credential re-encryption procedure. Read Secrets and Proxy Security before changing key or internal-origin policy.

Backup

From a source/release directory:

scripts/production/backup-conductor.sh \
  --compose-file compose.production.yml \
  --env-file .env \
  --backup-dir ./backups \
  --retention-days 7

The script invokes the application-owned SQLite online-backup command inside the running container. It does not copy a live WAL database. It then creates a 0600 archive with:

• database/conductor.db
• configuration/.env
• configuration/compose.production.yml
• manifest.json
• SHA256SUMS

It writes progress to stderr and a machine-readable result to stdout and backup-result.json.

SkeletonWorks uses:

sudo /opt/skeletonworks/scripts/backup-conductor.sh \
  --fqdn conductor.example.com --retention-days 7

Its archive additionally contains .secrets and defaults to /opt/skeletonworks/backups/conductor/<fqdn>/.

Restore

Restore replaces live state. Verify that the chosen archive and the intended target match.

scripts/production/restore-conductor.sh --list --backup-dir ./backups
scripts/production/restore-conductor.sh \
  --backup-file ./backups/conductor_<timestamp>.tar.gz \
  --compose-file compose.production.yml \
  --env-file .env

Interactive restore requires typing RESTORE. Automation must explicitly pass --force. By default, restore creates a current-state safety backup, validates paths/checksums/format/keys, stops Conductor, replaces the database, restores UID/GID 10001 and mode 0600, restarts, and waits for health.

SkeletonWorks:

sudo /opt/skeletonworks/scripts/restore-conductor.sh --fqdn conductor.example.com --list
sudo /opt/skeletonworks/scripts/restore-conductor.sh \
  --fqdn conductor.example.com --backup-file <archive>

Never combine a database with a different encryption key. A checksum-valid but mismatched key cannot decrypt stored REST credentials.

Administrative bootstrap and recovery

Browser setup is the normal way to create the first administrator. The following fallback works only while the database contains no users. It invokes the compiled command inside the running production service:

read -r -p 'New administrator username: ' CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME
read -r -s -p 'New administrator password (12+ characters): ' CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD
printf '\n'
export CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD

docker compose --env-file .env -f compose.production.yml exec -T \
  -e CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME \
  -e CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD \
  conductor node dist/scripts/bootstrapAdmin.js

unset CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD

If every administrator is inaccessible, recover one existing local administrator. Recovery cannot create an account or elevate a user. It re-enables the named administrator, replaces the password hash, and revokes that account's sessions:

read -r -p 'Existing administrator username: ' CONDUCTOR_RECOVERY_ADMIN_USERNAME
read -r -s -p 'New administrator password (12+ characters): ' CONDUCTOR_RECOVERY_ADMIN_PASSWORD
printf '\n'
export CONDUCTOR_RECOVERY_ADMIN_USERNAME CONDUCTOR_RECOVERY_ADMIN_PASSWORD
export CONDUCTOR_ALLOW_ADMIN_RECOVERY=I_UNDERSTAND

docker compose --env-file .env -f compose.production.yml exec -T \
  -e CONDUCTOR_ALLOW_ADMIN_RECOVERY \
  -e CONDUCTOR_RECOVERY_ADMIN_USERNAME \
  -e CONDUCTOR_RECOVERY_ADMIN_PASSWORD \
  conductor node dist/scripts/bootstrapAdmin.js

unset CONDUCTOR_ALLOW_ADMIN_RECOVERY CONDUCTOR_RECOVERY_ADMIN_USERNAME CONDUCTOR_RECOVERY_ADMIN_PASSWORD

Run recovery from a private operator terminal, confirm the target carefully, and clear any terminal capture governed by local policy. Create and retain a second enabled administrator so routine recovery can remain in the browser.

For SkeletonWorks, run the corresponding command from the stack directory in a root shell so the password variables are available to Compose. For an empty-installation bootstrap:

sudo -i
cd /opt/skeletonworks/conductor/conductor.example.com

read -r -p 'New administrator username: ' CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME
read -r -s -p 'New administrator password (12+ characters): ' CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD
printf '\n'
export CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD

docker compose -f docker-compose.yml exec -T \
  -e CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME \
  -e CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD \
  conductor node dist/scripts/bootstrapAdmin.js

unset CONDUCTOR_BOOTSTRAP_ADMIN_USERNAME CONDUCTOR_BOOTSTRAP_ADMIN_PASSWORD
exit

For guarded recovery of an existing administrator:

sudo -i
cd /opt/skeletonworks/conductor/conductor.example.com

read -r -p 'Existing administrator username: ' CONDUCTOR_RECOVERY_ADMIN_USERNAME
read -r -s -p 'New administrator password (12+ characters): ' CONDUCTOR_RECOVERY_ADMIN_PASSWORD
printf '\n'
export CONDUCTOR_RECOVERY_ADMIN_USERNAME CONDUCTOR_RECOVERY_ADMIN_PASSWORD
export CONDUCTOR_ALLOW_ADMIN_RECOVERY=I_UNDERSTAND

docker compose -f docker-compose.yml exec -T \
  -e CONDUCTOR_ALLOW_ADMIN_RECOVERY \
  -e CONDUCTOR_RECOVERY_ADMIN_USERNAME \
  -e CONDUCTOR_RECOVERY_ADMIN_PASSWORD \
  conductor node dist/scripts/bootstrapAdmin.js

unset CONDUCTOR_ALLOW_ADMIN_RECOVERY CONDUCTOR_RECOVERY_ADMIN_USERNAME CONDUCTOR_RECOVERY_ADMIN_PASSWORD
exit

Replace conductor.example.com with the deployed FQDN. Browser setup and another enabled administrator remain the preferred paths.

Upgrade

1. Read release notes and compatibility warnings.
2. Create and copy a verified backup off-host.
3. Change CONDUCTOR_VERSION or the pinned SkeletonWorks image.
4. Pull and recreate.
5. Wait for health and test login, project load, and a published deep link.

scripts/production/backup-conductor.sh
docker compose --env-file .env -f compose.production.yml pull
docker compose --env-file .env -f compose.production.yml up -d
curl --fail http://127.0.0.1:8080/api/health

SkeletonWorks upgrades are idempotent reruns of setup-conductor.sh --image <new pinned ref>.

Rollback

Application images and database schema must be treated as a pair. For a failed compatible deployment, return to the prior pinned image. If the release changed the database incompatibly, restore the pre-upgrade archive as well.

1. Preserve logs and failed-state data.
2. Set the previous image tag/digest.
3. Restore the matching pre-upgrade backup.
4. Verify health and application smoke tests.

Health and logs

• Liveness/ready endpoint: GET /api/health
• Container health: docker compose ... ps
• Logs: docker compose ... logs --tail=200 conductor

The health response includes the packaged version when built with release metadata.

Recovery objectives

The default scripts retain seven daily archives, but retention is not a substitute for off-host storage. Recovery point and recovery time objectives are operator decisions. At least one periodic clean-host restore drill should be scheduled and recorded.