conductor/docs/CONDUCTOR_UPGRADE.md

Conductor upgrade and recovery

This guide is for the VM administrator. Use update-conductor.sh to upgrade an existing Conductor installation. Use setup-conductor.sh for a fresh installation.

The upgrade bundle is currently a local deliverable, not a newly published Conductor release. It installs the already published Conductor v1.1.0 image at its verified digest. It includes the matching backup/restore scripts and the published Social Scheduler 0.5.0 browser package needed to migrate the older combined installation.

What happens

1. The script checks the current installation, target image, app compatibility and available disk space.
2. It saves the previous Conductor image and private configuration, then briefly stops Conductor to copy its complete data volume.
3. It starts an isolated test container using a new copy of that data. This container has no network access to Node-RED or the Internet. The script checks startup, saved-record preservation, encrypted secrets, installed assets and representative routes.
4. After those checks pass, it switches Conductor to the new image and copied data, checks the running service, updates its setup record, and installs the matching routine backup/restore scripts.
5. It retains the previous image archive, database, app files, configuration, keys and original volume for recovery.

Conductor is unavailable during the copy/check/switch portion. Image downloading and the image backup happen before that stop. Allow a maintenance window; duration depends on image download speed and the size of the data.

Node-RED is not restarted. Its social credentials, API tokens, enabled/paused setting, PostgreSQL schedules, media and posting activity are not changed. Caddy and the Conductor address stay as they are. No host Node.js installation is needed.

Supported starting points

Starting installation	Behavior
Standard Conductor v1.0.0	Preserves accounts, passwords, projects, publications and encrypted secrets; enables the v1.1.0 app host.
Conductor with Social Scheduler 0.4.0 built into its image	Migrates that browser app into persistent package storage, using the existing internal Node-RED connection settings.
Conductor v1.1.0 with independent apps	Preserves the packages and encrypted connections. The same image and app state produce an already-current result.
An unknown legacy app, unsupported package/API/version or custom persistent mount	Stops for a compatibility review before changing the live installation.

This version of the upgrader targets Conductor v1.1.0 / application host API 1. It does not promise compatibility with future Conductor versions that have not been tested. A subsequent release can update the same reusable upgrade process.

The standard installation has one Conductor service, port 8080, and a named volume mounted at /data. It retains its existing resource limits, network, restart policy, security settings and environment files. Compose is saved as valid JSON after the switch; Docker Compose supports that format. The physical data volume changes to the verified copy. Future tools should discover that volume from Docker or setup-result.json rather than assuming its old name.

Upgrade the existing VM

Transfer conductor-upgrade-1.1.0.tar.gz and its .sha256 file from the local release folder to the VM. The app files are inside this bundle; separate Node-RED .tgz packages are not needed for this Conductor upgrade.

Verify and extract it:

sha256sum -c conductor-upgrade-1.1.0.tar.gz.sha256
tar -xzf conductor-upgrade-1.1.0.tar.gz
cd conductor-upgrade
sha256sum -c SHA256SUMS

These checks confirm that the transferred kit and its contents are intact.

Use the full hostname already assigned by setup-conductor. For the current Skeleton Works instance:

sudo bash ./update-conductor.sh --fqdn conductor.skeletonworks.online --dry-run

The dry run reads protected setup records and the running installation. It does not pull an image, create a backup or lock, change keys, install an app, stop a service or write customer records. It lists legacy apps requiring migration and independent apps to preserve. Target image compatibility is checked during execution before Conductor is stopped.

Run the upgrade:

sudo bash ./update-conductor.sh --fqdn conductor.skeletonworks.online

Keep the backup directory printed in the successful result. Do not run the full Social Scheduler installer as an extra migration step. The app package and its connection migration are already included in this command.

The updater installs the matching backup-conductor.sh and restore-conductor.sh into /opt/skeletonworks/scripts. Existing cron entries continue to use those paths. The fresh setup script is also in the kit for provisioning other VMs.

Check the result

Action	What it proves
Sign in at the existing Conductor URL using the existing username/password.	The account, password and authentication configuration survived.
Open Projects and an existing saved project.	Saved customer work remains available.
Open Publishing and an existing published application.	Publication snapshots and access settings survived.
Open Social Scheduler from Included with Skeleton Works.	The app was migrated and registered once. Its title is Social Scheduler · Conductor and it inherits the default favicon.
Inspect Social connections and the existing post list.	The app can still reach its existing Node-RED connection and schedules. No new credentials should be required.
Rerun the same upgrade command.	An already-current result confirms that repetition does not reinstall apps, duplicate entries or regenerate keys.

These are read-only acceptance checks. A new real social post is not needed to validate the Conductor upgrade.

Interrupted or failed upgrades

An ordinary failure after the snapshot triggers automatic recovery using the untouched original data volume and previous image/configuration. Read the final message to confirm whether recovery succeeded.

After a power loss, forced interruption or an incomplete automatic recovery, rerun the same command. The journal causes the script to recover the previous installation before attempting another upgrade. It reports recovered when that recovery finishes; run the command once more when ready to retry.

Do not delete .upgrade-state.json or edit its phase to bypass recovery. Routine backup and restore refuse to run while an upgrade needs recovery. A maintenance lock prevents concurrent upgrade/backup/restore operations. Avoid changing Conductor settings or installing apps from another terminal during maintenance.

For a compatibility refusal, the live installation stays in place. Missing legacy package instructions mean the whole upgrade bundle must be present, or the relevant app package must be supplied with --app-package /path/to/package. Do not run fresh setup to work around that refusal.

For a disk-space failure, free space and retry. Keep recovery data until the new version has been accepted; Docker cleanup commands that remove unused images/volumes can remove the quick recovery copy.

Explicit recovery to an earlier image and data

Use the exact backup directory printed by the successful upgrade, for example:

sudo bash ./update-conductor.sh \
  --fqdn conductor.skeletonworks.online \
  --rollback /opt/skeletonworks/conductor/conductor.skeletonworks.online/.upgrades/REPLACE_WITH_BACKUP_ID \
  --dry-run

The directory name above is a placeholder. Copy the actual path from the earlier result.

Review the recovery plan, then run:

sudo bash ./update-conductor.sh \
  --fqdn conductor.skeletonworks.online \
  --rollback /opt/skeletonworks/conductor/conductor.skeletonworks.online/.upgrades/REPLACE_WITH_BACKUP_ID \
  --accept-data-rewind

Explicit rollback returns Conductor data to the backup time. Conductor projects, publications, users or app changes made afterward are not in that restored copy. The flag acknowledges this. The script first snapshots the current installation so that those newer records remain available for a separate recovery.

Node-RED and PostgreSQL are not rolled back: a social post already sent stays sent, and current backend schedules and credentials remain current.

Recovery validates archive checksums and paths before stopping the current service. It can reload the saved old image even if its local tag/image and original volume were removed. It restores into a new volume, with the corresponding old configuration and maintenance scripts. This also works when the active Conductor service is stopped. Repeating a completed rollback returns already-restored.

An automatic recovery after a failed upgrade uses the untouched original volume. If that volume or image was manually removed during the incomplete attempt, retain all backups and inspect the reported recovery issue; do not bypass the journal.

What to retain and protect

Upgrade snapshots are in:

/opt/skeletonworks/conductor/FULL_HOSTNAME/.upgrades/BACKUP_ID/

Each completed snapshot contains:

• image.tar: the exact previous runtime image, including any old built-in app.
• data.tar: consistent SQLite data and every file in the stopped Conductor volume.
• configuration/: original setup records, environment, encryption/session keys and Compose.
• maintenance/: the previous backup/restore scripts.
• original.json and manifest.json: configuration identity and file checksums.

The directory and files are private. Treat a copied recovery directory as containing customer data and credentials. Keep it in the administrator's protected backup storage. Upgrade snapshots are not automatically deleted by routine backup retention.

The script leaves unused original/failed candidate volumes in place for recovery or investigation. Remove them only after acceptance and after identifying the exact volume names in the upgrade journal/backup metadata. Do not remove the active volume recorded in setup-result.json.

Routine backup-conductor.sh / restore-conductor.sh remain appropriate for data recovery on a compatible runtime. Use update-conductor.sh --rollback when the runtime image must also be returned to its prior version.

Modular legacy app migration

Base Conductor and its fresh setup do not register Social Scheduler. The upgrade bundle contains app-owned packages under conductor-app-packages/. Only an app already recorded in the old installation is migrated. Uninstalled apps in that directory are never added automatically.

A future legacy integration can supply a verified conductor-app.json browser package plus a separate conductor-migration.json recipe:

{
  "migrationVersion": 1,
  "applicationId": "example-app",
  "fromCatalogVersions": ["0.4.0"],
  "connection": {
    "origin": {"environment": "EXAMPLE_API_ORIGIN"},
    "token": {"environment": "EXAMPLE_INTERNAL_TOKEN"},
    "values": {
      "workspaceId": {"environment": "EXAMPLE_WORKSPACE_ID", "default": "customer"}
    }
  }
}

The recipe names existing environment settings; it contains no credentials or executable migration code. The upgrader copies their values into the app host's encrypted connection record. Packages without an API can omit connection. Independent apps already installed through host API 1 need no legacy recipe and are not updated by a core upgrade.

For development/rebuilding and test evidence, see TESTING.md and the SLICE13 log.