vwiebe/conductor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
conductor/CODEX.md

27 KiB
Raw Blame History

Codex Project State

This file is the durable handoff record for Codex sessions working on Conductor. Read it before making changes, verify any drift against the current code, and update it at the end of each material work session.

Project

Conductor is a web-based, low-code builder for creating simple user interfaces backed by REST API endpoints. Its initial use case is IBM Concert Workflows / Rapid Infrastructure Automation, but the application must remain backend-agnostic.

Repository:

  • Remote: https://gitea.skeletonworks.online/vwiebe/conductor.git
  • Primary branch: main
  • Workspace: /home/vwiebe/projects/conductor
  • Initial fork commit: cce97392f72049007cf00b0aa6c77fb8c8256191

Sources of Truth

Use these files in this order:

  1. Current source code and automated tests
  2. ROADMAP.md for completed and remaining work
  3. TESTING.md for pending and accepted manual workflows
  4. docs/response-mapping-model.md for response-binding decisions
  5. shared/schemas/conductor-project.schema.json for the accepted project-document shape
  6. docs/REQUIREMENTS.md for product scope
  7. docs/ARCHITECTURE.md for architectural direction
  8. docs/BUILD_AND_TEST_PLAN.md for historical milestone and validation guidance

The docs/ copies of requirements, architecture, and future-idea specifications are authoritative. Exact duplicate root copies were removed during the roadmap consolidation.

Core Architectural Decisions

  • The canonical source of truth is one versioned JSON project document.
  • The Visual Editor, JSON Editor, Preview runtime, persistence layer, validation, and future AI tooling must operate on the same canonical document.
  • Runtime state must remain separate from the canonical document.
  • Projects are persisted in SQLite for the MVP.
  • External REST requests must flow through the backend proxy.
  • Secrets must never be intentionally exposed to the browser or stored in exported project JSON.
  • Top-level project.bindings is the canonical data-flow mechanism.
  • action.responseMapping is deprecated and must not be used for new behavior.
  • New action-response bindings should use onSuccess; legacy onClick response bindings remain compatible.
  • Production deployment should use an external reverse proxy.
  • Conductor must remain functional without AI services.

Current Implementation State

Foundation

  • React and TypeScript frontend
  • Node.js, Express, and TypeScript backend
  • SQLite database initialization and project persistence
  • Project CRUD API
  • Health and validation endpoints
  • Dockerfiles and Docker Compose configuration
  • Application layout, navigation, and shared project context
  • Project creation, save, update, and load controls

Project model and editing

  • Canonical project JSON model at schema version 0.1.0
  • Shared JSON schema and frontend TypeScript types
  • Backend schema validation
  • Valid and intentionally invalid example project definitions
  • Visual Editor and JSON Editor operating on shared project state
  • JSON syntax and schema validation before JSON Editor changes are applied

Active GUI components

All eleven schema-supported MVP components are available in the palette and have canvas, property-editor, and Preview support:

  • Label
  • Button
  • Text Input
  • Dropdown
  • Table
  • Text Area
  • Checkbox
  • Radio Group
  • Status/Message Panel
  • Container/Card (flat presentational title and body; nesting deferred)
  • JSON Viewer

REST and Preview runtime

  • REST action model supporting GET, POST, PUT, PATCH, and DELETE
  • Visual anonymous REST action creation, editing, duplication, and reference-aware deletion in Actions & Bindings
  • Progressive request controls for endpoint URL, headers, query parameters, static path parameters, and request bodies
  • Valid action edits synchronize immediately to the canonical document; invalid form drafts stay local
  • Anonymous server-side REST execution
  • Request URL, header, query, path, and body templates
  • Normalized proxy response envelope
  • Button-triggered action execution in Preview
  • Runtime component, action, and variable state
  • Response mappings to Label and JSON Viewer values
  • Response mappings to Dropdown options
  • Response mappings to Table rows
  • Response mappings to project variables
  • Table row selection
  • Binding and action diagnostics
  • Frontend unit tests covering templates, bindings, variables, dropdowns, and tables

Known Gaps and Risks

  • Basic, Bearer, and API-key authentication types are modeled but currently execute anonymously.
  • Secure credential storage and secret resolution are not implemented.
  • Slice 4 proxy enforcement and sanitized execution history are complete in the working tree. All proportional manual workflows and the independent rebuilt-Docker/isolated-Playwright re-test passed. The user granted explicit final sign-off on 2026-08-03 with the exact wording: “I grant explicit final Slice 4 sign-off.”
  • Visual REST action, Button event, request-input, response-binding, variable, and page onLoad authoring are implemented. Page-load manual acceptance passed on 2026-07-29.
  • Project documents use one backend schema/semantic validator across JSON Apply, create, update, save, stored-project load checks, and shared frontend diagnostics. Invalid documents never reach a database write.
  • Slice 5 validation and error handling passed consolidated automation, proportional manual acceptance, cleanup, and explicit final sign-off on 2026-08-04.
  • Backend and end-to-end automated test coverage remains incomplete.
  • IBM Bob/watsonx will not be used; AI is provider-neutral and post-MVP.
  • All five modeled authentication modes are required for MVP.
  • MVP completion is defined by the six workflows in MVP_SCOPE.md.
  • Root-level requirement and architecture documents duplicate files under docs/ and may drift.

Validation Status

The initial repository push was completed on 2026-07-18.

  • Remote main contains commit cce97392f72049007cf00b0aa6c77fb8c8256191.
  • The working tree was clean immediately after the push.
  • All :Zone.Identifier sidecar files were removed before the initial commit.
  • Node 20/npm 10 clean installs pass after repairing the frontend lockfile.
  • Slice 2 Increment 1 frontend validation passes under Node 20/npm 10: 9 suites and 456 tests, plus the production build.
  • Backend build and the schema fixture matrix pass: 13 valid fixtures, 2 expected schema failures, and 2 diagnostic-invalid fixtures that remain structurally valid.
  • Standalone frontend tsc --noEmit is blocked by the existing TypeScript 4.9 / @types/node@26.1.0 dependency mismatch; the CRA TypeScript production compile passes.
  • In-app browser automation could not start during implementation while Docker Desktop was not running. After Docker Desktop and the app were started, the user manually accepted Slice 2 Increment 1 on 2026-07-19, including a Button onClick event inserted through canonical JSON and reference-aware action deletion.
  • Docker Compose builds and starts both services.
  • Health, CRUD, cleanup, and restart persistence pass.
  • See BASELINE.md for exact evidence and remaining release work.
  • Manual Slice 1 acceptance passes for component configuration, Preview interaction, JSON synchronization, save/load persistence, and existing-component regressions.

Immediate Priorities

Unless the user chooses a different priority, proceed in this order:

  1. Continue Slice 3 with canonical secret references, action authoring, credential injection, compatibility validation, and redaction tests.
  2. Add validation to every project save path.
  3. Continue release-critical backend, end-to-end, and security regression coverage from Slice 6.
  4. Add backend, end-to-end, and security regression tests.
  5. Reconcile and consolidate project documentation.

See ROADMAP.md for the complete actionable checklist.

Working Conventions

  • Keep the canonical schema, frontend types, backend validation, examples, editor, and runtime synchronized.
  • Any project-model change must include schema validation of valid and invalid examples.
  • Prefer small, focused changes with tests over broad rewrites.
  • Do not execute deprecated action.responseMapping rules.
  • Do not store runtime values in canonical project JSON.
  • Do not commit secrets, environment files, SQLite databases, dependency directories, or build artifacts.
  • Preserve backend-agnostic REST behavior; do not couple core runtime logic to IBM-specific services.
  • Clearly distinguish implemented behavior from schema-only or planned behavior.
  • Update ROADMAP.md and the owning slice when a task is completed, added, removed, or materially re-scoped.

Session Start Checklist

At the beginning of a new session:

  • Read CODEX.md, ROADMAP.md, and the active slice plan.
  • Run git status --short --branch.
  • Inspect recent commits with git log --oneline -5.
  • Verify relevant source files before relying on this handoff; this document may lag behind code.
  • Confirm the selected task and its acceptance criteria.
  • Run focused baseline tests when dependencies are available.

Session End Checklist

After material work:

  • Update completed and remaining items in ROADMAP.md and the active slice plan.
  • Update the implementation state, gaps, validation status, and next priority in CODEX.md.
  • Record the exact commands run and whether they passed when validation results materially changed.
  • Ensure documentation matches any schema or runtime decisions made during the session.
  • Report modified files, tests, builds, and unresolved blockers to the user.

Latest Handoff

Date: 2026-07-20

  • Completed Slice 2 Increment 1: visual anonymous REST action creation, editing, duplication, and reference-aware deletion.
  • Added method, URL, header, query, static path-parameter, and body controls with immediate canonical-document synchronization.
  • Kept invalid drafts and all execution state outside canonical JSON; new and duplicated actions are anonymous and omit deprecated action.responseMapping.
  • Added focused validation for action IDs, URLs, duplicate IDs, header names, unsupported path-parameter templates, and referenced deletion.
  • Added valid-visual-rest-actions.json and 23 focused tests.
  • Validation passes under Node 20/npm 10: 9 frontend suites / 456 tests, frontend production build, backend TypeScript build, and the 13 / 2 / 2 schema matrix.
  • The user manually accepted the Increment 1 REST action authoring and referenced-action deletion workflow after starting Docker Desktop and the app.
  • Standalone frontend tsc --noEmit remains blocked by the existing dependency mismatch. Browser automation remained unavailable; user manual acceptance passed.
  • Slice 2 REST action and Button event configuration was committed and pushed as 3994eaa.

Increment 2 Addendum

  • Completed visual Button onClick action selection with no-action, add/change/clear, unrelated-event preservation, and missing-action diagnostics.
  • Canonical component.events updates immediately; automated coverage confirms Preview executes the selected REST action.
  • Reconciled request inputs: ComponentEvent.inputMap is compatibility-only and is not consumed. Request-input UI must write executed REST request templates.
  • Updated schema/type documentation and the representative visual REST fixture.
  • Validation passes: 12 frontend suites / 462 tests, frontend production build, backend build, and the 13 / 2 / 2 schema matrix.
  • Standalone frontend tsc --noEmit retains the recorded dependency failure.
  • Automated browser validation was unavailable. The user manually accepted the Increment 2 Button onClick workflow on 2026-07-19 by creating a REST action and assigning it visually to a Button.
  • Visual request-input authoring and its handoff were committed and pushed as 6fd084c.

Request Input Addendum

  • Added guided component/variable reference insertion to REST action URL, header, query, and body template fields.
  • The UI writes only executed {{components.<name>.value}} and {{variables.<name>}} templates; path parameters and ComponentEvent.inputMap remain excluded.
  • Header/query destinations replace the selected value; URL/body destinations append for further composition in the existing raw editor.
  • Added pure canonical-update and UI interaction coverage.
  • Validation passes: 14 frontend suites / 467 tests, frontend production build, backend build, and the 13 / 2 / 2 schema matrix.
  • Standalone frontend tsc --noEmit retains the recorded dependency failure.
  • Manual request-input acceptance passed on 2026-07-28, including executed component and variable references, canonical/runtime-state separation, destination filtering, and raw-template regression coverage.
  • The request-input continuation checklist was completed; durable acceptance evidence remains in TESTING.md and BASELINE.md.

Response Binding and Variable Addendum

  • Added visual create, edit, and delete support for canonical top-level project.bindings.
  • New action-response bindings select an existing REST action and unambiguous supported component or variable target and always default to trigger: "onSuccess".
  • Existing legacy onClick bindings remain editable and executable and can migrate one-way to onSuccess; the UI cannot create new legacy triggers. Deprecated action.responseMapping remains unused.
  • Added visual variable declaration, renaming, type/default-value/description editing, deletion, validation, and reference-aware deletion warnings. Preview continues to copy defaults into ephemeral runtime state.
  • Updated valid-visual-rest-actions.json with representative onSuccess component and variable targets.
  • Focused tests pass under Node 20/npm 10 at 2 suites / 14 tests; the full frontend suite passes at 15 suites / 477 tests. The CRA production compile, backend build, and 13 / 2 / 2 schema matrix pass.
  • Standalone frontend tsc --noEmit still fails in @types/node/ffi.d.ts because TypeScript 4.9 cannot parse the resolved @types/node@26.1.0; this remains separate from the passing CRA compile.
  • Docker Compose services and health checks passed on 2026-07-28. Request-input manual acceptance passed; response-binding/variable manual acceptance remains pending.

Boolean Variable Default Fix

  • On 2026-07-29, manual typed-variable acceptance found that entering true was reset to false before it could be completed.

  • Replaced the free-form boolean default editor with an explicit No default, true, and false selector.

  • Extended the Actions and Bindings interaction test to assert a canonical boolean true default.

  • Focused validation passed at 2 suites / 14 tests; the full frontend suite passed at 15 suites / 477 tests; the production build passed.

  • The user manually confirmed boolean true authoring after the rebuilt frontend was deployed. Typed declarations and invalid-draft isolation are accepted; the remaining response-binding and variable workflow is pending.

  • Component response-binding manual acceptance passed on 2026-07-29: a successful HTTP 200 response body populated the JSON Viewer through a canonical onSuccess binding. Intermittent httpbin.org 503 responses correctly did not apply the binding; httpbingo.org was used for the successful checkpoint. Variable response-binding manual acceptance also passed: a first response updated runtime statusText, a second request consumed that runtime value, and the canonical Not run default remained unchanged. Binding edit and selected deletion also passed, preserving the unrelated binding and re-enabling Add response binding. A deferred Actions and Bindings information-architecture task was added after the user reported that records are difficult to distinguish and scan. Unsupported TextInput targets and ambiguous duplicate-name targets were manually confirmed absent from new-binding choices. Referenced-variable warning, cancellation, accepted deletion, and unresolved-reference diagnostics also passed without changing unrelated records. Legacy onClick execution, diagnostics, and one-way migration to onSuccess also passed; neither migrated nor new records offered onClick afterward. Manual failure-state testing exposed a clipped Button error; the alert is now positioned below the fixed-height Button with wrapping, and stale errors clear on retry or success. Focused Preview tests passed at 2 suites/7 tests, the full frontend suite at 15 suites/478 tests, and the production build passed. The user confirmed readable 503 handling, onSuccess suppression, clean canonical state, and stale-error clearing. Save/reload persistence and runtime reset passed, completing response-binding and variable manual acceptance on 2026-07-29.

  • Page-load action authoring is implemented in the working tree and proportional manual acceptance passed on 2026-07-29.

Partial Manual Request-Input Checkpoint

  • On 2026-07-20, the user confirmed that adding the item query-parameter row and using Request value reference to insert the selected component into Query: item looked correct.
  • On 2026-07-28, the user completed and accepted the remaining Preview execution, outgoing-request inspection, runtime-state separation, raw-template regression, path-destination exclusion, and variable-resolution checks. The complete request-input workflow is accepted.

Page-Load Action Addendum

  • Implemented visual initial-page onLoad action selection and clearing in the Visual Editor, with immediate canonical page.events synchronization, unrelated-event preservation, duplicate normalization, and missing-action diagnostics.
  • Preview executes the initial page's first onLoad action once per mounted initialization and executes once again after leaving and re-entering Preview.
  • The runtime reuses request-template rendering, anonymous proxy execution, normalized responses, and top-level response bindings for supported component and variable onSuccess targets.
  • Page-load status, loading, errors, responses, component values, and runtime variable values remain ephemeral. No ComponentEvent.inputMap, deprecated action.responseMapping, authentication, secrets, or general orchestration was added.
  • Updated valid-visual-rest-actions.json with representative page-load configuration. No schema, frontend model, or backend changes were required.
  • Focused tests pass at 4 suites / 9 tests; the full frontend suite passes at 18 suites / 486 tests under Node 20/npm 10. The frontend production build and 13 valid / 2 expected-invalid / 2 diagnostic-invalid schema matrix pass.
  • Docker Compose rebuilt successfully; both services started, backend health passed, and the frontend responded on port 3000. Proportional manual page-load acceptance passed on 2026-07-29. The accepted Table workflow exposed a non-blocking Preview background-sizing limitation; ROADMAP.md tracks content-aware Preview sizing and post-MVP Table pagination. No commit or push was made.

Component Deletion Safety Addendum

  • Added reference-aware deletion warnings for components used by canonical response-binding source or target paths and by action URL, header, query, path, or body templates.
  • The warning names the affected binding or action and requires Cancel or Delete anyway. Confirmed deletion removes only the component so existing diagnostics can expose the dangling reference.
  • Unreferenced components retain immediate deletion.
  • Focused validation passed at 4 suites / 12 tests, the full frontend suite at 20 suites / 490 tests, and the production build passed.
  • The user manually confirmed every component-deletion safety retest passed on 2026-07-30. No commit or push was made.

Final Slice 2 Workflow Acceptance Addendum

  • On 2026-07-30, the user manually accepted the complete workflow-launcher and dependent-data scenarios.
  • The visually authored launcher used canonical component request templates, Button onClick, and an onSuccess JSON Viewer binding; success, readable HTTP 503 failure, onSuccess suppression, retry recovery, save/load, and fresh-Preview runtime reset passed.
  • The dependent-data workflow used page onLoad to populate Dropdown.options, passed populated, empty, failure, and recovery states, and supplied the selected runtime value to the later launcher request.
  • Canonical page events, component events, templates, and top-level bindings persisted. Runtime inputs, loaded options, loading, responses, errors, secrets, inputMap, and action.responseMapping did not.
  • Manual acceptance is distinct from automated validation. Slice 2 remains open for diagnostics confirmation and the other hardening/presentation tasks listed in ROADMAP.md.

Page-Load Diagnostics Correction Addendum

  • Actions & Bindings now includes page events when computing triggered action IDs, so a valid page onLoad action suppresses the false untriggered-action information message and false response-binding warning.
  • Diagnostic remediation now refers to supported component events or page onLoad rather than prescribing Button JSON.
  • Focused ActionInspector validation passed at 1 suite / 11 tests; the complete frontend suite passed at 20 suites / 493 tests with 0 snapshots; the production build passed.
  • Proportional manual confirmation passed on 2026-07-30: after a hard refresh loaded the rebuilt frontend, neither the page onLoad action nor its response binding showed the false untriggered diagnostic.

Runtime-Aware Preview Canvas Sizing Addendum

  • Preview observes rendered component-wrapper dimensions and grows the white canvas when runtime content exceeds configured design-time height.
  • The measurement layer is runtime-only; canonical component positions and sizes are unchanged.
  • Recalculation starts from design bounds so the page can contract after runtime content becomes smaller or empty.
  • Focused Preview validation passed at 1 suite / 9 tests; the complete frontend suite passed at 20 suites / 496 tests with 0 snapshots; the production build passed.

MVP Component Properties and Basic Appearance Addendum

  • Added visual JSON Viewer default JSON, Table column width, and Dropdown required controls; existing Text Input/Text Area required controls now affect Preview semantics.
  • Added canonical properties.style with font size, text color, and background color, applied in both canvas and Preview while preserving defaults when absent.
  • The shared schema validates the style shape and bounds; no theme system or post-MVP styling features were added.
  • Focused Preview property coverage passed at 1 suite / 13 tests; the complete frontend suite passed at 20 suites / 500 tests; frontend and backend builds passed; the 13 / 2 / 2 schema matrix passed.
  • Proportional manual authoring and persistence confirmation passed on 2026-07-31 after Docker rebuild and hard refresh. Passing automation and manual acceptance remain separately reported.
  • Proportional manual Table layout confirmation passed on 2026-07-30: populated rows remained inside the expanded white page, scrolling remained usable, an empty result contracted the page, and canonical Table size remained unchanged.

2026-07-31 Slice 2 Presentation Addendum

  • The user manually accepted all five MVP component-property and basic-appearance sections. Configuration authoring, canvas/Preview agreement, canonical synchronization, reset behavior, save/reload persistence, and fresh-Preview runtime reset passed.
  • Actions, response bindings, and variables now render as distinct summarized records with clearer section descriptions, hierarchy, spacing, editor attachment, and variable/target summaries. Focused ActionInspector validation passes at 1 suite / 12 tests; proportional manual acceptance passed on 2026-07-31.
  • v0.1.0 retains the existing fixed 30-second backend proxy timeout. No canonical per-action timeout field or editor control was added; bounded configurability is deferred to proxy-policy work.
  • The user manually accepted the Actions & Bindings presentation on 2026-07-31, including record separation, editor attachment, canonical edit isolation, resolved variable-target labeling, and diagnostic recovery.
  • The user explicitly granted final Slice 2 sign-off on 2026-07-31. Slice 2 is complete, and Slice 2a is authorized as a separate request-reference usability increment.

Slice 2a Request-Reference Usability Addendum

  • Contextual component and variable suggestions now appear in URL, header-value, query-value, and body-template controls after an unfinished {{ template opener.
  • Suggestions insert canonical executed templates while preserving free-form prefixes; ambiguous duplicate component names are excluded.
  • Header-name inputs suggest common names including Accept and Content-Type while retaining custom-name entry.
  • Guided Request value reference controls now present request destination first and value source second for key/value alignment.
  • Focused coverage passes at 4 suites / 23 tests; the complete frontend passes at 21 suites / 507 tests; the production build passes.
  • Proportional manual acceptance passed on 2026-08-01 in isolated Chromium against project Slice 2 Final (#12) at 1440 x 1100 and 700 x 1000. Contextual suggestions, duplicate-name exclusion and recovery, common/custom headers, guided-control order and responsive alignment, canonical insertion, replace/append behavior, path exclusion, continued editing, and canonical/runtime separation all passed. Saved project #12 was not updated. This manual evidence is separate from automation. The user granted explicit final Slice 2a sign-off on 2026-08-01; Slice 2a is complete.

Slice 3 Encrypted Secret Lifecycle Addendum

  • The user approved AES-256-GCM credential encryption in SQLite with a server-only 32-byte master key and opaque canonical references.
  • The backend now has encrypted secret persistence, metadata-only lifecycle APIs, internal-only credential resolution, and fail-closed missing/malformed-key handling. The threat and lifecycle contract is in docs/SECRETS.md.
  • Backend TypeScript build passed. A controlled Docker lifecycle check returned no credential value, found no plaintext token in SQLite, survived restart, and passed deletion/404 cleanup. The disposable test record and active disposable key were removed.
  • Canonical secretReferenceId, protected-mode frontend authoring, runtime injection for all four credential-backed modes, structured compatibility errors, and reflected-response credential redaction are implemented. Controlled echo checks passed for every mode; full frontend coverage remains 21 suites / 507 tests, focused authoring coverage passes at 2 suites / 32 tests, and frontend/backend builds pass.
  • Guided secret lifecycle UI, compatible action selection, reference-aware deletion, and durable encryption tests are implemented. Backend tests/build, full frontend coverage at 21 suites / 507 tests, focused authoring at 2 suites / 32 tests, and the frontend production build pass.
  • Durable injection/failure/redaction/URL-sanitization tests, persistent-key Docker restart execution, consolidated 22-suite / 509-test frontend validation, backend tests/build, frontend production build, and the 13 / 2 / 2 schema matrix pass.
  • Proportional Slice 3 manual acceptance passed on 2026-08-01 using an unsaved browser copy of project #12. Masked lifecycle, metadata-only records, compatible selection, opaque canonical references, protected execution/redaction, deletion safety, cleanup, and missing-reference failure behavior passed. The final secret list was empty and the saved project was unchanged. A non-blocking stale blocked-deletion warning is tracked in ROADMAP.md. The user granted explicit final Slice 3 sign-off on 2026-08-02; Slice 3 is complete.