Controller leases and live guest reconciliation (#854/#855)

Controller leases and live guest reconciliation (#854/#855)

Purpose

Qualify durable controller ownership, worker lifetime independent of the broker, and actual guest account/RDP/OpenBao reconciliation. Results are scoped to these components; lifecycle configuration and limits remain authoritative for rollout.

System topology

A disposable Debian 12 KVM guest runs xrdp/xorgxrdp/XFCE and the pinned Guacamole 1.6.0 translator. A separate owned OpenBao 2.3.1 container exposes TLS only on a loopback port. Synthetic credentials and generated TLS/SSH material remain in a 0700 fixture directory. The reconciler uses a one-hour token limited to `secret/data/desktop-reconcile/*`; the fixture-only root token bootstraps that policy. The worker supervisor tests use real local foreground processes and private pipes.

Pinned artifacts:

  • Debian base SHA-512: `08fea112563461f251f3c95a5c5cf8cb25eb60f74cec03e85a97ff91d3efef3059d35837598bbb476008f20db6d3bdc7143c5f2f2a9a6da394a0acc601fd5986`.
  • Guacd: `guacamole/guacd@sha256:8974eaa9ba32f713daf311e7cc8cd7e4cdfba1edea39eed75524e78ef4b08f4f`.
  • Guacamole WAR SHA-256: `b41ceb1e2df010b54db563e0b00edb8d5fe9f073c6168462e4c978df0fc6e716`.
  • OpenBao: `quay.io/openbao/openbao@sha256:41dc3e47da01575e1ffea70aa635180b57ff999264398313334c259a697bf7a2`.

Procedure

From the repository root, use independently verified local base/WAR artifacts:

python3 scripts/desktop-comparison/vm_fixture.py start \
  --base /path/to/debian-12-genericcloud-amd64.qcow2 \
  --sha512 08fea112563461f251f3c95a5c5cf8cb25eb60f74cec03e85a97ff91d3efef3059d35837598bbb476008f20db6d3bdc7143c5f2f2a9a6da394a0acc601fd5986 \
  --output /tmp/desktop-lifecycle-fixture
python3 scripts/desktop-comparison/vm_fixture.py status --output /tmp/desktop-lifecycle-fixture
# Continue only when provisioned=true.
python3 scripts/desktop-comparison/prepare-vm-rdp.py /tmp/desktop-lifecycle-fixture \
  --guacamole-war /path/to/guacamole-1.6.0.war
python3 scripts/desktop-comparison/reconciliation-fixture.py configure /tmp/desktop-lifecycle-fixture
AGENTIC_DESKTOP_RECONCILE_FIXTURE=/tmp/desktop-lifecycle-fixture \
  cargo test --manifest-path management/Cargo.toml --lib \
  desktop_guest_live_pinned_rdp_openbao_and_cleanup -- --ignored
python3 scripts/tests/test-desktop-worker-supervisor.py
python3 scripts/desktop-comparison/reconciliation-fixture.py cleanup /tmp/desktop-lifecycle-fixture

Fixture creation is intentionally non-idempotent: choose a new private directory. Resource creation and cleanup inside the fixture exercise their retry semantics. The configure helper packages the manual owned-fixture setup used for this run; cleanup was exercised through the helper. Never substitute a production guest.

Verification

  • Sixteen independent SQLite connections produce exactly one controller owner.

Restart, renewal replay, wrong-worker ACK and generation changes do not release that slot. Idle expiry cannot be renewed or reset by late input.

  • The broker test starts a real process, denies a competing controller before its

command executes, revokes the active worker on membership loss, then permits replacement only after confirmed teardown. The final separate-supervisor run passed in 20.36 seconds.

  • Three standalone process tests pass: stalled-broker expiry, Fence followed by

Renew, and broker process-group SIGKILL causing control-pipe loss and worker teardown.

  • The live Rust adapter test provisions two distinct accounts through scoped TLS

OpenBao, confirms fresh pinned RDP/XFCE sessions, rejects a wrong certificate pin, and verifies sign-out plus credential retirement. Final run passed in 9.53 seconds.

  • Full library suite: 1046 passed, four opt-in tests ignored, 61.42 seconds. The

subsequent generation-change controller check passed separately. Six managed seed tests pass, including helper permissions and SSH denial material.

  • Formatting, Python compilation, documentation links and diff checks are recorded

with the final manifest. The management binary check also passed. The updated runtime Dockerfile ships Python/SSH and the adapter assets; its image build remains a CI gate. Changed-module coverage is not yet recomputed.

Sanitized hashes and cleanup receipts are in the qualification manifest. Private logs, account mappings, images, root tokens and generated credentials are not published.

Troubleshooting

The initial OpenBao dev process could not write `/.vault-token.tmp` under the unprivileged UID. The owned replacement uses `-dev-no-store-token`; token and TLS checks were preserved. The seed regression initially expected every generated file to be 0644; it now explicitly requires 0755 for the root helper and 0644 for its non-executable configuration. An initial broker test used the wrong grant accessor; it was corrected to the existing `expose_token()` API before passing.

House rules for agents

Verify the fixture directory owner/mode and exact QEMU command before mutation. Check Docker ownership labels and exact IDs before cleanup. Use pinned SSH/TLS identity, synthetic credentials, bounded commands and private logs. Fail readiness or cleanup on uncertainty; never replace failed verification with metadata alone.

What not to fix

Do not modify host SSH, production xrdp, organizational OpenBao policy, identity realms or existing guests to make this fixture pass. These tests do not qualify malicious native-worker containment or isolation from a root-capable co-resident harness. Browser input routing and isolated per-attachment RDP execution remain separate acceptance work.

Audit trail

Source, test-log and cleanup hashes are recorded in the linked manifest. The live run used `/tmp/desktop-854-live24`; both owned OpenBao containers, the host SSH tunnel and the QEMU process were stopped and checked absent. Private fixture data is retained for local diagnosis.