feat(gateway): add reconciler lease for HA multi-replica deployments

[derekwaynecarr]

Summary

Introduce a database-backed reconciler lease so that only one gateway replica runs the watch and reconcile loops in Postgres-backed HA deployments. SQLite (single-replica) deployments skip the lease and run unconditionally as before.

The lease is a lightweight JSON record in the objects table using CAS for cross-replica safety. A lease coordinator on each replica attempts acquisition, runs renewal while holding, and releases on shutdown for fast failover. Watch and reconcile loops now accept a cancellation channel for cooperative shutdown.

Related Issue

Closes #1429

Changes

Testing

• [ x] mise run pre-commit passes
• [ x] Unit tests added/updated
• [ x] E2E tests added/updated (if applicable)

Checklist

• [x ] Follows Conventional Commits
• [x ] Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[derekwaynecarr]

This is WIP

[derekwaynecarr]

Need to figure out what we want to do in CI for HA setups.

[derekwaynecarr]

/ok to test 26b9e70

[derekwaynecarr]

/ok to test ac1e51f

[github-actions]

Label test:e2e-kubernetes applied, but pull-request/1577 is at 26b9e70 while the PR head is ac1e51f. A maintainer needs to comment /ok to test ac1e51febee115a734e08341307100f4f512d231 to refresh the mirror. Once the mirror catches up, re-run Branch E2E Checks from the Actions tab.

[TaylorMutch]

/ok to test ac1e51f

[github-actions]

Label test:e2e applied for ac1e51f. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute the standard E2E suite after building the required gateway and supervisor images once. The matching required CI gate status on this PR will flip green automatically once the run finishes.

[TaylorMutch]

My agent had this for feedback:

---

• Findings

  1. Medium: shutdown does not wait for the lease coordinator to finish. /private/tmp/openshell-pr1577-review/crates/openshell-server/src/lib.rs:350 spawns watchers without retaining a JoinHandle, then shutdown only waits for listener tasks before cleanup_on_shutdown() at /private/tmp/
     openshell-pr1577-review/crates/openshell-server/src/lib.rs:446. That makes lease release and loop cancellation best-effort; cleanup can run while reconcile/watch tasks are still active, and process exit can leave the lease until TTL expiry. The coordinator should be tracked and awaited
     before shutdown cleanup.

  2. Medium: the holder releases the DB lease before stopping its child loops. In run_as_holder, shutdown calls lease.release(guard).await before cancel_tx.send(true) and awaiting the watch/reconcile handles at /private/tmp/openshell-pr1577-review/crates/openshell-server/src/compute/
     mod.rs:816. A standby can acquire immediately while the old holder’s reconcile sweep is still running; cancellation is only observed after reconcile_store_with_backend() returns at /private/tmp/openshell-pr1577-review/crates/openshell-server/src/compute/mod.rs:887. Cancel and await the
     child loops before deleting the lease.

[TaylorMutch]

I did test this locally with postgres and validated this worked as expected, so I am okay to merge this as-is to get the ball rolling forward

[TaylorMutch]

Output from my test:

---

• Validated PR #1577 locally on branch pr-1577.

What passed

• Created k3d cluster openshell-dev-pr-1577.

• Deployed bundled Postgres with replicaCount: 2 via mise run helm:skaffold:run.

• Verified:

  • openshell StatefulSet ready at 2/2.
  • Postgres ready at 1/1.
  • Lease row existed in Postgres with holder openshell-0.
  • Only openshell-0 ran reconcile sweeps; openshell-1 stayed standby.

• Deleted holder pod openshell-0.

  • Lease moved to openshell-1.
  • openshell-1 began reconcile sweeps.

• Re-deployed bundled Postgres with replicaCount: 1.

  • openshell-0 acquired the lease and began reconcile sweeps.

Important caveat
Failover worked, but it appeared to wait for TTL expiry rather than immediate graceful release. After deleting/scaling down the current holder, the old holder remained in the lease row until expiry, then the next replica acquired it. That matches the lifecycle concern from the review:
shutdown/release is not strongly coordinated.

[derekwaynecarr]

thanks @TaylorMutch , merging so we can move forward next with your PR.