Agent workflow: branch off main, rehearse on PRE, never act on PROD, tear PRE down #54

Merged

pit merged 1 commit from hermes/agent-workflow into main

2026-10-07 16:40:43 +00:00

pit commented

2026-10-07 16:34:19 +00:00

Owner

Summary

The four rules an Agent works this repo by now have a home, and the machinery they depend on exists.

docs/agents/
├── domain.md
├── issue-tracker.md
└── workflow.md          # NEW: branch, rehearse on PRE, never act on PROD, destroy PRE

 Makefile
+worktree:  ## Create a worktree + fresh branch off origin/main, link the secrets, seed state
-pre:       ## Rehearse: create the PRE Guest, apply the service to it, leave Prod untouched
-ssh-pre:   ## Open a shell on the PRE Guest
+pre:       ## Rehearse: create the PRE Guest, apply the service to it, leave PROD untouched
+ssh-pre:   ## Open a shell on the PRE Guest (its DHCP address, read from state)
 pre-down:  ## Tear the PRE Guest down and clear its host key, leaving nothing behind

pre-down's description is untouched. Its body changes from a fixed $(PRE_HOST) to the address read from state:

 pre-down:
-	tofu destroy $(PRE_CONTAINER)
-	ssh-keygen -R "$(PRE_HOST)"          # 10.12.0.142, hardcoded
+	@ip=$(call pre_ip)                  # the DHCP address, from state
+	tofu destroy $(PRE_CONTAINER)
+	if [ -n "$$ip" ]; then ssh-keygen -R "$$ip"; fi

 tofu/
  container.tf          # PROD Guest — pinned, vmid 141, 10.12.0.141
  pre.tf                # PRE Guest
-   vm_id = var.guest_pre.vmid          # 142
-   ipv4 { address = var.guest_pre.ip } # 10.12.0.142/24
+   (no vm_id)                          # provider assigns one
+   ipv4 { address = "dhcp" }           # DHCP leases the address
+   wait_for_ip { ipv4 = true }
+output forgejo_pre_ipv4  # the address, read back from state
 providers.tf
+  random_vm_ids = true   # concurrency-safe vmid assignment

PROD's identity is untouched: vmid 141, 10.12.0.141 — the Edge and the router rule name that address. PRE's becomes dynamic so two Agents can rehearse at once without colliding on the Node.

Evidence

make worktree — one real run, then the tree it produced:

$ make worktree smoke-test
Preparing worktree (new branch 'hermes/smoke-test')
branch 'hermes/smoke-test' set up to track 'origin/main'.

  branch:  hermes/smoke-test (from origin/main)     # == origin/main @ b39caec
  secrets: tofu/.env tofu/npm/.env ansible/.env ansible/vault-pass  -> linked
  state:   tofu/terraform.tfstate tofu/npm/terraform.tfstate         -> copied

The PRE change, as one read-only tofu plan — before this branch PRE pins the address, after it takes a lease:

   ~ ip_config { ~ ipv4 {
-      address = "10.12.0.142/24"
-      gateway = "10.12.0.1"
+      address = "dhcp"
     } }
+  + wait_for_ip { + ipv4 = true }
Plan: 0 to add, 1 to change, 0 to destroy.   # only PRE; PROD and the base image untouched

The rehearsal path, end to end — the address now flows from state into the playbook:

$ make -n pre
tofu apply plan.out
ip=$(cd tofu && tofu output -json forgejo_pre_ipv4 | jq -r '.["eth0"]' | cut -d/ -f1)
PRE is at $ip
cd ansible && ansible-playbook -i inventories/pre/hosts -e ansible_host=$ip --check --diff site.yml \
           && ansible-playbook -i inventories/pre/hosts -e ansible_host=$ip site.yml

A scratch Ansible probe proves the PRE vars derive from that address while PROD's resolve unchanged:

$ ansible-playbook -i inventories/pre/hosts -e ansible_host=10.12.0.150 probe.yml
  "PRE vars derive from ansible_host"   # domain/root_url/mcp allowlist all == 10.12.0.150
$ ansible-playbook -i inventories/prod/hosts probe-prod.yml
  "PROD vars unchanged"                 # forgejo.thepit.space, 10.12.0.141

Fixing an early bug in this branch (caught in review): tofu output -raw on the ipv4 map errors, and under set -e that aborted pre-down before it destroyed PRE.

-ip=$(tofu output -raw forgejo_pre_ipv4 | jq -r '.["eth0"]')
+ip=$(tofu output -json forgejo_pre_ipv4 | jq -r '.["eth0"]' || true)

$ tofu output -raw  m   # m is a map
Error: Unsupported value for raw output ... Use the -json option.    exit 1
$ tofu output -json m | jq -r .eth0
10.12.0.9                                                            exit 0

Gate checks: tofu fmt -check -recursive clean · tofu validate success · playbook syntax OK · wiki manifest has workflow.md under exclude.

Merge Danger

Door: two-way. Docs, a Makefile Target and a Stack config change, all revertible; nothing here mutates PROD.

Blast Radius: PROD-safe / PRE-replacing.

PROD is untouched: same Guest, same vmid, same address, same playbook path.
The next PRE apply replaces the existing rehearsal Guest (address -> DHCP, vmid re-assigned) — the one destructive-ish effect, and only on the disposable Guest.
Terminology Prod -> PROD lands across ~14 files; identifiers (make prod, inventories/prod, the typed gate word) stay lowercase on purpose — the glossary says so, so nobody "fixes" them.

Closes #52

## Summary The four rules an Agent works this repo by now have a home, and the machinery they depend on exists. ```text docs/agents/ ├── domain.md ├── issue-tracker.md └── workflow.md # NEW: branch, rehearse on PRE, never act on PROD, destroy PRE ``` ```diff Makefile +worktree: ## Create a worktree + fresh branch off origin/main, link the secrets, seed state -pre: ## Rehearse: create the PRE Guest, apply the service to it, leave Prod untouched -ssh-pre: ## Open a shell on the PRE Guest +pre: ## Rehearse: create the PRE Guest, apply the service to it, leave PROD untouched +ssh-pre: ## Open a shell on the PRE Guest (its DHCP address, read from state) pre-down: ## Tear the PRE Guest down and clear its host key, leaving nothing behind ``` `pre-down`'s description is untouched. Its body changes from a fixed `$(PRE_HOST)` to the address read from state: ```diff pre-down: - tofu destroy $(PRE_CONTAINER) - ssh-keygen -R "$(PRE_HOST)" # 10.12.0.142, hardcoded + @ip=$(call pre_ip) # the DHCP address, from state + tofu destroy $(PRE_CONTAINER) + if [ -n "$$ip" ]; then ssh-keygen -R "$$ip"; fi ``` ```diff tofu/ container.tf # PROD Guest — pinned, vmid 141, 10.12.0.141 pre.tf # PRE Guest - vm_id = var.guest_pre.vmid # 142 - ipv4 { address = var.guest_pre.ip } # 10.12.0.142/24 + (no vm_id) # provider assigns one + ipv4 { address = "dhcp" } # DHCP leases the address + wait_for_ip { ipv4 = true } +output forgejo_pre_ipv4 # the address, read back from state providers.tf + random_vm_ids = true # concurrency-safe vmid assignment ``` PROD's identity is untouched: vmid 141, `10.12.0.141` — the Edge and the router rule name that address. PRE's becomes dynamic so two Agents can rehearse at once without colliding on the Node. ## Evidence `make worktree` — one real run, then the tree it produced: ```text $ make worktree smoke-test Preparing worktree (new branch 'hermes/smoke-test') branch 'hermes/smoke-test' set up to track 'origin/main'. branch: hermes/smoke-test (from origin/main) # == origin/main @ b39caec secrets: tofu/.env tofu/npm/.env ansible/.env ansible/vault-pass -> linked state: tofu/terraform.tfstate tofu/npm/terraform.tfstate -> copied ``` The PRE change, as one read-only `tofu plan` — before this branch PRE pins the address, after it takes a lease: ```diff ~ ip_config { ~ ipv4 { - address = "10.12.0.142/24" - gateway = "10.12.0.1" + address = "dhcp" } } + + wait_for_ip { + ipv4 = true } Plan: 0 to add, 1 to change, 0 to destroy. # only PRE; PROD and the base image untouched ``` The rehearsal path, end to end — the address now flows from state into the playbook: ```text $ make -n pre tofu apply plan.out ip=$(cd tofu && tofu output -json forgejo_pre_ipv4 | jq -r '.["eth0"]' | cut -d/ -f1) PRE is at $ip cd ansible && ansible-playbook -i inventories/pre/hosts -e ansible_host=$ip --check --diff site.yml \ && ansible-playbook -i inventories/pre/hosts -e ansible_host=$ip site.yml ``` A scratch Ansible probe proves the PRE vars derive from that address while PROD's resolve unchanged: ```text $ ansible-playbook -i inventories/pre/hosts -e ansible_host=10.12.0.150 probe.yml "PRE vars derive from ansible_host" # domain/root_url/mcp allowlist all == 10.12.0.150 $ ansible-playbook -i inventories/prod/hosts probe-prod.yml "PROD vars unchanged" # forgejo.thepit.space, 10.12.0.141 ``` Fixing an early bug in this branch (caught in review): `tofu output -raw` on the `ipv4` *map* errors, and under `set -e` that aborted `pre-down` before it destroyed PRE. ```diff -ip=$(tofu output -raw forgejo_pre_ipv4 | jq -r '.["eth0"]') +ip=$(tofu output -json forgejo_pre_ipv4 | jq -r '.["eth0"]' || true) ``` ```text $ tofu output -raw m # m is a map Error: Unsupported value for raw output ... Use the -json option. exit 1 $ tofu output -json m | jq -r .eth0 10.12.0.9 exit 0 ``` Gate checks: `tofu fmt -check -recursive` clean · `tofu validate` success · playbook syntax OK · wiki manifest has `workflow.md` under `exclude`. ## Merge Danger **Door:** two-way. Docs, a Makefile Target and a Stack config change, all revertible; nothing here mutates PROD. **Blast Radius:** PROD-safe / PRE-replacing. - PROD is untouched: same Guest, same vmid, same address, same playbook path. - The next PRE apply **replaces** the existing rehearsal Guest (address -> DHCP, vmid re-assigned) — the one destructive-ish effect, and only on the disposable Guest. - Terminology `Prod` -> `PROD` lands across ~14 files; identifiers (`make prod`, `inventories/prod`, the typed gate word) stay lowercase on purpose — the glossary says so, so nobody "fixes" them. Closes #52

pit added 1 commit

2026-10-07 16:34:19 +00:00

docs(agents): how an Agent works — branch, rehearse on PRE, never touch PROD 19e170cae1

The four rules the Operator works this repo by had no home: `AGENTS.md` and
`docs/agents/` said how an Agent reads the tracker and the domain docs, but not
how it works.

- New repo-only `docs/agents/workflow.md`: fresh branch off `main` in a
  worktree, rehearse on PRE, never act on PROD, destroy PRE. Each rule carries
  its why and its limit; the procedure stays in the deploy runbook.
- `make worktree <issue>-<slug>`: branch off `origin/main`, link the
  gitignored secrets from the primary checkout, seed a state copy.
- PRE's identity becomes dynamic — DHCP address, provider-assigned vmid — so
  two Agents can rehearse at once without colliding on the Node. PROD stays
  pinned at vmid 141 / 10.12.0.141, which the Edge and the router rule name.
- Vocabulary normalises to PROD and PRE; `Rehearsal` joins the glossary.
  Identifiers keep lowercase `prod` (`make prod`, `inventories/prod`).
- ADR 0004 amended: the per-worktree state copy and PRE's dynamic identity.

Closes #52

pit force-pushed hermes/agent-workflow from 19e170cae1 to aa65d8f3fe

2026-10-07 16:37:44 +00:00

Compare

pit merged commit dfc1be2629 into main

2026-10-07 16:40:43 +00:00

pit deleted branch hermes/agent-workflow

2026-10-07 16:40:43 +00:00

pit referenced this pull request from a commit

2026-10-07 16:40:44 +00:00

Agent workflow: branch off main, rehearse on PRE, never act on PROD, tear PRE down (#54)