[EPIC] A Makefile packages the deploy procedure: make prod / make pre / make pre-down #27
Labels
No labels
needs-info
needs-triage
ready-for-agent
ready-for-human
wontfix
needs-info
needs-triage
ready-for-agent
ready-for-human
review/merge-ready
review/needs-fix
review/needs-human
review/needs-review
wayfinder:grilling
wayfinder:map
wayfinder:prototype
wayfinder:research
wayfinder:task
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
olympus/infra-forge#27
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem Statement
Deploying and rehearsing this stack means recalling a runbook's worth of exact incantations, layer by layer: which directory to stand in, which
.envto source for that layer, when-var-fileis mandatory, which inventory to point Ansible at, and which-targetset selects the Guest in play. Getting any one of those wrong acts on the wrong environment or aborts mid-deploy; a hand-runtofu applyin the Guest Stack has no notion of Prod-versus-PRE at all.Rehearsals make it worse. PRE's config is parked on a separate branch, so rehearsing means checking files out into the working tree and restoring them afterwards — and the documented teardown does not actually restore the parked inventory file (a stages-the-file checkout is being undone with a reads-the-index one), so a rehearsal can silently leave PRE content on
main.An Agent is as likely as the Operator to run these commands, and an Agent that improvises the sequence is exactly as wrong as a careless one. There is no packaged form of the procedure for either to run.
Solution
A single
Makefileat the repo root packages every step of the procedure as a named Target, so the Operator or an Agent runsmake prodormake preand gets the layers applied in the right order, against the right environment, with the gates, the pre-flight checks and the post-flight checks built in.Each Target is the glossary's own word for the thing it does (Guest, Edge, service, Prod, PRE), and each encodes the knowledge the runbook carried in prose: the working directory, the
.envto source, the mandatory-var-file, the inventory, the-targetset, and a full plan reviewed before any apply.PRE stops being a parked branch. Its three config files move onto
main, and Prod and PRE are told apart by Target selection alone — somake precan never disturb Prod, andmake prodcan never disturb PRE. The broken teardown disappears with the mechanism it belonged to.The docs follow: the runbook gains a "the Makefile" section ahead of the raw commands, which stay as the explanation and the if-make-is-unavailable path; the glossary gains Target; and an ADR records why Prod and PRE share one Stack split by target rather than splitting into two.
User Stories
As an Operator, I want one command that deploys the whole Prod stack, so that I run six remembered incantations as one.
As an Operator, I want
make prodto run Guest, then Edge, then service, so that the Edge is published before the runner is expected to reach the public URL.As an Operator, I want each layer runnable on its own (
make guest,make edge,make service), so that a re-run after a one-layer change does not touch the layers that did not change.As an Operator, I want every apply to render the full plan first, so that I never apply a plan I have not seen.
As an Operator, I want to confirm an apply by typing the environment's word, so that a reflexive
yor a pipedyescannot apply to the wrong environment.As an Operator, I want the plan I reviewed and the plan that is applied to be the same artifact, so that "reviewed" is not an honour system.
As an Operator, I want read-only Targets I can run freely (
guest-plan,edge-plan,service-check), so that inspecting the estate never mutates it.As an Operator, I want a Target that takes the PBS snapshot before a service-touching run, so that the runbook's mandatory pre-flight gate stops being something I have to remember.
As an Operator, I want a Target that copies the encrypted state outside the repo after an apply, so that the runbook's mandatory post-flight gate is performed rather than remembered.
As an Operator, I want
make verifyto run the external checks (the301/200answers, the SSH clone, the clone URLs), so that "it deployed" is established by evidence rather than by the playbook exiting zero.As an Operator, I want
make preto create the PRE Guest and apply the service to it, so that a rehearsal is one command and leaves Prod alone.As an Operator, I want
make pre-downto destroy the PRE Guest and clear its host key, so that a finished rehearsal leaves nothing behind.As an Operator, I want
make preto skip the Edge, so that a rehearsal does not publish a second WAN-published host.As an Operator, I want
make preto skip the snapshot and the state backup, so that a disposable Guest does not accumulate backup artifacts.As an Operator, I want
make prodto refuse to start when a secret file it needs is missing, so that a half-configured run fails at the top, not in the middle of a layer.As an Operator, I want the Makefile to source each layer's
.envitself, so that the Guest Stack's provider credentials stop being an undocumented step.As an Operator, I want the Edge's
-var-fileto be passed automatically, so that the one required argument of that Stack cannot be forgotten.As an Operator, I want
tofu initto run before a plan or an apply, so that a fresh clone works without a remembered first step.As an Operator, I want
make help, and a baremake, to list the Targets instead of running anything, so that a reflexivemakecannot deploy.As an Operator, I want
make sshandmake ssh-preto reach the Guests, so that the thing I most often do by hand is also a Target.As an Operator, I want
make fmtandmake validate, so that the lint steps are the same everywhere they are invoked.As an Agent, I want to run the packaged procedure instead of improvising the sequence, so that my run is the Operator's run.
As an Agent, I want the read-only Targets to be the ones that are safe to run at will, so that exploratory inspection needs no permission dance.
As an Agent working the issue tracker, I want the deploy Targets to be discoverable by name from
make help, so that I do not have to read the Makefile to learn the vocabulary.As an Operator, I want PRE's config to live on
main, so that a rehearsal needs no branch checkout and no restore afterwards.As an Operator, I want Prod and PRE to be separated by Target selection, so that neither can reach the other's resources by accident.
As an Operator, I want a hand-run
tofu apply(the no-make path) to keep meaning exactly what it means today, so that the Makefile does not change the behaviour of the manual path it wraps.As an Operator, I want the runbook to keep the raw commands and their reasoning, with the Makefile presented first, so that the procedure is still documented for a machine without make.
As an Operator, I want the glossary to define Target, so that the vocabulary of this interface is the same vocabulary as the rest of the docs.
As a future reader, I want an ADR recording why Prod and PRE share one Stack split by target instead of being two Stacks, so that I do not "fix" that split into two states.
Implementation Decisions
Make is the interface. A single root
Makefile; nojust/task(not installed, not warranted), no shell-script wrapper set. Zero-install: make is already present and used everywhere.Default goal is a help listing, never a deploy. A bare
makeprints the Targets.Target set (the deliverable's public interface):
guest-plan,guestedge-plan,edgeservice-check(--check --diff),servicesnapshot,verify,state-backupprod,pre,pre-downssh,ssh-pre,fmt,validate,helpComposition semantics.
make prod= snapshot → guest → edge → service → verify → state-backup, one step after the other.make pre= guest (PRE) → service (PRE inventory), with no Edge, no snapshot, no state backup.make pre-down= destroy the PRE Guest → clear its SSH host key.Prod/PRE are split by target selection on one shared Stack and state. Prod selects the Guest resource (plus the shared base-image resource); PRE selects the PRE Guest resource (plus the same base-image resource, so a fresh machine can build either). The base-image resource rides both selections.
No environment toggle variable. The split is expressed as
-targetselection only; the runbook's raw commands each gain the matching-targetso the manual path and the Makefile agree.The apply gate is three steps, every apply: plan to a saved artifact → render that artifact for review → confirm by typing the environment's word → apply that artifact. The reviewed and applied plans are the same bytes.
The environment word is an attention gate, not an authorisation boundary. Agents are expected to use these Targets; the word exists to stop the wrong environment, not the wrong actor.
The snapshot is scripted, not printed. A PBS snapshot of the Prod Guest via the installed
community.proxmoxsnapshot module (LXC-capable), auto-called before a service-touching run. PBS backup jobs are deferred.The state backup is scripted, not printed. Both Stacks' state is tarred into one timestamped tarball under the existing outside-repo backup directory, encrypted with
gpg --symmetric. The passphrase is the existing vault password from the Ansible layer's env file (reused — commented, since one secret then protects both the vault and the state).The Makefile enforces, per Target: the right working directory, the right
.envsourced, the Edge's-var-filealways,initbefore plan/apply, the right inventory, and refusal when a required secret file is absent.PRE lands on
main. The three parked configuration files — the PRE Guest resource, its host-variable overrides, and its inventory entry — move ontomain; the parked branch is deleted; the parked-branch language leaves the runbook and theguest_prevariable description, while the glossary's PRE entry stays (PRE is still the throwaway Guest) minus the "parked" clause.New glossary term: Target — a named make goal that packages one step of the procedure. ADR 0003 records the shared-state/target-split decision.
A reversal to record: the layers run Guest → Edge → service, not Guest → service → Edge. The runner's config polls the WAN-published URL and the role waits for the runner to appear online; that wait needs the Edge first. No role change.
Testing Decisions
A good test here checks the composed command line, not the recipe. The deliverable is a set of shell wrappers, so the external behaviour of a Target is what it would run — its working directory, the
.envit sources, the flags, the inventory, the-targetset. Recipe internals (helper variables, variable names, ordering of shell tokens that do not change the command) are not tested.The seam is one: invoke a Target read-only and inspect it. Every Target is inspectable with
make -n <target>, which composes and prints the command without executing it. This is the highest available seam — there is no application code to test underneath, and the Makefile is the interface.What gets tested: the target surface itself — that
prodcomposes the six steps in order; thatprecomposes Guest then service with the PRE inventory and no Edge; thatedgecarries the mandatory-var-file; that each apply carries its gate; that a missing secret file stops the target before any mutating command is composed.Prior art: none in-repo. This repo has no test framework; its existing verification is the runbook's external checks and its convention of stating gates as steps. The tests proposed here are dry-run assertions over the Makefile, in keeping with that convention. They are lightweight by design: the Makefile's behaviour is its interface, and the expensive end-to-end check is the existing PRE rehearsal.
Out of Scope
A PBS backup Target. The snapshot is in scope; scheduled backup jobs against the NAS storage are not.
A Prod destroy/rollback Target. Rollback stays the runbook's ordered manual procedure; it is destructive and rare.
The
0 changedidempotence check stays a documented manual second run; baking a second full playbook run intomake verifydoubles a Prod run.Reordering the role or reworking the runner's wait so service could precede the Edge. Rejected in favour of the documented order.
A general-purpose task runner (
just,task) or a shell-script library in place of make.A remote state backend (remote backend, OpenBao). The encrypted local-then-copied state stays the interim answer.
Further Notes
The parked-branch teardown bug is real and was reproduced: a
git checkout <branch> -- <path>(which stages the file) cannot be undone by a pathspec-onlygit checkout(which reads the index), so the parked inventory file survived the documented restore. This disappears with the parked-branch mechanism; the fix is the mechanism's removal, not a patched incantation.The parked branch is stale in a way worth knowing: it was cut before the docs tree and the repo's English translation landed, so a naive merge of it would delete
GLOSSARY.md,LICENSE,AGENTS.mdand all ofdocs/, and revert the.gitignore. Only three of its paths are genuinely PRE's. Do not merge the branch; move the three files.Baseline established before any work: the Prod Guest answers, the service and the runner are both active, the Edge Stack reports no drift, and both Stacks' state files are present.
The backup recipe is a reconstruction. The existing artifacts in the backup directory are an encrypted tarball plus two plaintext PRE-state files; the original encryption command was not recorded anywhere. The chosen recipe (gpg symmetric, reused passphrase) is therefore a reconstruction — a restore from a fresh backup should be verified once before trusting the recipe.
Tickets
The breakdown of this spec, in dependency order.
main; the parkedforgejo-prebranch retires. No blockers.make edge-plan/make edge: the Edge Stack through the Makefile. Blocked by #29.make service-check/make service: the service layer through the Makefile. Blocked by #29.make snapshotandmake state-backup: the pre-flight and post-flight gates. Blocked by #29.make verify: the external checks as a Target. Blocked by #29.make prod: the whole Prod deploy in one command. Blocked by #30, #31, #32, #33.make preandmake pre-down: the PRE rehearsal. Blocked by #28, #29, #31.The frontier — tickets whose blockers are all done — is #28 and #29.
A Makefile packages the deploy procedure: make prod / make pre / make pre-downto [EPIC] A Makefile packages the deploy procedure: make prod / make pre / make pre-downClosing: all nine sub-issues (#28-#36) are done, and the deliverable is on
main.The Makefile packages the target surface the spec named — help, prod, guest-plan, guest, edge-plan, edge, verify, snapshot, state-backup, fmt, validate, service-check, service, ssh, ssh-pre, pre, pre-down — with a bare
makeandmake helplisting it (.DEFAULT_GOAL := help).The composed command lines agree with the spec (checked with
make -n, the seam the spec picked):make -n prodcomposes snapshot -> guest -> edge -> service -> verify -> state-backup, in that order.make -n precomposes guest (PRE-target) -> service againstinventories/pre/hosts, with no Edge, no snapshot and no state backup.make -n edgecarries-var-file=forgejo.tfvarsalways, then the shared gate:tofu show plan.out-> type the environment word ->tofu apply plan.out, so the reviewed and applied plans are the same artifact.Docs landed: Target is in
GLOSSARY.md; ADR 0004 (docs/adr/0004-prod-and-pre-share-one-stack.md) records the shared-state/target-split decision and is in the ADR index; the runbook's "## The Makefile" section sits ahead of the raw commands, which keep the matching-targetso the packaged and manual paths agree.PRE's config is on
main(tofu/pre.tf,ansible/inventories/pre,ansible/host_vars/forgejo-pre.yml) and noforgejo-prebranch survives locally or on the remote, so the broken parked-branch teardown is gone with the mechanism.tofu fmt -check -recursiveis clean.Not re-run here: the live
make verifychecks against Prod and an end-to-endmake pre. Those were exercised when their sub-issues landed; the spec's own Out of Scope keeps the expensive end-to-end check as the PRE rehearsal itself.