`make prod` ansible check fails if dest dir doesn't exist #50

New issue

Closed

opened 2026-10-07 06:53:06 +00:00 by pit · 0 comments

pit commented

2026-10-07 06:53:06 +00:00

Owner

Problem

make prod (and make pre) stop at the service step. The service Target rehearses with ansible-playbook --check --diff before the real run (Makefile:266-271), and the check itself fails on any Guest that has not been converged, so the service is never applied.

Reproduced live on fresh PRE (LXC 142): --check exits failed=1:

TASK [forgejo : download the pinned release binary, checksum first]
fatal: [forgejo-pre]: FAILED! => {"msg": "Destination /srv/forgejo/bin does not exist", ...}

The Prod Guest has also never had the forgejo_mcp role applied (/srv/forgejo-mcp absent), so this is the normal first make prod, and every make pre rehearsal (PRE is created fresh each time). It is not an edge case — it is the default path.

Root cause

Check mode makes first-class steps no-ops that later steps depend on, and several modules do not tolerate their input being absent even under --check:

file: state=directory reports changed, creates nothing.
get_url fails when its dest's parent directory is absent.
unarchive fails when its src (the archive) is absent; copy with remote_src when its source is absent.
systemd: state=started/restarted fails on a unit file that was never rendered (Could not find the requested service …: host).
become_user: postgres fails before reaching any module when the apt install that creates the user was check-skipped: Unprivileged become user would be unable to read the file.

uri and wait_for are unaffected — Ansible auto-skips modules with no check-mode support. On a converged Guest none of this fires: every input exists, the check previews, and the run is unchanged.

Reproduced live on fresh PRE, in order — the forgejo role fails first, before forgejo_mcp runs:

forgejo : download the pinned release binary — /srv/forgejo/bin missing.
forgejo : ensure the service role exists — postgres user absent (check-skipped install).
forgejo_mcp : fetch the pinned release archive — /srv/forgejo-mcp/releases missing.
forgejo_mcp : enable and start the service and its restart forgejo-mcp handler — unit never rendered.

Fix

Prefer declarative shapes; guard only what --check genuinely cannot produce. Do not make the check mutate: the Makefile and runbook both state it writes nothing, so a check_mode: false "create the tree even when checking" is not the fix.

One stat per fact, read in check mode, used as the guard predicate:

forgejo_tree (/srv/forgejo/bin exists) around the get_urls, the two systemd starts, and the two handlers in the forgejo role.
forgejo_pg (/usr/bin/psql exists) around the five postgres tasks in the forgejo role.
forgejo_mcp_tree (/srv/forgejo-mcp/releases exists) around the install block (§2), the start block (§4), and the handler in forgejo_mcp.

- name: check whether the release tree exists yet
  ansible.builtin.stat:
    path: "{{ forgejo_mcp_releases_dir }}"
  register: forgejo_mcp_tree
  changed_when: false

- name: install the pinned release
  when: not ansible_check_mode or forgejo_mcp_tree.stat.isdir | default(false)
  block:
    - name: fetch the pinned release archive, checksum first
      ansible.builtin.get_url: { … }
    - name: unpack the release binary straight into bin/, stripped
      ansible.builtin.unarchive: { … }
      notify: restart forgejo-mcp

Plus the declarative fold in forgejo_mcp: unarchive straight into {{ forgejo_mcp_home }}/bin with extra_opts: ["--strip-components=1"], deleting the separate install the binary the unit runs copy task and the intermediate releases/<stem>/ directory.

Constraint to keep: get_url must stay. unarchive has no checksum parameter (its own docs say to use get_url when checksum validation is wanted), so fetching inside unarchive would drop the pinned-sha256 verification the runbook records. get_url is also the cache: it revalidates with a HEAD (no body) on later runs, while unarchive-from-URL re-downloads the full ~5.7 MB on every run and every check (measured against a local server log).

Acceptance criteria

--check against a not-yet-converged Guest completes failed=0 and writes nothing to the Guest (both roles).
Stage 1 (forgejo role only) converges; stage 2 (site.yml, adding forgejo_mcp) converges on top; a re-run reports 0 changed.
--check against a converged Guest previews the install unchanged (failed=0, nothing newly skipped).
The MCP endpoint answers 401 to a credential-less request and 403 to an unlisted Host, on PRE.
make prod reaches step 4's real apply on a fresh Guest instead of dying in its check.

Evidence (live on PRE 142)

Fresh --check: failed=1 with Destination /srv/forgejo/bin does not exist (pre-fix) → failed=0 (post-fix), nothing written.
Stage 1 changed=22; stage 2 changed=7; re-run changed=0; converged --check failed=0.
POST http://10.12.0.142:8089/mcp → 401; unlisted Host → 403; forgejo, forgejo-runner, forgejo-mcp all active.
PROD was not touched and not verified — PROD runs after this merges.

Out of scope

Any change to the --check-must-mutate-nothing contract.
Replacing the checksum pin or dropping get_url (see the constraint above).

## Problem `make prod` (and `make pre`) stop at the service step. The `service` Target rehearses with `ansible-playbook --check --diff` before the real run (`Makefile:266-271`), and the check itself fails on any Guest that has not been converged, so the service is never applied. Reproduced live on fresh PRE (LXC 142): `--check` exits `failed=1`: ``` TASK [forgejo : download the pinned release binary, checksum first] fatal: [forgejo-pre]: FAILED! => {"msg": "Destination /srv/forgejo/bin does not exist", ...} ``` The Prod Guest has also never had the `forgejo_mcp` role applied (`/srv/forgejo-mcp` absent), so this is the normal first `make prod`, and every `make pre` rehearsal (PRE is created fresh each time). It is not an edge case — it is the default path. ## Root cause Check mode makes first-class steps no-ops that later steps depend on, and several modules **do not tolerate their input being absent** even under `--check`: - `file: state=directory` reports `changed`, creates nothing. - `get_url` fails when its `dest`'s parent directory is absent. - `unarchive` fails when its `src` (the archive) is absent; `copy` with `remote_src` when its source is absent. - `systemd: state=started`/`restarted` fails on a unit file that was never rendered (`Could not find the requested service …: host`). - `become_user: postgres` fails *before reaching any module* when the apt install that creates the user was check-skipped: `Unprivileged become user would be unable to read the file`. `uri` and `wait_for` are unaffected — Ansible auto-skips modules with no check-mode support. On a converged Guest none of this fires: every input exists, the check previews, and the run is unchanged. Reproduced live on fresh PRE, in order — the `forgejo` role fails **first**, before `forgejo_mcp` runs: 1. `forgejo : download the pinned release binary` — `/srv/forgejo/bin` missing. 2. `forgejo : ensure the service role exists` — `postgres` user absent (check-skipped install). 3. `forgejo_mcp : fetch the pinned release archive` — `/srv/forgejo-mcp/releases` missing. 4. `forgejo_mcp : enable and start the service` and its `restart forgejo-mcp` handler — unit never rendered. ## Fix Prefer declarative shapes; guard only what `--check` genuinely cannot produce. Do **not** make the check mutate: the Makefile and runbook both state it writes nothing, so a `check_mode: false` "create the tree even when checking" is not the fix. One `stat` per fact, read in check mode, used as the guard predicate: - `forgejo_tree` (`/srv/forgejo/bin` exists) around the `get_url`s, the two `systemd` starts, and the two handlers in the `forgejo` role. - `forgejo_pg` (`/usr/bin/psql` exists) around the five `postgres` tasks in the `forgejo` role. - `forgejo_mcp_tree` (`/srv/forgejo-mcp/releases` exists) around the install block (§2), the start block (§4), and the handler in `forgejo_mcp`. ```yaml - name: check whether the release tree exists yet ansible.builtin.stat: path: "{{ forgejo_mcp_releases_dir }}" register: forgejo_mcp_tree changed_when: false - name: install the pinned release when: not ansible_check_mode or forgejo_mcp_tree.stat.isdir | default(false) block: - name: fetch the pinned release archive, checksum first ansible.builtin.get_url: { … } - name: unpack the release binary straight into bin/, stripped ansible.builtin.unarchive: { … } notify: restart forgejo-mcp ``` Plus the declarative fold in `forgejo_mcp`: `unarchive` straight into `{{ forgejo_mcp_home }}/bin` with `extra_opts: ["--strip-components=1"]`, deleting the separate `install the binary the unit runs` copy task and the intermediate `releases/<stem>/` directory. **Constraint to keep:** `get_url` must stay. `unarchive` has no `checksum` parameter (its own docs say to use `get_url` when checksum validation is wanted), so fetching inside `unarchive` would drop the pinned-sha256 verification the runbook records. `get_url` is also the *cache*: it revalidates with a HEAD (no body) on later runs, while `unarchive`-from-URL re-downloads the full ~5.7 MB on every run **and every check** (measured against a local server log). ## Acceptance criteria - [ ] `--check` against a not-yet-converged Guest completes `failed=0` and writes nothing to the Guest (both roles). - [ ] Stage 1 (`forgejo` role only) converges; stage 2 (`site.yml`, adding `forgejo_mcp`) converges on top; a re-run reports `0 changed`. - [ ] `--check` against a converged Guest previews the install unchanged (`failed=0`, nothing newly skipped). - [ ] The MCP endpoint answers `401` to a credential-less request and `403` to an unlisted `Host`, on PRE. - [ ] `make prod` reaches step 4's real apply on a fresh Guest instead of dying in its check. ## Evidence (live on PRE 142) - Fresh `--check`: `failed=1` with `Destination /srv/forgejo/bin does not exist` (pre-fix) → `failed=0` (post-fix), nothing written. - Stage 1 `changed=22`; stage 2 `changed=7`; re-run `changed=0`; converged `--check` `failed=0`. - `POST http://10.12.0.142:8089/mcp` → `401`; unlisted `Host` → `403`; `forgejo`, `forgejo-runner`, `forgejo-mcp` all `active`. - PROD was **not** touched and not verified — PROD runs after this merges. ## Out of scope - Any change to the `--check`-must-mutate-nothing contract. - Replacing the checksum pin or dropping `get_url` (see the constraint above).

pit added the

ready-for-agent

label

2026-10-07 11:45:17 +00:00

pit referenced this issue from a commit

2026-10-07 11:47:38 +00:00

fix(ansible): --check survives a not-yet-converged Guest

pit referenced this issue from a pull request that will close it,

2026-10-07 11:47:46 +00:00

fix(ansible): --check survives a not-yet-converged Guest (Closes #50) #51

pit referenced this issue from a commit

2026-10-07 12:07:40 +00:00