Prompt regression testing in CI: catch prompt changes before they ship

• A {{placeholder}} gets renamed in the prompt but not in the code that fills it.
• A constraint such as "answer in JSON" is trimmed while shortening the text.
• A wording tweak shifts the output format that downstream code parses.

The first is mechanical and checkable. The others need the prompt to run, and a human to look at the output. Do the cheap check automatically and make the expensive one easy.

prompts/
  code-review.md      # Review this diff for bugs: {{diff}}
  bug-triage.md
inputs/
  code-review.diff    # a saved sample input

The layout and commit habits are in how to version prompts in git.

This script compares the variables of each modified prompt with the base branch. I ran it on a scratch repo: removing {{lang}} from a prompt printed the FAIL line and exited 1; adding text while keeping every variable exited 0.

#!/usr/bin/env bash
set -euo pipefail
base="${1:-origin/main}"
fail=0
vars() { grep -o '{{[a-zA-Z_][a-zA-Z0-9_]*}}' | sort -u || true; }
for f in $(git diff --name-only --diff-filter=M "$base" -- prompts/); do
  lost=$(comm -23 <(git show "$base:$f" | vars) <(vars < "$f"))
  if [ -n "$lost" ]; then
    echo "FAIL $f: placeholder removed or renamed: $lost"
    fail=1
  fi
done
exit "$fail"

FAIL prompts/code-review.md: placeholder removed or renamed: {{lang}}

It only looks at modified files. New prompts have nothing to regress against, and deleting a file is a review decision.

name: prompts
on:
  pull_request:
    paths: ["prompts/**"]
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Placeholders intact
        run: bash scripts/check-prompts.sh origin/${{ github.base_ref }}
      - name: Word-level diff
        run: git diff --word-diff=plain origin/${{ github.base_ref }} -- prompts/

fetch-depth: 0 matters: the default shallow checkout has no base branch to compare against. The word diff makes a one-sentence edit readable; see how to diff two prompts.

Structure checks do not tell you the output stayed useful. For that, fill the variables from a saved input and run the prompt through the agent command you already use, once for the base version and once for the new one. Keep the API key in a CI secret, and store both outputs as an artifact so a reviewer can compare them.

ple get "code review" --var diff="$(cat inputs/code-review.diff)" | claude -p

ple get prints the prompt raw with variables filled, so it pipes into any agent. The loop that scores variants is covered in test your prompts with your own agent. Write the reason for each change in a changelog entry.

It does not score output quality, and model output varies between runs, so do not fail a build on exact output text. Use the structural check as the gate, and the saved outputs as evidence.