10  Recipe: run the full garak baseline before sign-off

Field	Value
Tool	garak
Section	Scanning
Level	hard
Time	hours (full catalogue; plan per model)
Layer/Category	ATTACK / A-2
Tags	scan, baseline, garak, avid, regression

10.1 Problem

The scoped scan answers one question cheaply (LLM01 here, data-leakage there) — but nothing signs off on a model with a stack of single-question runs. An assessment needs the full-catalogue baseline: every probe family, one seed for reproducibility, the raw JSONL retained, and AVID reports as the hand-off artefact — repeated after every model, system-prompt or guardrail change so the diff is the change’s security statement.

10.2 Solution

Budget a window (the full catalogue is hours per model), then run the baseline end to end:

# 1. Confirm the pinned tool (Kailash ATTACK/A-2)
garak --version

# 2. Plan the spend: the catalogue is big — see what you are buying
garak --list_probes -v          # tiers + versions, no target needed

# 3. The full baseline (no --spec = the whole probe catalogue),
#    generous generations + a fixed seed + parallel headroom:
garak --target_type ollama --target_name <model> \
      -g 20 -s 1337 --parallel_attempts 16 \
      --report_prefix baseline-<model-tag>

What success looks like:

• the banner + 📜 reporting to baseline-<model-tag>_<timestamp>.jsonl names THIS run’s deliverable file
• per-probe progress lines end in per-detector pass/fail fractions; the closing summary lists the weakest probes by detector score
• the run’s JSONL (attempts + statuses) and the hit log are both in the working directory — the retained evidence

Convert and diff — the assessment hand-off and the regression loop:

# 4. AVID reports for the ticket/assessment
garak --report baseline-<model-tag>_<timestamp>.jsonl

# 5. The regression loop: after a system-prompt / guardrail / model change,
#    same seed, same target, new prefix:
garak --target_type ollama --target_name <model> \
      -g 20 -s 1337 --parallel_attempts 16 \
      --report_prefix retest-<change-tag>
garak --report retest-<change-tag>_<timestamp>.jsonl

# then compare per-probe detector scores across the two reports —
# what moved is what the change bought (or broke)

Expected output shape (abridged):

garak LLM safety & security scanning tool v<version> ( https://github.com/NVIDIA/garak ) at <ISO timestamp>
📜 reporting to baseline-<model-tag>_<timestamp>.jsonl
...
🔍 probing <probe> (<n> generation(s))
   <detector>  passed  <n>/<m>
...
✅ <n>/<m> probes passed / ❌ <k> failed (or the closing summary's shape)

10.3 Discussion

• Baseline vs scoped runs: scope runs answer questions (--spec tag:…, per-family specs); the baseline is the umbrella evidence — one run, whole catalogue, before an assessment is signed off. In practice: full baseline first, scoped re-runs after every change (the same-seed diff does the day-to-day work).
• Seeds are the experiment control: -s 1337 fixed across runs keeps the probe/target interaction set comparable; the report diff is then a statement about the change, not sampling noise. Change only one thing between runs.
• Generations are the confidence dial: -g 20 for a baseline (detector fractions stabilise), higher on gated models. Bootstrap confidence stats are available (--confidence_interval_method bootstrap with the --bootstrap_* knobs) when a claim needs an interval, not a fraction.
• --parallel_attempts is for API-tolerant targets: hosted or local serving stacks that accept parallel load run much faster (garak hints at the knob when the generator is parallel-capable); keep the seed constant so parallelism does not corrupt comparability.
• The target endpoint is the blast-radius boundary: run the server loopback on a lab box (Serve a local model for scanning); never point the baseline at anything you do not own — the full catalogue is far more invasive than a scoped run.
• Evidence retention: keep the JSONL + hit log + the AVID conversion in the assessment record; cite the ATLAS technique ids from the manifest row (AML.T0051/T0054, dataset-verified) rather than re-deriving them. The report-verification loop recipe covers the full reporting discipline (--record-based verbatim capture for anything that lands in a ticket).

10.4 See also

• garak reference page on the docs site: https://docs.kailash.tools/reference/tools/garak.html
• Serve a tiny local model and run your first LLM01 scan end-to-end — the scoped, minutes-cost loop this baseline complements
• Turn scan reports into verifiable assessment evidence — the reporting discipline
• promptfoo / pyrit / counterfit recipes — the sister-family walkthroughs
• Kailash OS #81 — garak packaging

---

ref: v0.17.1.pre1-unstable-2026-10-09 - verified 2026-10-14. CLI surfaces from the landed --help/CLI contract (flags above verified against the guide’s garak page); packaged-binary smoke PENDING: kailash-os#81.