1 · Describe the check
Write it in plain English. You review the test before it is stored or run.
How a sentence becomes a Jaffa test
Person · non-technical
1 · Describe
A tester or analyst writes what to check, in exam vocabulary. No code, no selectors.
"Student goes offline mid-question, reconnects — answers reconciled and scored correctly"
AI agent · skills + tools
2 · Draft & validate
The agent discovers Jaffa's capability catalog, composes a declarative test, and self-repairs against the validator. It can only use catalogued capabilities.
→ list_capabilities (12) → validate_test ✗ repair → validate_test ✓ → finalize_draft
human gate
Person · reviewer
3 · Approve
The draft is reviewed as plain English, YAML and C# side by side. Nothing reaches Jaffa without this click. Drafts are single-use.
plan ✓ yaml ✓ C# ✓ [ Approve — store & run ]
Jaffa · deterministic
4 · Store
The definition lands in the Jaffa store as a suite member — the same declarative artifact that was reviewed is the one that executes.
tests/offline-delivery/ offline-answer-…test.yaml registry.json ← suite entry
Jaffa · deterministic
5 · Execute
The runner maps each capability to a real browser/service action. The AI is out of the loop — a model never decides pass or fail.
✓ start_session ✓ go_offline … reconcile ✓ score_equals → GREEN
Execution fan-out across the platform matrix
⚠ Hypothetical topology. In this POC every run executes locally on Chromium (headless by default,
HEADED=1 for a visible browser). In the real engagement, each card below is the same stored test dispatched by Jaffa to a VM/device pool — Selenium today, Playwright-ready. Wall-clock time = the slowest cell, not the sum.Sequential multi-platform run · visual comparison vs a previous version
⚠ Hypothetical, but grounded in Jaffa's real feature set: one cross-platform API for
Windows/macOS/iOS/Android, checkpoint screenshots embedded in the rich HTML report, and
jaffa.Vision matching for surfaces the DOM can't address (canvas, drag-drop, media).
The same stored test walks an ordered platform queue — same seeded data, same named
checkpoints — so every capture is comparable against the baseline taken on the previous app version.
Diffs never fail the run by themselves: they are queued for human review, with AI triage as an
advisory note (Track 2 never gates).
Capability contract — what the AI layer needs, and how Jaffa provides it
| Capability required | In this POC (mocked) | How to achieve it in real Jaffa |
|---|---|---|
| Discoverable capability catalog ✓ mocked |
capabilities.json — 12 hand-written entries the agent reads via list_capabilities. |
Generate the manifest from Jaffa's page models / IPOM with a small .NET reflection or source-generator pass over the pattern library — the catalog stays in lock-step with the code instead of being maintained by hand. Patterns & Practices · IPOM |
| Declarative test definition (data, not code) ✓ mocked |
YAML of capability calls; C# shown as a display-only preview. | Keep the YAML as the reviewed artifact and compile it to native Jaffa NUnit classes with a ~200-line .NET source generator, so stored tests are first-class Jaffa tests. Jaffa's language-agnostic patterns make the same definition portable to the coming Python/Playwright targets. Apply Jaffa to Any Framework |
| Validate / compile step for agent self-repair ✓ mocked |
validate.ts: schema + catalog + sequencing checks returning precise errors. |
Two deterministic gates: schema validation before codegen, then dotnet build with a Roslyn analyzer that rejects any generated test referencing APIs outside the capability library. Compiler errors feed straight back to the agent. .NET 10 toolchain |
| Test store + suite registry ✓ mocked |
jaffa-store/ files + registry.json. |
Jaffa already has suites — store each approved definition in the suite repo via a pull request, so the human approval gate and the audit trail are git-native (created-by, approved-by, review comments). Suites · config conventions |
| Ephemeral per-test data ✓ mocked |
Fresh browser context + session per run; teardown destroys it. | Already Jaffa's design: it creates only the data a test needs and destroys it afterwards. The catalog's start_session maps 1:1 onto Jaffa's data builders. test-data generation |
| Multi-platform dispatch, one API ◌ simulated |
The VM fan-out and sequential queue above are animations. | Native: one API drives iOS, Android and macOS; Replay automation covers Windows/WebView2, PWA, lockdown/kiosk and BEK with a retry queue. The AI layer only names a platform list — Jaffa owns scheduling. Cross-Platform Mobile · Replay Automation |
| Network partition / offline control ✓ mocked |
Playwright context.setOffline() — a real partition, locally. |
Use CDP network interception to cut and restore connectivity mid-scenario on desktop browsers, and Replay's fine-grained WebView control for the native app path. Chrome DevTools (CDP) · IBOM |
| Checkpoint screenshot capture ◌ simulated |
Checkpoint names exist; captures are simulated. | Free: Jaffa's HTML reports are self-contained with embedded screenshots. Add a named-checkpoint convention (Report.Checkpoint("item-answered")) so captures are keyed identically on every platform and version. Rich HTML Test Reports |
| Visual baseline compare across versions ◌ simulated |
The match-percentage matrix above is an animation. | jaffa.Vision matches on-screen regions the DOM can't address (page.Find(byte[])) and already puts reviewer images in the report. Add a baseline store keyed test × platform × checkpoint × app-version; diffs open a human review with an approve-as-new-baseline action. Visual Matching (jaffa.Vision) |
| Multi-role scenarios (admin + student) ✗ not in catalog yet |
Single-role only — the catalog has one session. | Native: Jaffa drives admin + student (and more) browsers side-by-side in a single test. Expose it to the agent as new catalogued capabilities (start_role_session, as_role) — the guardrail model is unchanged. Multi-Browser Testing |
| Perf / load variants of the same scenario ✗ out of scope |
Not attempted. | Jaffa runs the same tests from one machine to a cloud cluster and mixes assessment-taking, marking and admin flows in one perf run with live metrics — a future agent skill could emit a load profile alongside the functional test. Multi-Role Performance · Testing at Scale |
| Human approval gate + audit ✓ real |
Single-use draft ids behind /api/approve; agent has no store/run tools. |
Not a Jaffa feature by design — it stays in the AI layer. Pair the app's gate with the suite-repo pull request so approval is recorded where engineers already review. AI results remain advisory; Jaffa's deterministic run decides pass/fail. Track-2 doctrine |
| Failure triage feed ◌ partial |
Plain-English run summary after each execution. | Point the triage agent at Jaffa's machine-readable reports and CI logs to cluster red runs by root cause — the highest-value AI use on the suite they have today, before any re-architecture. Rich HTML Test Reports · CI logs |
✓ working in this POC (mocked where noted) · ◌ simulated/partial here, real feature exists in Jaffa · ✗ gap — needs new catalog capabilities or is future scope. Feature names in blue are from Jaffa's own README.
Build a test — no code
Assertions
Spec preview
Suites (stored via the builder)
Live pipeline · Spec → Agent → Validate → Approve → Store → Run → Matrix
Matrix — platform fan-out
⚠ No suite run yet. Approve & run a suite in the Builder tab, or replay a recorded flow above. Every cell but the Chromium one is SIMULATED — this POC always executes locally on Chromium.
Coverage of Janison Insights
Where these tests would live in Janison's own suite — and, more usefully, where they would not.
The areas and entities are Janison's, observed 27 July 2026 from jaffa.Insights.Tests
and the running product.
Areas
Each area is a division of Insights and of their test suite. Entities are the namespaces beneath it.