Check types

Seven deterministic checks and one judge. Every path resolves relative to the repository root (--repo), and every check accepts an optional name and timeout_s.

verify:
  - type: file            # a path exists (or does not, with exists: false)
    path: packages/core/src/ledger/chain.ts
  - type: grep            # a JS regex is present (min_count) or absent in a file or glob
    path: "packages/core/src/**/*.ts"
    pattern: "export function verifyChain"
    expect: present
  - type: symbol          # a TypeScript file exports a symbol (function, const, class, type, re-export)
    file: packages/core/src/index.ts
    export: parseSpecFile
  - type: schema          # a JSON or YAML file validates against a JSON Schema (2020-12)
    path: .byspec/last-run.json
    schema: packages/core/schema/run.schema.json
  - type: command         # a shell command exits as expected; optional regex or JSON assertions on stdout
    run: pnpm byspec parse specs/three.spec.md --format json
    expect_exit: 0
    expect_stdout_json:
      - path: count
        equals: 3
  - type: test            # runs `${config.test_command} ${pattern}`; passes on exit 0
    pattern: packages/core/test/parser.test.ts
  - type: http            # start a server, hit a loopback URL, assert on the response
    start: pnpm start
    ready: http://127.0.0.1:3000/health       # polled every 250 ms until 2xx, or error on timeout
    request:
      method: GET                             # GET (default), POST, PUT, PATCH, DELETE, HEAD
      url: http://127.0.0.1:3000/api/items
      headers: { Accept: application/json }
    expect_status: 200                        # default 200
    expect_body: '"items"'                    # optional regex against the body
    expect_json:                              # optional dot-path assertions, as for command
      - path: items.length
        equals: 0
  - type: judge           # LLM verdict on declared evidence only; runs after every check above passes
    rubric: |
      PASS only if every write to the ledger uses an append-mode handle.
      If the evidence is insufficient, answer UNVERIFIABLE.
    evidence:
      - type: file
        path: packages/core/src/ledger/append.ts
      - type: command
        run: pnpm vitest run packages/core/test/ledger.test.ts
      - type: check_output   # reuse the evidence of this criterion's own check at that index
        index: 1
    min_confidence: 0.8

http URLs (ready and request.url) must point at localhost, 127.0.0.1, or ::1; any other host is a schema error. The start process runs with the allowlisted environment in its own process group and is always stopped (SIGTERM, then SIGKILL) before the check returns, whatever the outcome. http, command, and test checks are never cached and run sequentially unless parallel: true.

Every check records evidence (matches, stdout/stderr, response bodies, ajv errors) under .byspec/evidence/, capped at judge.max_evidence_bytes with truncation recorded.

Source: README.md § Check types

Criteria outside the --changed or --only scope are SKIPPED and never touch the ledger.

Source: README.md § Verdicts

All checks execute with cwd = repo root (or check.cwd if set, which MUST resolve inside the repo root). All checks record evidence. All checks have a timeout (timeout_s, default from config run.check_timeout_s, 120). A timed-out check has status error and the criterion verdict is UNVERIFIABLE, not FAIL; the report says why.

file#

- type: file
  path: packages/core/src/ledger/chain.ts
  exists: true          # default true

Pass when existence matches. Evidence: none beyond the boolean.

grep#

- type: grep
  path: "packages/core/src/**/*.ts"   # glob allowed
  pattern: "export function verifyChain"   # JS regex source
  expect: present       # present | absent
  min_count: 1          # only with present

Evidence: matching file paths with line numbers, capped at 200 matches.

symbol#

- type: symbol
  file: packages/core/src/index.ts
  export: parseSpecFile

Uses ts-morph to confirm the file exports a symbol by that name (function, const, class, interface, type alias, or re-export). Evidence: the export declaration text, one line.

schema#

- type: schema
  path: .byspec/last-run.json
  schema: packages/core/schema/run.schema.json

Validates a JSON or YAML file against a JSON Schema (2020-12). Evidence: ajv error list, empty on pass.

command#

- type: command
  run: pnpm byspec parse fixtures/specs/three.spec.md --format json
  expect_exit: 0                # default 0
  expect_stdout: "\"count\":\\s*3"   # optional regex against full stdout; use "^$" to assert empty
  expect_stdout_json:           # optional; parse stdout as JSON, then assert
    - path: criteria.length     # dot path, supports .length
      equals: 3
  env: { BYSPEC_PROVIDER: mock }  # merged over an allowlisted base env
  parallel: false

Evidence: stdout and stderr, each capped at judge.max_evidence_bytes (default 20000) with truncated: true when cut, plus exit code and duration. The base env passed to child processes is an allowlist: PATH, HOME, NODE_OPTIONS, CI, GITHUB_*, BYSPEC_*, AWS_* plus anything in check.env. Anti-pattern: passing process.env through wholesale.

test#

- type: test
  pattern: packages/core/test/parser.test.ts   # optional; appended to config test_command

Runs ${config.test_command} ${pattern}. Pass on exit 0. Evidence: same as command, plus a parsed summary line when the output matches the vitest summary format.

Registry#

packages/core/src/checks/registry.ts exports a static Record<CheckType, CheckRunner>. Adding a type in v2 (http) means adding a file and a map entry. No dynamic loading.

Source: docs/byspec-implementation-spec.md § 8. Deterministic check types

- type: http
  start: pnpm start                     # optional; launched through the v1 executor, killed after the request
  ready: http://127.0.0.1:3000/health   # optional; polled every 250 ms until 2xx or timeout_s
  request:
    method: GET                         # default GET
    url: http://127.0.0.1:3000/api/items
    headers: { Accept: application/json }
    body: ""                            # string; sent as-is
  expect_status: 200                    # default 200
  expect_body: '"items"'                # optional regex against the body
  expect_json:                          # optional; same dot-path assertions as command.expect_stdout_json
    - path: items.length
      equals: 0
  timeout_s: 60

Rules: the start process runs with the v1 allowlisted env and cwd = repo root, in its own process group; it is killed (SIGTERM, then SIGKILL after 2 s) when the check ends, whatever the outcome. url and ready MUST be http:// or https:// on localhost, 127.0.0.1, or [::1]; any other host is a schema error (the check is for the code under test, not the internet, and the sandbox has no egress anyway). Evidence: check_result with { status, headers, body (capped) }, and the start process's stdout/stderr as command_output. --changed treats http checks like command checks (no paths, so in scope by default). Never cached.

Source: docs/byspec-v2-spec.md § 5. `http` check type

When the judge runs#

The judge runs for a criterion only when all three hold: every deterministic check has status pass; the criterion has at least one judge entry; --no-judge was not passed. When --no-judge is passed and a criterion with a judge entry has all deterministic checks passing, its verdict is UNVERIFIABLE with message judge disabled; deterministic-only evidence is not the full verification the author asked for. A criterion may declare at most one judge entry in v1 (schema-enforced).

judge check shape#

- type: judge
  rubric: |
    PASS only if the ledger write is append-only: no code path truncates,
    rewrites, or deletes entries. FAIL if any evidence shows in-place mutation.
  evidence:
    - type: file
      path: packages/core/src/ledger/append.ts
    - type: file
      path: packages/core/src/ledger/chain.ts
      lines: [1, 80]
    - type: command
      run: pnpm vitest run packages/core/test/ledger.test.ts
    - type: check_output
      index: 0            # reuse evidence from the criterion's own check at that index
  min_confidence: 0.7     # default from config

Evidence resolution (information hiding)#

packages/core/src/judge/evidence.ts is the only module that assembles judge input. It:

1. Resolves each declared evidence item to an Evidence record, reading files through core/fs.ts (root-jailed), running commands through the same executor as Section 8.5.
2. Caps each item at max_evidence_bytes and marks truncation.
3. Refuses any item type not in {file, command, check_output}.
4. Never includes: the git diff, the list of changed files, other criteria, other criteria's evidence, or the contents of any file not declared.

This module has a dedicated test (JDG-003) asserting that an undeclared file is not present in the assembled prompt even when it sits next to a declared one.

Prompt#

System prompt (fixed, in judge/prompt.ts): the judge is an independent verifier; it MUST decide only from the evidence provided; it MUST answer UNVERIFIABLE when the evidence is insufficient rather than guessing; it MUST NOT infer behavior from file names or comments alone. User message contains: the criterion (id, title, ears, pattern), the rubric, and the evidence items in order, each in a fenced block labeled with its kind and ref. The spec's prose is not included.

Output: a single JSON object matching the JudgeResult schema (minus token accounting), requested via the provider's structured-output mechanism when available (tool use on Bedrock Converse and the Anthropic API), else via instruction and parsed with one retry on schema failure. Temperature 0.

Verdict mapping#

• Judge PASS with confidence >= min_confidence → PASS.
• Judge PASS with lower confidence → UNVERIFIABLE, reason "low confidence".
• Judge FAIL at any confidence → FAIL.
• Judge UNVERIFIABLE, provider error, or unparseable output after retry → UNVERIFIABLE.

Caching#

Judge calls are cached in .byspec/cache/judge/<key>.json where key = sha256(rubric + ordered evidence sha256s + model id). Cache hits set cache_hit: true and cost nothing. --no-cache bypasses. Prompt caching at the provider level (Bedrock and Anthropic) is enabled for the system prompt block when the provider supports it.

Source: docs/byspec-implementation-spec.md § 9. The LLM judge