Chapter 24: record-gates, pricing the truth before buying it

Chapter 15 introduced the release gates: four claims about Jev that only the real model can settle. This is the tool that asks it. And since asking costs money, it starts by telling you how much, before anything is spent.

Run without --send, it builds every request each gate's recording would send and prints, per gate, how many requests, their estimated input tokens, and their worst case at --price-per-mtok (default 0.042), every request billed three times exactly as jev.max_cost budgets it. Nothing leaves the machine.

buck2 run //tools/record-gates:record-gates -- --model jev-1.13.0 [--repeats 30]

Gate	What it asks
Layout parity	The same rows, one question each, sent both ways: row as state (a request per row) and row in instructions (every row a question in one request).
Unsure band	One question about one row, asked `--repeats` times (default 30), to measure the spread.
Label tree	The label-tree search's first round, lookahead included, with the "does any label fit" question it carries.
Ranking	`ORDER BY jev_prob … LIMIT` over two rows, one request each.

With --send it records all four, once, as replay fixtures in tests/fixtures/ (or --out-dir). It needs exactly one API key, TYPESAFE_API_KEY or --api-key-file, and --max-cost (default $0.01) becomes jev.max_cost for the runs that go through the extension.

flowchart LR
  subgraph direct["layout parity, unsure band: built here"]
    R1["main.rs builds the requests"] --> F1["jev_mock::forward"]
  end
  subgraph ext["label tree, ranking: through the extension"]
    P["a throwaway Postgres with the built extension"] --> F2["a jev-mock that forwards and records"]
  end
  F1 -->|"adds the key"| J["api.typesafe.ai"]
  F2 -->|"adds the key"| J
  F1 --> X["tests/fixtures/*.json"]
  F2 --> X

Aside: why layout parity is sent from here. The extension cannot send row in instructions at all. That layout waits on this very gate, and LayoutParity in postjevsql-core has no values until it passes (chapter 9). So the tool builds that request itself, straight from jev-protocol, to measure the thing the extension is not yet allowed to do.

Try it. Run the pricing line above: it needs no key and sends nothing. --send refuses before reading a key if any fixture already exists; --rerecord is the deliberate way to spend again.

For the people who maintain it

Path	What
src/	The tool. Chapter 25.
BUCK	`:record-gates`, compiled with the same `POSTJEVSQL_EXT`, `POSTJEVSQL_PG_MAJOR` and `POSTJEVSQL_POSTGRES_BIN` that `tests/BUCK` gives the tests.
Cargo.toml	Generated by `tools/cargo-gen` (chapter 20).

--model must be a versioned id: the gates are recorded against a pinned version.

← Previous: Chapter 23, third-party-notices/src/ · Up: tools · Next: Chapter 25, record-gates/src/ →