1# Chapter 24: record-gates, pricing the truth before buying it
2
3Chapter 15 introduced the release gates: four claims about Jev that only the
4real model can settle. This is the tool that asks it. And since asking costs
5money, it starts by telling you how much, before anything is spent.
6
7Run without `--send`, it builds every request each gate's recording would
8send and prints, per gate, how many requests, their estimated input tokens,
9and their worst case at `--price-per-mtok` (default 0.042), every request
10billed three times exactly as `jev.max_cost` budgets it. Nothing leaves the
11machine.
12
13```
14buck2 run //tools/record-gates:record-gates -- --model jev-1.13.0 [--repeats 30]
15```
16
17| Gate | What it asks |
18| --- | --- |
19| Layout parity | The same rows, one question each, sent both ways: row as state (a request per row) and row in instructions (every row a question in one request). |
20| Unsure band | One question about one row, asked `--repeats` times (default 30), to measure the spread. |
21| Label tree | The label-tree search's first round, lookahead included, with the "does any label fit" question it carries. |
22| Ranking | `ORDER BY jev_prob … LIMIT` over two rows, one request each. |
23
24With `--send` it records all four, once, as replay fixtures in
25`tests/fixtures/` (or `--out-dir`). It needs exactly one API key,
26`TYPESAFE_API_KEY` or `--api-key-file`, and `--max-cost` (default $0.01)
27becomes `jev.max_cost` for the runs that go through the extension.
28
29```mermaid
30flowchart LR
31  subgraph direct["layout parity, unsure band: built here"]
32    R1["main.rs builds the requests"] --> F1["jev_mock::forward"]
33  end
34  subgraph ext["label tree, ranking: through the extension"]
35    P["a throwaway Postgres with the built extension"] --> F2["a jev-mock that forwards and records"]
36  end
37  F1 -->|"adds the key"| J["api.typesafe.ai"]
38  F2 -->|"adds the key"| J
39  F1 --> X["tests/fixtures/*.json"]
40  F2 --> X
41```
42
43> **Aside: why layout parity is sent from here.** The extension cannot send
44> row in instructions at all. That layout waits on this very gate, and
45> `LayoutParity` in postjevsql-core has no values until it passes (chapter
46> 9). So the tool builds that request itself, straight from jev-protocol, to
47> measure the thing the extension is not yet allowed to do.
48
49> **Try it.** Run the pricing line above: it needs no key and sends
50> nothing. `--send` refuses before reading a key if any fixture already
51> exists; `--rerecord` is the deliberate way to spend again.
52
53## For the people who maintain it
54
55| Path | What |
56| --- | --- |
57| [src/](src/) | The tool. Chapter 25. |
58| [BUCK](BUCK) | `:record-gates`, compiled with the same `POSTJEVSQL_EXT`, `POSTJEVSQL_PG_MAJOR` and `POSTJEVSQL_POSTGRES_BIN` that `tests/BUCK` gives the tests. |
59| [Cargo.toml](Cargo.toml) | Generated by `tools/cargo-gen` (chapter 20). |
60
61`--model` must be a versioned id: the gates are recorded against a pinned
62version.
63
64← Previous: [Chapter 23, third-party-notices/src/](../third-party-notices/src/) · Up: [tools](../) · Next: [Chapter 25, record-gates/src/](src/) →