1# Chapter 9: postjevsql-core/src, six decisions and a gate
2
3Eight files, and the best way to read them is as answers to questions the
4extension asks itself on the way to sending a request.
5
6## 1. batch.rs: how does this row reach the model?
7
8The planner looks at the distinct questions asked about a row and chooses
9its *layout*. A row with two or more questions is sent as the request's
10state, with each question a branch: one request, the row billed once. A row
11with exactly one question *could* travel inside its question's instructions,
12beside other rows' questions in the same request, which would be far
13cheaper in requests. But nobody has measured whether Jev answers as well that
14way. So that layout waits on a release gate, and until then it cannot be
15built: its evidence type, `LayoutParity`, is an enum with no values, so no
16`Planner` can hold one, and every row is row as state.
17
18> **Aside.** An enum with no values is Rust's way of writing "this cannot
19> happen yet" so that the compiler, not a code reviewer, enforces it. There
20> is no `if` to forget and no flag to flip by accident. The day the gate
21> passes, someone gives `LayoutParity` a value, and every place that needed
22> one starts compiling.
23
24## 2. cache_key.rs: have we asked exactly this before?
25
26`sha256(model ‖ prompt version ‖ namespace ‖ layout ‖ state ‖ question)`,
27over the exact bytes of the request, each field length-prefixed so that no
28two different inputs run together into the same bytes. The question's id is
29left out, because the API says it "is not sent to the underlying model": two
30ids for one question are one judgment. The key is computed from the
31`Placement` the planner returned, so the layout in the key is the one sent.
32
33## 3. token_ratio.rs: how many tokens is this?
34
35Jev's tokenizer is unpublished, and the terms forbid working it out. So the
36ratio of characters to tokens is *learned*, per model, from what answers
37reported in `usage.input_tokens`, falling back to a conservative 2.6
38characters per token when nothing is known. A request costs 267 tokens of
39fixed overhead plus its characters over the ratio, and the worst case is
40three billed attempts, which is what `jev.max_cost` budgets.
41
42## 4. gcra.rs: may this request go now?
43
44The account's two rate limits (tokens per second, requests per minute) as
45GCRA: one atomic word per limit holding a *theoretical arrival time*. A
46request is admitted when that time is not too far in the future, and moves
47it on by its cost. One compare-and-swap loop per limit, where a token bucket
48would need two words and a lock. `Adaptive` halves the rate on a 429 and
49climbs back as answers arrive.
50
51## 5. label_tree.rs: which of ten thousand labels?
52
53A Choice takes at most 255 options. For more, arrange the labels as a tree,
54and ask one Choice per node, best-first. A node's score is the product of
55the probabilities down to it, which can only fall with depth, so a partial
56path bounds every leaf below it, and pruning on that bound is exact. Each
57round asks the three best unexpanded nodes (K = 3) in one request, with
58*lookahead* (their likeliest children's Choices, when the token budget
59allows) and an *order twin* (a close call asked again in reverse order, the
60two averaged). Neither changes the answer, only how fast it arrives.
61
62```mermaid
63flowchart TD
64  R["Root: Office · Care · Transport"] -->|"1.00"| O["Office: Engineering · Finance"]
65  R -->|"0.00"| C["Care"]
66  R -->|"0.00"| T["Transport"]
67  O -->|"1.00"| E["Engineering: Backend · Frontend"]
68  O -->|"0.00"| F["Finance"]
69  E -->|"1.00"| B["Backend ✓"]
70  E -->|"0.00"| FE["Frontend"]
71```
72
73(The recorded label-tree gate, from
74[tests/fixtures/label_tree.json](../../../tests/fixtures/label_tree.json):
75Ana, the backend engineer, placed in two requests. The first also carried
76Office's and Care's Choices ahead of time, as lookahead, and the "does any
77label fit" question, which came back 0.77. The second settled Backend, and
78asked the tree's other open nodes too, because the result names its
79runner-up exactly, and that takes knowing every leaf that could be second.)
80
81## 6. shortlist.rs: which of a thousand labels with no tree?
82
83Two rounds at any size: ⌈N/254⌉ Choices over even chunks of your list,
84labels only, all in one request, keeping the top 3 of each; then one Choice
85over those finalists, with their descriptions. Nothing is reordered, ever.
86The probability returned is among the finalists, which is exactly what was
87measured, and not a probability over all N, which nothing measured.
88
89## And the gate: does any label fit at all?
90
91A Choice's probabilities always add up to 1, so some label always wins, even
92when none is right. `gate.rs` builds the yes/no question both searches send
93beside their first round, "Does any of these labels fit?", whose answer does
94not depend on the others and can fall near zero. The `_full` SQL functions
95return it as `fit`. It never ranks labels: a Noul's probability and a
96Choice's are not comparable.
97
98> **Try it.** `cargo test -p postjevsql-core label_tree` runs the search's
99> tests, `exact_against_exhaustive` among them: every combination of
100> answers, searched both ways, and the best-first result must equal the
101> exhaustive one.
102
103## For the people who maintain it
104
105| File | What |
106| --- | --- |
107| [lib.rs](lib.rs) | The modules, and the crate's re-exports. |
108| [batch.rs](batch.rs) | `Planner`, `Planned`, `Placement`, and the uninhabited `LayoutParity`. |
109| [cache_key.rs](cache_key.rs) | `CacheKey`, `KeyScope`, `Layout`, `PROMPT_VERSION`. |
110| [token_ratio.rs](token_ratio.rs) | `TokenRatio` (learned, or `FALLBACK`), `Sample`, `REQUEST_TOKENS` (267) and `BILLED_ATTEMPTS` (3). |
111| [gcra.rs](gcra.rs) | `Limit`, `Adaptive`, `Limiter`, `Ceilings`, `Rate`, `Admission`. |
112| [label_tree.rs](label_tree.rs) | `Branch::from_paths`, `Tree`, `Search`, `Params` (K, `Tau`, `Lookahead`, `TwinBelow`, `Describe`), `Outcome`. |
113| [shortlist.rs](shortlist.rs) | `Shortlist`, `Search`, `Params` (`keep`, default 3), `CHUNK` (254), `Outcome`. |
114| [gate.rs](gate.rs) | The "does any label fit" Noul, built by `Tree::gate` and `Shortlist::gate`. |
115
116← Previous: [Chapter 8, postjevsql-core/](../) · Up: [postjevsql-core](../) · Next: [Chapter 10, postjevsql-sidecar/](../../postjevsql-sidecar/) →