Chapter 18: jev-client — the patient one

jev-client asks Jev a question and keeps at it: it retries what is worth retrying, gives each attempt its own time limit, resends a request that never left, and puts the request id on every error. It has no network and no clock of its own. It reaches the world through two small interfaces, called ports, which whoever uses it fills in.

The problem with "just call fetch"

Sending one HTTP request is easy. Sending it correctly when the network is flaky takes rules, and the rules have to be the same everywhere, or one project ends up paying for a question twice because it resent something the others knew not to. Here are the ones this crate keeps:

• A 429 or a 503 is worth another go; a 403 is not.
• A request that never reached the server can be resent at once, for free.
• A request that broke halfway may already have been billed, so resending it is a real retry, counted and backed off.
• Nothing waits forever: 10 seconds per attempt, 30 seconds for the lot.

Those rules are jev-protocol's retry policy. This crate is the loop that applies them, and only this crate has one.

Ports: the loop does not care which road it takes

flowchart LR
  C["Client::ask"] -->|"send one request"| T{{"Transport"}}
  C -->|"now, sleep, timeout, jitter"| R{{"Runtime"}}
  T --- H["jev-http: HttpsTransport"]
  T --- W["jev-worker: FetchTransport"]
  T --- P["postjevsql-pg: a Postgres backend"]
  T --- F["the tests: a scripted fake"]
  R --- HR["TokioRuntime"]
  R --- WR["WorkerRuntime"]
  R --- FR["a virtual clock"]

A Transport sends one HttpRequest and returns the whole HttpResponse, or a TransportError whose TransportErrorKind says what happened (NotSent, Connect, Interrupted, Tls, TooLarge, Config). It may also wait in admit before an attempt, for a rate limit shared with other senders, and it is told to distrust_connection after a timeout.

A Runtime supplies now, wall_clock, sleep, timeout and jitter. In the tests, sleep just moves a counter forward, so a case that waits 30 simulated seconds finishes in microseconds.

If you have written a JavaScript function that takes fetch as an argument so a test can pass in a fake, this is that idea, with the clock passed in as well.

Aside. The futures these traits return need not be Send. That is for postjevsql: its transport runs inside one Postgres backend, tied to one thread, and a Send bound would have shut it out.

One ask, start to finish

flowchart TD
  start["ask(state, questions)"] --> bytes["request_bytes"]
  bytes --> left{"budget left?"}
  left -->|"no"| timedout["TimedOut"]
  left -->|"yes"| admit["transport.admit<br/>(wait not charged to the budget)"]
  admit --> send["send, under min(10 s, budget left)"]
  send -->|"200"| parse{"Response::parse"}
  parse -->|"ok"| done["Answered"]
  parse -->|"wrong"| resp["Response error"]
  send -->|"NotSent, under 2 redials"| left
  send -->|"Tls / TooLarge / Config"| fail["Transport error"]
  send -->|"400 max_tokens_exceeded"| api["Api error, not resent"]
  send -->|"other status, Connect, Interrupted, timeout"| delay{"next_delay<br/>and still inside 30 s?"}
  delay -->|"no"| giveup["Api / Transport / TimedOut"]
  delay -->|"yes"| sleep["sleep, add x-typesafe-retry-count"]
  sleep --> left

A few details the picture leaves out:

• A redial is the same attempt sent again on a fresh connection: it does not count as an attempt, does not back off, and sends no retry-count header. Each ask allows two; a third NotSent is treated as a broken exchange and retried.
• A timed-out attempt makes the client call distrust_connection, so the next attempt does not reuse a connection that may be stuck.
• Every request carries Authorization: Bearer <key>, marked sensitive so it stays out of HPACK's dynamic table and out of Debug output, and Content-Type: application/json. A retry adds x-typesafe-retry-count, as both vendor SDKs do.
• An optional Observer hears each Event as it happens: Sending, Retrying (with cause, delay and request id), Redialing and Answered (with attempts and token usage). client.observed(observer) attaches one.

What comes back

Ok(Answered) holds the verified Response, the request_id, the number of attempts, and the answering response's raw headers and body, for a caller that wants something the parse does not read.

Err(ClientError) is one of Invalid (nothing was sent), Api (the API refused, after any retries), Transport, TimedOut, or Response (a 200 that does not answer the questions). Every variant that saw a response carries its request id, the only handle for a billing dispute. nothing_billed() is true only when nothing can have cost money: the request was never built, or its one attempt was refused outright. A spend guard uses it to decide whether to release what it held for the request.

Try it

cargo test -p jev-client

Fifteen tests, every one on virtual time: the server's retry-after-ms honoured to the millisecond, three 503s backing off 0.5 s then 1 s, a stalled attempt costing exactly 10 s, and three stalls spending exactly the 30-second budget. The scripted fake is in Chapter 18¾.

Using it takes a transport, a runtime, a pinned model and a key:

let client = jev_client::Client::new(transport, runtime, model, &key)?;
let answered = client.ask(&state, &questions).await?;

For the people who maintain it

In this folder

← Previous: Chapter 17½: inside jev-protocol/src · Up: jevcrates · Next: Chapter 18½: inside jev-client/src →