Evalt
Evidence-bound AI cost optimization
Point Evalt at the prompt and model you use now. It tests cheaper models, prompt revisions, few-shot examples, and reasoning levels—then replaces your route only when a lower-cost configuration matches or beats the quality your approved cases measured.
answer = evalt.run(prompt, input, route="support", incumbent_model=CURRENT_MODEL)
We infer a draft success rule, ask for only the next examples that reduce uncertainty, reserve unseen examples, improve the prompt iteratively, and show the cheapest model that still clears your quality floor. One answer starts setup; it never counts as proof by itself.
Step 1 of 5
Start with the exact job, prompt, and model you need to improve.
Hard rules veto a candidate even when its average score rises.
Nothing in this replay preview.
Outputs, pass/fail decisions, judge reasons, prompt revisions, and model names will stream here.
Held-out verdict
Set a hard maintenance cap. Evalt queues a new tournament when traffic supplies enough fresh outcomes or the model catalog changes; a challenger is promoted only after it passes the frozen contract.
The exported report is machine-readable. Fail a build when the selected prompt drops below your floor.
evalt check evalt-result.json --min-pass-rate 0.90Measured only on this frozen replay fixture. This is product QA, not a claim about your prompt, demand, or future model versions.
The deliverable is evidence, not vibes
Original and winner run on the same held-out cases.
Repeated runs expose inconsistency in the target model.
Hard rules can veto an attractive average score.
Open the exact case, output, and failed requirement.
Live spend stays visible under a server-enforced cap.
Export the prompt, frozen contract, results, and model labels.
The production interface
Name the model you trust today. Evalt returns an answer immediately, learns from outcomes you accept or correct, and promotes a cheaper prompt/model/reasoning configuration only after it matches the incumbent on held-out cases. A separate capped test budget funds each comparison.
python -m pip install evalt-0.7.0-py3-none-any.whl
from evalt import Evalt
evalt = Evalt(api_key=OPENROUTER_API_KEY)
answer = evalt.run(
prompt,
ticket,
route="support-routing",
incumbent_model="openai/gpt-5-mini",
test_budget_usd="auto",
)
send(answer.content)
answer.accept() # or answer.correct(expected)
evalt.route_status("support-routing")One default, advanced controls when you need them
incumbent_model establishes the quality to preserve; Evalt finds the lowest-cost configuration that matches it.
Set target_accuracy with objective="lowest_cost_at_accuracy" when a fixed reliability gate is more useful than the incumbent.
Set price_usd with objective="best_within_price" for a new workload with no trusted baseline.
test_budget_usd="auto" is capped by max_test_budget_usd; it is never permission for unlimited background spend.
Cost comparisons use the 90th-percentile input and output lengths observed on the route.
Low, medium, and high effort compete as distinct configurations when a model supports them.
“100%” means 100% on the approved held-out suite—not a promise that unknown future inputs cannot fail.
Use it your way
The SDK is the adoption path. Pro is the business: encrypted history, CI gates, and automatic route maintenance after new models arrive. Managed inference is optional for teams that want one Evalt credential and one bill.
Run the durable Python router or this web workflow with your OpenRouter key. You pay the provider directly; Evalt adds no usage fee.
The subscription pays for continuous optimization. Use your own provider key, or fund optional managed inference at provider cost plus the current visible 20% service fee.
The free demo and BYOK route are available without subscribing.
Secure checkout is processed by Stripe for Jonathan Larson, the operator of Evalt.