- merchantCorner Grocer
- date2026-09-14
- total41.90expected 41.09
- tax3.12
- line_items6
Pick your model on your data. Keep the receipts.
Flash Evals runs your own text, image, and audio examples through competing models, and keeps every input, output, score, latency, and cost on the same record. See which one wins, and exactly why.
$ git clone https://github.com/oogway-lab/flash-evals Item 017 / 240illustrative run
A run sends every example in your dataset to every model you pick. Here is one receipt-extraction item going to three of them.
- merchantCorner Grocer
- date09/14/2026expected 2026-09-14
- total41.09
- tax3.12
- line_items6
- merchantCorner Grocer
- date2026-09-14
- total41.09
- tax3.12
- line_items6
Subtotal · 240 itemsillustrative
Every cell rolls up into a per-model leaderboard: quality, latency, and estimated cost side by side, with failures counted, not hidden.
| Code | Model | Fields | p50 | $/1k | Verdict |
|---|---|---|---|---|---|
| A | anthropic/ | 96.8% | 1.79 s | 4.12 | Most accurate |
| B | google/ | 93.1% | 0.58 s | 0.31 | Fastest |
| C | openai/ | 91.4% | 0.88 s | 0.49 | |
| D | google/ | 86.2% | 2.31 s | 0.22 | Cheapest |
| Estimated spend for this run: $1.23 · 3 failed cells kept for review | |||||
On this billqty · item
- DatasetsYour own text, images, and audio, with reference labels.
- PromptsTask instructions with a structured output schema.
- Field-level diffsEvery structured answer checked against its label.
- LLM judgesA model scores outputs against the rubric you write.
- Latency + costRecorded per cell. Costs are estimates, labeled as such.
- WorkflowsChain prompt steps into a graph when one prompt is not enough.
- 6STT variantsCompare up to six transcription configs per audio run.
- 88MCP toolsAgents work with the same datasets, runs, and notes.
- Review notesAnnotate the cases that need a human look.
- Result reuseAn exact repeat is served from your stored result, not re-billed.
Audio · item 004hi-en · 0:07
Speech-to-text runs score transcripts against your reference, compare up to six config variants, and check speaker turns where the provider supports them.
Mera OTP 4829 hai, please confirm karo.
Mera OTP 4829 hai, please confirm karo.
Mera OTB 4829 hai, please confirm karo.
Mera OTP 4829 hi, please confirm karo.
AGENT COPY
The built-in MCP server gives an agent the same datasets, prompts, runs, and review notes as the web app. Same API logic, same data.
- list_datasets3 datasets
- create_eval_runrun 0413 queued
- get_run_progress180 / 240 cells
- get_run_summaryleader: claude-sonnet-4.5
- list_run_cells12 cells below 0.6
- save_run_notenote saved
Run it yourselfabout ten minutes
Clone it, start PostgreSQL, seed the synthetic examples, and open localhost:3000. No account or provider key is needed to look around.
git clone https://github.com/oogway-lab/flash-evalscd flash-evalspnpm install --frozen-lockfilepnpm run setup:localdocker compose up -d --wait postgresnode scripts/build-packages.mjspnpm run db:migratepnpm run seedpnpm run dev
Add a provider key (OPENROUTER_API_KEY by default) to make real model calls. Runs, judges, and prompt tests can incur provider charges.