Flash Evals
FLASH EVALSSelf-hosted model evaluationflashevals.dev · AGPL-3.0-only · early alphaRUN 0412 · receipts-v32026-10-25 05:38

Pick your model on your data. Keep the receipts.

Flash Evals runs your own text, image, and audio examples through competing models, and keeps every input, output, score, latency, and cost on the same record. See which one wins, and exactly why.

$ git clone https://github.com/oogway-lab/flash-evals

Item 017 / 240illustrative run

A run sends every example in your dataset to every model you pick. Here is one receipt-extraction item going to three of them.

receipt_017.jpg
Reference labelmerchantCorner Grocerdate2026-09-14total41.09tax3.12line_items6
Responses, printed as they arrive
Bgoogle/gemini-2.5-flash-lite620 ms
  • merchantCorner Grocer
  • date2026-09-14
  • total41.90expected 41.09
  • tax3.12
  • line_items6
Fields 4/50.62 s$0.0003 est.
Copenai/gpt-4o-mini910 ms
  • merchantCorner Grocer
  • date09/14/2026expected 2026-09-14
  • total41.09
  • tax3.12
  • line_items6
Fields 4/50.91 s$0.0005 est.
Aanthropic/claude-sonnet-4.51840 ms
  • merchantCorner Grocer
  • date2026-09-14
  • total41.09
  • tax3.12
  • line_items6
Fields 5/51.84 s$0.0041 est.

Subtotal · 240 itemsillustrative

Every cell rolls up into a per-model leaderboard: quality, latency, and estimated cost side by side, with failures counted, not hidden.

Illustrative leaderboard for a 240-item receipt extraction run
CodeModelFieldsp50$/1kVerdict
Aanthropic/claude-sonnet-4.596.8%1.79 s4.12Most accurate
Bgoogle/gemini-2.5-flash-lite93.1%0.58 s0.31Fastest
Copenai/gpt-4o-mini91.4%0.88 s0.49
Dgoogle/gemma-3-27b-it 3 hard failures · invalid JSON86.2%2.31 s0.22Cheapest
Estimated spend for this run: $1.23 · 3 failed cells kept for review

On this billqty · item

Audio · item 004hi-en · 0:07

Speech-to-text runs score transcripts against your reference, compare up to six config variants, and check speaker turns where the provider supports them.

Reference

Mera OTP 4829 hai, please confirm karo.

Asoniox/stt-async-v5 · keyword boost

Mera OTP 4829 hai, please confirm karo.

Transcript 100%Speakers 2 / 2
Bsoniox/stt-async-v5 · no boost

Mera OTB 4829 hai, please confirm karo.

Transcript 85.7%Speakers 2 / 2
Copenai/gpt-4o-transcribe

Mera OTP 4829 hi, please confirm karo.

Transcript 71.4%Speakers not supported
Judge · OTP leakageFlagged in A, B, C

AGENT COPY

The built-in MCP server gives an agent the same datasets, prompts, runs, and review notes as the web app. Same API logic, same data.

  1. list_datasets3 datasets
  2. create_eval_runrun 0413 queued
  3. get_run_progress180 / 240 cells
  4. get_run_summaryleader: claude-sonnet-4.5
  5. list_run_cells12 cells below 0.6
  6. save_run_notenote saved
POST http://127.0.0.1:3001/mcp · Streamable HTTP

Run it yourselfabout ten minutes

Clone it, start PostgreSQL, seed the synthetic examples, and open localhost:3000. No account or provider key is needed to look around.

Node 24.21.0 · pnpm 12.6.0 · Docker Compose v2
  1. git clone https://github.com/oogway-lab/flash-evals
  2. cd flash-evals
  3. pnpm install --frozen-lockfile
  4. pnpm run setup:local
  5. docker compose up -d --wait postgres
  6. node scripts/build-packages.mjs
  7. pnpm run db:migrate
  8. pnpm run seed
  9. pnpm run dev
The seed resets application tables. Point it at a disposable database.

Add a provider key (OPENROUTER_API_KEY by default) to make real model calls. Runs, judges, and prompt tests can incur provider charges.

Terms

LicenseAGPL-3.0-onlyHostingYour machineModel callsYour keysCost figuresEstimatesStatusEarly alphaTOTAL DUE TO US0.00flashevals.devTHANK YOUfor evaluatingPlease keep this receipt
GitHub