Evaluation & optimization

An LLM evaluation platform with prompt optimization.

Prove the new prompt is actually better. Build a dataset from real feedback or by hand, sweep it across prompt versions and models, read the result cell by cell — and let the optimizer draft the candidate rewrites before you move the production alias.

experiment.sh
# every id here is a UUID, not a version number
curl -X POST $ACRUX/experiments \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "dataset_id": "b7f1c0e2-…",
    "prompt_id": "4d9a7c18-…",
    "version_ids": ["e21c8a5f-…", "9c3b6d40-…"],
    "models": ["gpt-4o-mini"]
  }'
# → 201 { "id": … }  ← that is $EXP_ID

# start a run (async, returns 202 + run_id)
curl -X POST $ACRUX/experiments/$EXP_ID/runs \
  -H "Authorization: Bearer $API_KEY"

# poll until succeeded, then read the report
curl $ACRUX/runs/$RUN_ID/report \
  -H "Authorization: Bearer $API_KEY"
# → per-cell avgScore, passRate, deltaVsBaseline

# drill into one cell (label|model, encoded)
curl $ACRUX/runs/$RUN_ID/cells/v2%7Cgpt-4o-mini \
  -H "Authorization: Bearer $API_KEY"
# → per-example output, score, judge reason

What you get

Datasets, experiments, per-cell results and an optimizer.

Datasets, however you start

Examples can come from the feedback your team left on real traces, from rows you write by hand, or from live calls a standing rule already scored badly. Existing traffic helps; it is not a prerequisite.

Experiments as sweeps

An experiment is a dataset crossed with prompt versions and models. Ask "does the new version beat production on the cheaper model?" and get a grid instead of an opinion.

Per-cell results

Read every combination back individually — scores, outputs, and the judge reason — so a win in the aggregate cannot hide a regression on the cases you care about.

An optimizer that drafts the next version

Point an optimize run at a dataset and a model rewrites your live prompt — several candidates, each scored against the same cases as the version in production. Promoting one commits a real version and moves the alias.

Datasets

Three ways to fill a dataset.

An example is two things: the prompt variables to render, and the criteria a judge should check. Where those rows come from is up to you — a brand-new prompt with no traffic can still be evaluated today.

From feedback

Pick the traces your team marked wrong and pull them in. Each row brings the variables the call was made with, the comment as its criteria, and the session history behind it.

By hand

New dataset creates an empty one, and Add example writes a row: the variable fields, and what a good answer has to do. This is the path for a prompt that has no traffic yet.

From live scores

A standing evaluation rule collects the calls it scored below your threshold. One action turns that pile into a dataset of real failures, ready to run against.

A complaint reused word-for-word as a rubric grades correct answers badly, so criteria stays editable in place — rewording it is routine curation, not a rare correction.

Guide: evaluate a promptGuide: evaluate with conversation history

Offline and online

Evaluate before deployment, monitor after it.

The same judge runs in two places: on demand against a fixed dataset, and continuously against traffic that is already live.

Offline experiments

A grid of prompt versions and models over one dataset, run when you ask for it. This is the one that gates a promotion, because every variant sees identical inputs.

Online rules

A standing rule matches live LLM calls and a worker scores them shortly after the response returns. It is a measurement, not a gate — nothing is held back from your user while judging happens.

What judging costs

A rule samples 10% of matching calls and stops at 500 judge calls a day by default. Judge calls run through your own gateway keys, so a breached budget disables the rule and notifies the owners.

Only LLM calls are ever judged. Tool spans are recorded and visible, but a rule never scores one — so a rule cannot be used to grade a tool result directly.

Guide: score live traffic with an evaluation rule

Optimizer

Let a model draft the next prompt version.

The hardest failure is the one with no stack trace: the answer came back well formed, your code accepted it, and a human read it and said it was wrong. An optimize run turns that judgment into candidate rewrites and scores them the way a hand-written version would be scored.

What it reads

A dataset, and the version the chosen alias points at — production by default. You pick the model that writes the rewrites, and can swap the built-in instructions for a prompt of your own.

What it produces

Several candidate rewrites, each rendered against the same examples and judged with the same criteria, ranked in one report beside the version that is live.

What you do with it

Read the diff and the optimizer's own rationale, then promote the candidate that earned it. Promoting commits an ordinary numbered version and moves the alias — there is no separate optimized-prompt object.

A rewrite that drops or invents a {{ variable }} is discarded before it is ever scored, so a candidate you can promote is always one your code can still render.

Guide: improve a prompt from feedback

In the dashboard

Everything without writing a line of code.

The API and SDKs cover the automated path. For the day-to-day, the dashboard does the same work in the browser.

  • Build a dataset from feedback, or write the first examples by hand.
  • Configure a run across prompt versions and models in one form.
  • Watch the run progress as the worker processes each cell.
  • Compare outputs side by side, then drill into a single cell.
  • Start an optimize run from a dataset and review each candidate's diff.
  • Promote the winning version to production the moment you are convinced.
An AcruxCore run report comparing two variants on one model, with a leaderboard scoring v1 at 80 and production at 70, and a matrix marking the production row as the baseline.
A run report: the leaderboard, then the grid, with production marked as the baseline everything is measured against.

Go deeper

Documentation for this piece.

Guide: evaluate a promptGuide: improve a prompt from feedbackGuide: score live traffic with a ruleGuide: evaluate with conversation historyAPI: datasetsAPI: experiments & runs

The rest of the platform

It composes with the other five.

Each piece works on its own, and they get better together — a trace links back to a prompt version, a dataset is built from feedback.

PromptsA prompt management platform for versioned, templated message sets. Move a production alias between versions without redeploying your app.GatewayAn LLM gateway that speaks OpenAI to every provider you connect. Bring your own keys; get routing, fallbacks, cost and caching.TracingLLM observability for every call — spans for model, tokens, latency and cost. Or export your own spans over OpenTelemetry.ToolsLLM tool calling from a versioned catalog: functions bound to a prompt alias and handed to the model.AuditAn audit log of every recorded action — keys, members, gateway, secrets, prompts and tools — filtered by area, event or actor.

Build your first evaluation dataset.

Write two examples by hand or pull them from feedback, then run one prompt across two versions and compare. No credit card required.