This post compares ten Braintrust alternatives on eval features, license, self-hosting and price. Tools that replace the whole Braintrust platform come first, then tools that replace only its evals.

The short answer

• If you need the whole platform, not only evals: AcruxCore. Like Braintrust, it has prompt versions, a gateway, tracing and evals. Unlike Braintrust, it is Apache-2.0 and self-hosts for free on Postgres and Redis.
• If you already run MLflow: MLflow covers the same four parts in one server.
• If you only need evals: Opik or LangWatch. Both have online scoring, review queues, simulated conversations and a prompt optimizer, under Apache-2.0.
• Largest open-source community: Langfuse, with 36,135 GitHub stars. No eval feature needs Langfuse's paid enterprise license.
• Evals as tests in CI, with no server: promptfoo or DeepEval.
• A hosted platform charged per seat: LangSmith, at $39 per seat a month.

Braintrust competitors at a glance​

License, hosting and price

A span is one step inside a trace. Each vendor bills in its own unit. A Langfuse unit is one trace, span or score, and LangWatch counts events.

Eval features in each tool's free version

Every tool here has datasets, experiments you can compare, and scorers you write in code. Online scoring means a rule scores a sample of production calls as they happen. A review queue sends chosen traces to people who grade them.

Why look for Braintrust alternatives?​

• Braintrust has no plan between $0 and $249 a month. Pro then charges $3 for each GB over 5 GB and $1.50 for each 1,000 scores over 50,000.
• There is no hard spending limit. Braintrust's billing FAQ says so directly. You can set spend alerts, but they do not stop usage.
• Self-hosting needs an Enterprise contract. Even then, Braintrust hosts the web UI and the sign-in. Only the data plane, which stores your logs and serves the API, runs in your cloud. The data plane needs a license key.
• Braintrust stores your provider keys. In May 2026, Braintrust confirmed unauthorized access to one of its AWS accounts, which held the provider API keys customers had saved. Braintrust asked every customer to rotate those keys, and said one customer was affected.
• The free plan changed. On 16 March 2026, the Free plan became Starter. Organizations that signed up before that date keep some features that new Starter organizations lack, such as full role-based access.

Sources: pricing, billing FAQ, self-hosting, architecture, TechCrunch on the breach, Starter plan.

Braintrust pricing in 2026​

Every plan has unlimited users. Startups at Series A or earlier can get Pro free for 6 to 12 months.

How we picked and ranked​

• The tool does Braintrust's core job: you run a dataset through your app and score the results.
• The tool is active. Each one had a release or a commit in October 2026.
• Braintrust is a platform, so tools that replace the whole platform rank first. Braintrust versions prompts, logs production traces, runs a gateway and runs evals. Open-source tools with all four parts come first, then open-source tools focused on evals. Source-available tools, code-only libraries and closed hosted products follow, in that order.
• Inside each group, a tool with more eval features in its free version ranks higher.
• Every fact comes from the vendor: the pricing page, the docs, the license file or the GitHub API, read on 21 October 2026. We do not use other comparison lists as sources.

Open-source Braintrust alternatives with a gateway​

These two tools replace all four parts of Braintrust: prompt versions, a gateway, tracing and evals. Both are Apache-2.0 and free to run on your own servers.

1. AcruxCore​

AcruxCore vs Braintrust: AcruxCore has the same four parts as Braintrust: prompt versions, a gateway, tracing and evals. AcruxCore is Apache-2.0 and self-hosts on Postgres and Redis, while Braintrust self-hosts only on Enterprise.

Best for: teams that want to replace the whole Braintrust platform, not only the evals, and run it on their own servers.

Where AcruxCore differs

• The gateway refuses a call that would go over a spend budget. It also limits requests and tokens per key. Braintrust has spend alerts that do not stop usage.
• When you self-host AcruxCore, your provider keys stay in your own database.
• Roles, the audit log and prompt aliases such as production are free. Braintrust puts roles on Pro and audit logs on Enterprise.
• An experiment runs several prompt versions on several models against one dataset. A compare page then shows which examples improved and which got worse.
• Your team can rate a reply thumbs down and leave a comment. A dataset built from those ratings uses each comment as that example's grading criteria.
• An optimize run reads those rated replies and rewrites your production prompt. AcruxCore then scores each rewrite against the current production version.
• Online rules score a sample of live calls. A rule alerts the team's owners when its mean score for the day falls below a threshold you set.

Limits of AcruxCore

• If evals are all you need, Opik and LangWatch have more eval features: a metric library, review queues and a conversation simulator. AcruxCore has one built-in judge, and you can write your own judge prompt.
• AcruxCore has no CI integration.
• The repository went public in August 2026 and has 11 GitHub stars.

Our comparison: AcruxCore vs Braintrust. The eval guides start at Evaluate a prompt against a dataset.

Sources: license, compose file, pricing.

2. MLflow​

MLflow vs Braintrust: MLflow also has prompt versions, a gateway, tracing and evals, in one Apache-2.0 server from the Linux Foundation.

Best for: teams that already run MLflow for classic machine learning.

Where MLflow differs

• MLflow ships judges for correctness, safety, retrieval and tool calls. It can also tune a judge to agree with your own labels.
• MLflow simulates conversations and scores each session.
• The gateway rejects calls over a USD budget. Braintrust has spend alerts only.
• Tests written with pytest run your evals in CI.

Limits of MLflow

• Online scoring runs LLM judges only, not code scorers.
• Review queues are experimental. Some kinds of review-queue input work only on Databricks.

Our comparison: AcruxCore vs MLflow.

Sources: license, judges, online evaluation, review queues, simulation.

Open-source Braintrust alternatives for evals​

These three tools replace Braintrust's eval work. None of them has a production gateway in its free version. The core of each uses a license approved by the Open Source Initiative, and each is free to self-host.

3. Opik​

Opik vs Braintrust: Opik, from Comet, has every eval feature in the feature table at the top of this post, in its free self-hosted version. Its cloud plan costs $19 a month, against $249 for Braintrust Pro.

Best for: teams that want Braintrust's eval features on their own servers.

Where Opik differs

• Opik ships 43 built-in metrics. Among them are 19 LLM judges, such as hallucination, G-Eval and agent task completion.
• Online rules score live traces, including whole conversations.
• Opik simulates a user, so you can test an agent over many turns.
• The optimizer package rewrites prompts and agents for you, with six algorithms, including GEPA and an evolutionary one.

Limits of Opik

• Self-hosted Opik has no user management, so everyone who can reach it sees everything.
• A self-hosted install runs MySQL, Redis, ClickHouse, ZooKeeper and MinIO.

Our comparison: AcruxCore vs Opik.

Sources: repository, metrics, online rules, multi-turn, self-hosting, pricing.

4. LangWatch​

LangWatch vs Braintrust: LangWatch is an Apache-2.0 platform built around testing agents with simulated users, and it self-hosts for free.

Best for: teams that ship chat or voice agents and want to test whole conversations.

Where LangWatch differs

• Scenario, its testing library, tests your agent against a simulated user. It also tests voice agents and runs red-team attacks.
• LangWatch optimizes prompts with DSPy and scores each new version with Scenario tests.
• Review queues and online monitors are in the open-source version. The enterprise folder holds SSO, SCIM and billing.

Limits of LangWatch

• LangWatch has 4,986 GitHub stars. Opik has 22,798.
• CI support is an SDK call that exits with code 1 when an evaluation fails. There is no GitHub Action.

Sources: license, enterprise folder, agent testing, CI, pricing.

5. Langfuse​

Langfuse vs Braintrust: Langfuse is the open-source LLM platform with the most GitHub stars, 36,135. Its cloud plan costs $29 a month, against $249 for Braintrust Pro.

Best for: teams that want tracing, prompts and evals in one open-source tool with a large community.

Where Langfuse differs

• LLM judges, review queues, datasets, the playground and experiments are all free when self-hosted. The enterprise license covers admin features, such as audit logs and SCIM.
• A GitHub Action runs an experiment on each pull request.
• Langfuse Core keeps data 90 days. Braintrust Pro keeps it 30 days.

Limits of Langfuse

• Langfuse has no conversation simulator and no prompt optimizer.
• A self-hosted Langfuse runs Python code scorers only through AWS Lambda. Without Lambda, it runs TypeScript scorers only.

ClickHouse acquired Langfuse in January 2026, and the core is still MIT.

Our comparison: AcruxCore vs Langfuse. For more options starting from Langfuse, see 8 Langfuse alternatives.

Sources: license, self-host pricing, code evaluators, CI, pricing.

Source-available: Arize Phoenix​

6. Arize Phoenix​

Phoenix vs Braintrust: Phoenix self-hosts in one container with no feature limits. Its license, the Elastic License 2.0, is not approved by the Open Source Initiative.

Best for: engineers who want free self-hosted evals on OpenTelemetry, with the fewest parts to run.

Where Phoenix differs

• Phoenix ships judges for faithfulness, hallucination, tool selection and refusals.
• The server runs Python and TypeScript code scorers in a sandbox.
• Phoenix's docs say self-hosting has "no feature gates". The license forbids selling Phoenix to others as a hosted service.

Limits of Phoenix

• Phoenix has no review queues, but you can annotate a single trace by hand.
• Online scoring on the server is not released yet, and its GitHub issues are still open.

Our comparison: AcruxCore vs Phoenix.

Sources: license, license page, server evals, review queues issue, pricing.

Eval libraries that run in CI​

These two tools run evals as code, with no server to keep running. Neither one scores production traffic in its free version.

7. DeepEval and Confident AI​

DeepEval vs Braintrust: DeepEval is an Apache-2.0 Python library for evals. Confident AI is its closed, hosted platform, priced below Braintrust Pro.

Best for: Python teams that want evals as pytest tests.

Where DeepEval differs

• DeepEval's metrics cover RAG, agents, conversations, voice, MCP and safety.
• G-Eval and DAG, two judge builders in DeepEval, let you write your own judge.
• deepeval test run runs your evals as pytest tests in CI.
• The library simulates conversations and generates test cases for you.
• DeepEval optimizes prompts with four algorithms, such as GEPA and MIPROv2.

Limits of DeepEval

• The open-source library has no web UI. Side-by-side experiments, online scoring and review queues need Confident AI.
• Self-hosting Confident AI needs an Enterprise plan and a license key.

Sources: repository, metrics, CI, Confident AI self-hosting, pricing.

8. promptfoo​

promptfoo vs Braintrust: promptfoo is an MIT command-line tool that runs evals and red-team tests from a config file. OpenAI agreed to acquire it in March 2026.

Best for: engineers who want eval and security tests to run in CI, with no server.

Where promptfoo differs

• A matrix view shows every prompt and model side by side, for every test case.
• Red teaming is its main focus, with attacks mapped to OWASP and the EU AI Act.
• A GitHub Action runs the evals on each pull request.
• promptfoo optimize rewrites prompts, and a simulated user tests multi-turn chats.

Limits of promptfoo

• promptfoo tests before you ship. It does not score production traffic.
• Its self-hosted sharing server has no sign-in. The promptfoo docs say it is not recommended for production.

Sources: license, OpenAI acquisition, GitHub Action, self-hosting, pricing.

Closed-source alternatives​

LangSmith and W&B Weave publish no server code. Self-hosting either one needs an enterprise contract.

9. LangSmith​

LangSmith vs Braintrust: LangSmith, from the LangChain team, has datasets, judges, review queues, online scoring and multi-turn simulation. It charges per seat, where Braintrust charges a flat price.

Best for: apps built on LangChain or LangGraph.

Where LangSmith differs

• LangSmith scores the steps an agent took and simulates multi-turn chats.
• LangSmith has review queues, online judges, and test runners for pytest and Vitest.
• Plus costs $39 per seat, so a team of six or fewer pays less than Braintrust Pro, before trace charges.

Limits of LangSmith

• LangSmith keeps base traces 14 days. Keeping a trace longer costs extra.
• An online scoring rule moves every trace it matches to the longer, paid retention.

Our comparison: AcruxCore vs LangSmith. For more options starting from LangSmith, see 8 LangSmith alternatives.

Sources: pricing, self-hosting, online rules, multi-turn simulation.

10. W&B Weave​

Weave vs Braintrust: Weave, from Weights & Biases, has review queues, monitors and leaderboards. Its Pro plan starts at $60 a month, against $249 for Braintrust Pro.

Best for: teams that already track training runs in Weights & Biases.

Where Weave differs

• Leaderboards rank models and prompts across many evaluations.
• Monitors run LLM judges on a sample of live calls.
• Weave keeps LLM traces in the same W&B account as your training runs and model registry.

Limits of Weave

• Pro includes 1.5 GB of trace data a month, then $0.10 per MB. Eval runs that log full inputs and outputs count against that 1.5 GB.
• Weave has no built-in prompt optimizer.

Sources: pricing, self-managed Weave, comparing evaluations, monitors.

Tools other lists still recommend​

Older lists of Braintrust alternatives still include these tools. Their status on 21 October 2026:

• Galileo was acquired by Cisco and is now sold as Splunk Agent Observability. (source)
• Humanloop shut down on 8 September 2025, after its team joined Anthropic. (source)
• Patronus AI now calls itself a research lab for training data and simulated environments. Its pricing page shows no prices. (source)
• HoneyHive has a free plan and Enterprise, with no public paid plan in between. Self-hosting needs Enterprise. (source)
• Future AGI went public on GitHub in April 2026 under Apache-2.0, with an enterprise folder under its own license. (source)

Which Braintrust alternative should you pick?​

Start at the top and stop at the first question you answer yes to.

Best open-source Braintrust alternative​

• AcruxCore if you need the whole platform: prompts, a gateway, tracing and evals, free to self-host.
• Opik if you only need evals. It has the most eval features of any free self-hosted tool.
• Langfuse for the largest community, with every eval feature free.

Best self-hosted Braintrust alternative for a small team​

• AcruxCore for one app on Postgres and Redis, with a gateway that enforces spend budgets.
• Arize Phoenix for evals only, with the fewest parts to run: one container on SQLite.

If you test agents over many turns​

• LangWatch and Opik test your agent against a simulated user.
• MLflow simulates conversations and scores each session.

If evals should run as tests in CI​

• promptfoo for a config file, a GitHub Action and red teaming.
• DeepEval for pytest and metrics for RAG, agents and conversations.

If you want to stay hosted​

• LangSmith has review queues, online judges and simulation. It costs less than Braintrust Pro for teams of six or fewer.
• Opik Pro costs $19 a month, against $249 for Braintrust Pro.

FAQ​

Is Braintrust open source?​

No. The Braintrust server is closed source. The SDKs are Apache-2.0, and the autoevals scorer library is MIT. (autoevals)

Can you self-host Braintrust?​

Only on the Enterprise plan. The data plane runs in your AWS, GCP or Azure account and needs a license key. Braintrust still hosts the web UI and sign-in. (self-hosting)

How much does Braintrust cost?​

Starter is free, with 1 GB of data and 10,000 scores a month. Pro costs $249 a month. Enterprise has a custom price. (pricing)

Was Braintrust breached?​

In May 2026, Braintrust confirmed unauthorized access to one of its AWS accounts. It asked every customer to rotate the AI provider keys stored in Braintrust, and said one customer was affected. (TechCrunch)

Does Braintrust have spending limits?​

No. Braintrust has spend alerts, but its billing FAQ says it has no hard limits that stop usage. (billing FAQ)

Related posts​

• AcruxCore vs Braintrust: the same prompt and tool run on both, with screenshots.
• 8 Langfuse alternatives: the same comparison, starting from Langfuse.
• 8 LangSmith alternatives: the same comparison, starting from LangSmith.
• 10 LLM observability tools tested hands-on: screenshots of each tool.
• Self-hosting LLM observability: seven open-source tools measured: memory, disk and trace storage for each stack on one machine.

Agenta is an open-source platform from Agentatech UG. Its README describes a workspace where you build agents by chatting with them. You then talk to those agents in Slack, Telegram or WhatsApp. Behind the agent pages, Agenta also ships a prompt registry, a playground, evaluation, annotation queues and tracing. Agenta documents those parts in its v1.0 docs, which it marks as no longer actively maintained. Those parts still ship in version 0.122.7, the version we ran. This post compares those parts with AcruxCore, because those are the parts the two products share.

We ran Agenta 0.122.7 on one machine with its own Docker Compose file. The prompt, the customer message and the model were the same on both sides. Each section below shows what we saw on Agenta, then what AcruxCore does with the same prompt.

At a glance​

Evaluation​

Agenta's source defines 15 built-in evaluator types. Eleven of them are rules that need no model: exact match, starts with, ends with, contains, contains any, contains all, regex, contains JSON, JSON diff, Levenshtein distance and similarity match. Semantic similarity compares embeddings. A webhook test sends the output to your own URL for a score. A custom code evaluator runs your code inside Agenta. The fifteenth type is an LLM judge.

We created an LLM judge from the evaluators page. The judge prompt is editable and ships with a grading prompt already written. A test panel on the right runs the judge against an app before you save it.

Then we started an auto evaluation from the evaluations page. The wizard took the app, the revision, a test set and the evaluators. We picked revision v4 of vip-support-triage, a three-row test set called vip-triage-cases, and the judge. The run finished in 2.37 seconds, and the judge's average score was 66.67%.

The same page offers four kinds of evaluation. Auto runs a test set through evaluators. Human collects ratings from people. Live scores production traffic. SDK runs an evaluation from your own code and reports it back.

AcruxCore covers three of the four. An experiment runs a stored prompt over a dataset on the server and scores each answer with an LLM judge. A local experiment runs your own app and your own code scorers through the SDK, and AcruxCore stores the scores. An evaluation rule samples live traces and scores them with a judge. The difference is the server-side rule library. AcruxCore scores on the server with an LLM judge only. A regex or an exact match has to be written as a code scorer in your own process.

Prompt versions and deployment​

Agenta stores a prompt as an app with variants, and each variant has versions. The commit dialog shows what changed before you save. The dialog also lets you save as a new version or as a new variant, and deploy in the same step. The deploy list has three environments by default: production, staging and development.

A registry page compares two versions as JSON. The diff covers the whole stored config: the messages, the template format, the list of input keys and the model settings.

Each prompt also has an overview page. It charts requests, latency, cost and tokens for that prompt, and shows which version each environment runs.

AcruxCore has one line of versions per prompt and no variants. Each version is immutable. An alias points at a version, and an alias can have any name, so canary or eu-prod works as well as production. The diff tab compares the template text of any two versions.

Jinja2 templates​

Both products render Jinja2. In Agenta you set the prompt syntax to Jinja2, and the services container renders it in a sandbox. In AcruxCore every prompt is a template.

The difference is how each one finds the variables. Agenta's editor read our template and listed two inputs: company, and an object called ticket with id and title fields. The editor missed is_vip and tickets, the two variables that appear only inside {% if %} and {% for %}. The editor also listed ticket, the loop variable that the template sets itself.

The playground has no field for the missing variables, so both {% else %} branches ran. The model told a VIP customer with two open tickets that it could see no open tickets.

The stored list of input keys matters outside the playground too. Agenta's invoke endpoint checks a request against that list. A call that sent is_vip and tickets came back with HTTP 400 and "Invalid inputs: Expected '['company', 'customer_message', 'ticket']'". The call worked after we edited input_keys by hand in the JSON view and committed a new version.

AcruxCore builds the variable list from the template on every commit. The list holds every name the template reads, inside a condition or a loop as well. A name that the template sets itself, such as a loop variable, stays off the list. The playground then shows a field for each variable, and the render endpoint checks a request against the same list.

Where model calls run​

Agenta can serve a stored prompt as an API. A POST to /services/chat/v0/invoke names the app and the environment. Agenta then renders the prompt, calls the model and returns the answer. Each prompt can also set fallback models and a retry policy. So for calls through the invoke endpoint, Agenta is in the request path.

The invoke service has no spend cap, no response cache and no virtual keys. We searched Agenta's source for each one. Budgets exist only as API rate limits. A cache exists only for tool-catalog search. Budget-capped keys exist only in the Enterprise credit system for Agenta Cloud. Agenta also has a separate LLM gateway. Its purpose is to keep provider keys out of the agent sandbox, and it is off by default.

AcruxCore's gateway is in the request path for every call. It checks a budget before it calls the provider and returns 402 BUDGET_EXCEEDED when a call would go over it. It rate-limits each virtual key and can answer from its cache. A gateway call can name a stored prompt and its alias instead of sending messages, so AcruxCore also serves a stored prompt in one call. Fallback models are set per gateway model rather than per prompt.

Latency overhead, measured​

We timed one fixed call five ways: "Reply with the single word: pong." on openai/gpt-4o-mini, with max_tokens 5. Every path ends at OpenRouter, so the model's own time is the same on each path. The baseline is a direct HTTPS call to OpenRouter. The other four paths are these:

• Agenta traced SDK — the same call from the OpenAI client, traced by ag.init() and the OpenTelemetry instrumentor.
• Agenta invoke — one POST to /services/chat/v0/invoke. Agenta fetches the deployed prompt, renders it and calls the model.
• AcruxCore gateway — one POST to the gateway, which calls OpenRouter and writes a trace.
• AcruxCore render + gateway — a render call for the stored prompt, then the gateway call. This is two HTTP calls; the gateway's prompt reference would make it one.

Each run is 100 rounds after 3 warm-up rounds. The order of the paths rotates every round, so a slow moment on the network lands on every path equally. We ran it three times. Both platforms ran on the same machine.

The brackets are 95% bootstrap intervals for the gap in medians. All 12 intervals cross zero, so this benchmark cannot tell any path apart from a direct call, or from another path. OpenRouter's own variation decided the result. The direct call's median moved between 1,018 and 1,166 ms across the three runs, and its p95 reached 5.8 seconds in run 2. Every one of the 1,500 measured calls succeeded.

The AcruxCore gateway's point gap was positive in all three runs. Its overhead against a direct OpenAI call, measured on a quieter baseline, is in the gateway overhead post.

The raw samples for all three runs are committed, and summarize.py recomputes every number in the table from them.

Full script: latency_bench.py.

Tracing​

A call through Agenta's invoke endpoint traces itself. The trace has a chat_v0 workflow span with a litellm_client child, and shows inputs, outputs, tokens and cost. Buttons on the trace open it in the playground, add it to a test set, annotate it, or send it to an annotation queue.

A call from your own code needs more setup. Agenta's Python SDK traces through OpenTelemetry. You call ag.init(), instrument the OpenAI client, and decorate your function. The trace is linked to the prompt only after a store_refs call. Without it, the trace page showed "No references found".

ag.init()
OpenAIInstrumentor().instrument()

@ag.instrument()
def triage(variables):
    config = ag.ConfigManager.get_from_registry(app_slug=APP_SLUG, environment_slug="production")
    prompt = PromptTemplate(**config["prompt"]).format(**variables)
    response = client.chat.completions.create(**prompt.to_openai_kwargs())
    ag.tracing.store_refs({"application.slug": APP_SLUG, "environment.slug": "production"})
    return response.choices[0].message.content

In AcruxCore the gateway writes the trace on the server, so the script has no tracing code. The trace links to the prompt version that produced it.

async with AcruxCore(api_key=KEY, base_url=BASE_URL) as hub:
    rendered = await hub.prompts.render("vip-support-triage", "production", VARIABLES)
    result = await hub.gateway.chat(rendered.model, rendered.messages, prompt_version_id=rendered.version_id)

Both scripts are committed and were run before this post linked them: ag_trace_run.py and acx_sdk_run.py.

Tools​

Agenta has two kinds of tool. A prompt can carry tool schemas, typed in the playground with the "+ Tool" button and saved with the prompt version. Agents get a tool catalog of third-party actions from Composio, plus MCP servers that you connect by URL. Agenta runs those actions on the server, and a self-hosted install needs a Composio API key for them. There is no page where you define your own tool and keep versions of it.

AcruxCore's Tool Catalog holds your own tools. Each tool has versions. The gateway runs the tool, and a page measures call volume, error rate and latency per tool.

Both products also ship their own MCP server, for a coding agent such as Claude Code. Agenta's server searches Agenta's documentation, and none of its tools reads your prompts, traces or evaluations. AcruxCore's server can commit a prompt version, promote it, run it through the gateway and read the resulting trace.

What Agenta does that AcruxCore doesn't​

The built-in evaluators have their own section above. Four more Agenta features have no match in AcruxCore.

Annotation queues. A trace can go into a named queue. A reviewer works through the queue one item at a time, answers the questions the queue defines, and moves to the next item. AcruxCore lets a person score a trace, but it has no queue, no assignment and no progress tracking.

An agent builder. The home page asks "What do you want to build?" and turns a plain-language description into an agent. It offers 29 templates across engineering, support, sales, knowledge and ops. Agents run on a schedule or on an event, and can answer in Slack. AcruxCore has no agent builder, no schedules and no chat channels.

Webhooks on deploy. A webhook fires when a configuration is deployed. It can post to your own URL with an HMAC signature, or start a GitHub Actions workflow through a repository or workflow dispatch. AcruxCore sends alerts by email only.

Variants. Agenta keeps several branches of one prompt side by side, and each branch has its own versions. AcruxCore keeps one line of versions per prompt.

Is AcruxCore an Agenta alternative?​

For Agenta's agent workspace, no. AcruxCore has no agent builder, no schedules and no chat channels. For evaluation, only in part. AcruxCore runs LLM judges on the server and code scorers in your own process, but it has no rule library on the server and no annotation queue.

For prompts that run in production, yes. AcruxCore finds every variable in a Jinja2 template without manual edits. It keeps versions of your own tools and runs them. Its gateway checks a budget and a rate limit before a call reaches the provider. Agenta's invoke endpoint has no spend cap and no cache, and Agenta has no catalog for your own tools.

Verdict​

Agenta wins the evaluation rows. Its evaluators, annotation queues and webhooks on deploy have no equivalent in AcruxCore. AcruxCore wins the rows about the call itself: a gateway that can refuse a call over budget, a cache, templates whose variables are found without manual edits, and tools that are versioned and run by the platform. The latency benchmark could not tell the two apart: on OpenRouter, every interval crossed zero.

To choose, start at the top and stop at the first question you answer yes to.

License, pricing, team model and community numbers for both, each with a source link and a checked-on date, are on the compare page. Memory, disk and container counts for a self-hosted Agenta are in Self-hosting LLM observability: seven open-source tools measured.

To compare more tools, see 10 LLM observability tools tested hands-on, 8 LangSmith alternatives and 8 Langfuse alternatives.

To try AcruxCore, start with the Quickstart.

This post compares eight alternatives to Langfuse on self-hosting effort, license, price and the features that need an enterprise plan. For each Langfuse alternative, it lists what the tool adds and what it lacks.

The short answer

• Self-hosted without ClickHouse: AcruxCore runs on Postgres and Redis. MLflow and Arize Phoenix each run as one Python process on SQLite.
• Cheapest paid cloud: Opik, at $19 a month.
• A gateway, prompts, traces and evals in one open-source tool: AcruxCore or MLflow.
• Built for LangChain apps: LangSmith.

Langfuse alternatives: feature comparison​

License, hosting and price

Features

A unit on Langfuse is one trace, one step inside a trace, or one score. A span is one step inside a trace. "Data stores" counts databases, caches, queues and file stores, not the application containers. Closed source means the vendor publishes no server code. An MCP server lets a coding assistant, such as Claude Code or Cursor, work with the platform's data.

If you want the self-hostable tools at a glance, the comparison page puts AcruxCore next to Langfuse and five other self-hostable platforms, feature by feature.

Why look for alternatives to Langfuse?​

• Self-hosting runs four data stores. A Langfuse install runs Postgres, ClickHouse, Redis and S3-compatible storage, plus a web and a worker container.
• Some features need an enterprise license, even self-hosted. Audit logs, project-level roles, data retention controls, automatic user provisioning (SCIM), protected prompt labels and server-side masking are in the enterprise (ee/) folders.
• Langfuse has no gateway. Langfuse records model calls but does not send them to the provider. The Langfuse docs use LiteLLM as the gateway in front of it.
• Every step is billed. Each step and each score inside a trace counts as one unit. The Hobby plan stops at 50,000 units with no overage, and Core charges $8 per 100,000 units over the included 100,000.
• ClickHouse owns Langfuse. ClickHouse acquired Langfuse in January 2026. The Langfuse core stays MIT, and Langfuse Cloud stays a product separate from ClickHouse Cloud.

Sources: Langfuse license, compose file, self-host pricing, cloud pricing, LiteLLM page, acquisition.

How we picked​

• The tool does Langfuse's core jobs: tracing, and at least one of prompts or evals.
• The tool is active today. Each tool had a release or a commit in October 2026.
• The price is public, or the tool is free to self-host.
• Every fact comes from the vendor: the pricing page, the docs, the license file or the repository's compose file, read on 17 October 2026. The MCP servers, except OpenLIT's, were checked on 18 October 2026. We do not use other comparison lists as sources.

Self-hosted Langfuse alternatives​

These five tools use a license approved by the Open Source Initiative, and each one is free to self-host.

1. AcruxCore​

AcruxCore vs Langfuse: both are open source and cover prompts, tracing and evals. AcruxCore also has a gateway. A self-hosted AcruxCore needs only Postgres and Redis.

Best for: small teams that want prompts, a gateway, traces and evals in one self-hosted app.

Where AcruxCore differs

• The gateway is part of the product. It holds your provider keys, caches responses, falls back to a second model, and stops calls at a spend budget or a rate limit.
• You define the tools a model can call and keep versions of them. AcruxCore runs a tool when your code calls its API.
• Roles and the audit log are free in every AcruxCore install. Langfuse keeps the audit log and project-level roles for its enterprise license.
• A prompt run can rewrite the prompt for you and score the new version against a dataset.
• The MCP server lets a coding assistant run a stored prompt through the gateway and read the trace that the run produced.

Limits of AcruxCore

• The AcruxCore repository went public in August 2026 and has 11 GitHub stars. Langfuse has 35,957 stars.
• AcruxCore has no SAML or OIDC single sign-on. Sign-in is email or Google.
• AcruxCore deletes span content after 90 days by default. One setting covers the whole install, not each project.
• The SDKs are Python and TypeScript only.

Our hands-on comparison is AcruxCore vs Langfuse.

Sources: license, compose file, pricing.

2. MLflow​

MLflow vs Langfuse: MLflow is an Apache-2.0 ML platform with tracing, a prompt registry, judges and a gateway in one server.

Best for: teams that already run MLflow for classic machine learning.

Where MLflow differs

• MLflow can run as one Python process with a SQLite file, with no Docker.
• The gateway splits traffic by weight, falls back to another model, and rejects calls over a USD budget.
• User permissions and workspaces are in the open-source version, with no license key.
• The same server holds your training runs and model registry.
• MLflow's MCP server reads traces and runs judges on them. The MLflow docs call the MCP server experimental.

Limits of MLflow

• MLflow has no hosted plan of its own. The managed version needs a Databricks workspace.
• Judges that score live traces need a gateway endpoint set up first.

Our comparison: AcruxCore vs MLflow.

Sources: license, GenAI docs, self-hosting, compose file, managed MLflow.

3. OpenLIT​

OpenLIT vs Langfuse: OpenLIT is built on OpenTelemetry from the start and self-hosts on ClickHouse plus one app container.

Best for: teams that already run OpenTelemetry or Grafana and also want GPU metrics.

Where OpenLIT differs

• The SDK sends standard OpenTelemetry data, so you can also export it to Grafana, Datadog or any OTLP backend.
• OpenLIT Vault stores your provider keys, and your code fetches the keys at run time.
• A separate collector reports GPU metrics for NVIDIA, AMD and Intel cards.
• The SDKs cover Python, TypeScript and Go.

Limits of OpenLIT

• Roles, audit logs and alerts need the Enterprise Edition.
• OpenLIT Cloud has a waitlist only, so today you must run OpenLIT yourself.

Sources: license, compose file, docs, pricing.

4. Laminar​

Laminar vs Langfuse: Laminar is open-source tracing built for agents, with SQL over your traces and no prompt management.

Best for: teams that run agents with many steps, or agents that use a browser.

Where Laminar differs

• Laminar replays a browser agent's session as video, next to the matching steps of the trace.
• You can query traces, spans and metrics with SQL.
• Signals are rules written in plain English, such as "agent is stuck in a loop". Laminar checks every trace against these rules.
• Laminar bills by gigabyte of trace data, not by the number of steps.
• Laminar's MCP server gives a coding assistant read-only access to traces, and the assistant can query them with SQL.

Limits of Laminar

• Laminar has no prompt management, so you need another tool to version prompts.
• Laminar's free plan keeps data 7 days and allows 1 seat.

Our comparison: AcruxCore vs Laminar.

Sources: repository, compose file, pricing.

5. Opik​

Opik vs Langfuse: the whole Opik platform is Apache-2.0, and Opik has the cheapest paid cloud plan here.

Best for: teams that want open-source evals and automatic prompt optimization.

Where Opik differs

• Opik ships an optimizer package that rewrites prompts and agents for you, with six optimization algorithms.
• Opik has guardrails for PII, topics and prompt injection.
• Opik's free cloud plan keeps data 60 days for up to 10 members. Langfuse Hobby keeps 30 days for 2 users.
• Opik's MCP server reads traces and saves new prompt versions.

Limits of Opik

• The self-hosted version has no user management.
• A self-hosted install runs five data stores, one more than Langfuse.

Our comparison: AcruxCore vs Opik.

Sources: README and license, pricing, self-hosting, compose file, gateway, optimizer.

Source-available: Arize Phoenix​

6. Arize Phoenix​

Phoenix vs Langfuse: Phoenix self-hosts in one container with no feature limits. Its license, the Elastic License 2.0, is not approved by the Open Source Initiative.

Best for: engineers who want free self-hosted tracing and evals built on OpenTelemetry.

Where Phoenix differs

• Self-hosted Phoenix has no feature limits and no license fee.
• The Elastic License forbids selling Phoenix to others as a hosted service. Running Phoenix for your own team is allowed.
• Self-hosted Phoenix includes OAuth2, LDAP and role-based access.
• Tracing uses OpenInference, Arize's OpenTelemetry conventions.
• Phoenix's MCP server creates prompts, tags prompt versions and reads traces.

Limits of Phoenix

• The hosted plans are small: 25,000 spans on Free and 50,000 on Pro.

Our comparison: AcruxCore vs Phoenix.

Sources: license, self-hosting, compose file, pricing.

Closed-source alternatives​

LangSmith and Braintrust publish no server code. Self-hosting either one needs an enterprise contract.

7. LangSmith​

LangSmith vs Langfuse: LangSmith is the LangChain team's hosted platform. Langfuse is open source and free to self-host.

Best for: apps built on LangChain or LangGraph.

Where LangSmith differs

• LangSmith has a gateway in beta, with spend and rate limits.
• LangSmith has SDKs for Go and Java as well as Python and TypeScript.
• Startups with venture funding can get up to $10,000 in credits.
• LangSmith's MCP server pushes and tags prompts, reads traces, and works with deployments and sandboxes.

Limits of LangSmith

• LangSmith keeps base traces 14 days. Keeping traces 180 days costs extra.
• Plus costs $39 per seat, so three people pay $117 a month before they pay for extra traces.

We compared both on one app in Langfuse vs LangSmith. For more options starting from LangSmith, see 8 LangSmith alternatives.

Sources: pricing, billing, gateway, self-hosting.

8. Braintrust​

Braintrust vs Langfuse: Braintrust is a closed-source, hosted platform built around evals, with a flat price and unlimited users.

Best for: teams whose main work is evals and experiments.

Where Braintrust differs

• Loop is an agent inside Braintrust. It reads your logs and experiments, then edits prompts, scorers and datasets after you approve.
• The gateway caches responses for up to 7 days and falls back to another provider on errors.
• Braintrust gives Pro free for 6 to 12 months to startups that qualify.
• Braintrust's MCP server creates prompts, queries logs in SQL and runs evals.

Limits of Braintrust

• Braintrust's free plan keeps data 14 days.
• Self-hosting any part of Braintrust needs an Enterprise contract.

Our comparison: AcruxCore vs Braintrust. For more options starting from Braintrust, see 10 Braintrust alternatives.

Sources: pricing, self-hosting, gateway, Loop.

Tools other lists still recommend​

Older lists of Langfuse alternatives still include these six tools. Their status on 17 October 2026:

• Lunary is no longer open source. Its server repository returns 404 on GitHub, and Docker self-hosting needs the Enterprise Edition. (source)
• Agenta now calls itself "the open-source workspace for your agents". Its current docs no longer cover prompts, evaluation or observability; those pages are in an archived v1.0 section. (source)
• Helicone was acquired by Mintlify in March 2026. The service runs in maintenance mode: it gets security fixes, bug fixes and new models, but no new features. (source)
• Portkey was acquired by Palo Alto Networks in May 2026. The open-source gateway repository has had no merged code since 25 May 2026. (source)
• Galileo was acquired by Cisco and is now sold as Splunk Agent Observability. (source)
• Traceloop announced in March 2026 that it is joining ServiceNow. Its OpenLLMetry library stays open source and active. (source)

We left out Pydantic Logfire: its platform is closed source, and self-hosting it needs an enterprise contract. (source)

Which Langfuse alternative should you pick?​

Start at the top and stop at the first question you answer yes to.

Best Langfuse alternative for a startup​

• AcruxCore if you want one self-hosted app for prompts, gateway, traces and evals, and the hosted plan is free to use.
• Opik if you want a paid cloud plan: $19 a month for up to 50 members.
• MLflow if you already use MLflow and want a gateway in the same server.

Cheaper alternatives to Langfuse​

Best free Langfuse alternative​

• AcruxCore, MLflow or Arize Phoenix to self-host without ClickHouse. AcruxCore runs on Postgres and Redis. MLflow and Phoenix each run as one Python process on SQLite. Phoenix's license forbids selling it as a hosted service.
• Opik for a free cloud plan that keeps data 60 days for up to 10 members.

For AI agent observability​

• Laminar replays browser-agent sessions and checks every trace against plain-English rules.
• Opik and Phoenix trace OpenAI Agents SDK runs and score them with LLM judges.

If your app is built on LangChain​

• LangSmith is built by the LangChain team. LangChain and LangGraph runs send traces to LangSmith once LANGSMITH_TRACING and LANGSMITH_API_KEY are set.

Langfuse alternatives for enterprises​

• LangSmith Enterprise and Braintrust Enterprise include self-hosting, single sign-on and an uptime guarantee (SLA).
• Langfuse Enterprise costs $2,499 a month on the cloud. It adds audit logs, automatic user provisioning (SCIM) and an uptime SLA.

FAQ​

Is Langfuse Apache 2.0?​

No. The Langfuse core is under the MIT license. The ee/ folders are under a separate commercial license and need a Langfuse Enterprise agreement. (license)

Is Langfuse free?​

The self-hosted core is free. The Hobby cloud plan is free for 50,000 units a month, 2 users and 30 days of data. (pricing)

What does self-hosting Langfuse take?​

A web container, a worker container, Postgres, ClickHouse, Redis or Valkey, and S3-compatible storage. (self-hosting)

Who owns Langfuse now?​

ClickHouse acquired Langfuse in January 2026. The core stays MIT-licensed. (announcement)

Does Langfuse have an LLM gateway?​

No. Langfuse records model calls but does not send them to the provider. The Langfuse docs use LiteLLM as the gateway. AcruxCore and MLflow include a gateway. LangSmith and Opik have one in beta, and Braintrust has a hosted one in preview.

Related posts​

• AcruxCore vs Langfuse: the same prompt built and run on both, with screenshots and a latency benchmark.
• Langfuse vs LangSmith: one app run end to end on both.
• 10 LLM observability tools tested hands-on: ten tools tested hands-on, with screenshots of each.
• AcruxCore vs Langfuse vs LangSmith: the same app run on all three.
• Langfuse hands-on walkthrough: every Langfuse screen, step by step.
• 8 LangSmith alternatives for startups: the same comparison, starting from LangSmith.
• Self-hosting LLM observability: seven open-source tools measured: memory, disk and trace storage for Langfuse, AcruxCore and five more stacks on one machine.
• Self-hosting AcruxCore: the docker compose setup.

This post compares eight LangSmith alternatives on license, self-hosting, price and features. For each alternative to LangSmith, it lists what the tool adds and what it lacks.

The short answer

• Open source and free to self-host: Langfuse, AcruxCore, Opik, MLflow and Laminar.
• Tracing, prompts, evals and a gateway in one open-source tool: AcruxCore or MLflow.
• Evals first, hosted for you: Braintrust.

LangSmith competitors at a glance​

License and hosting

Features

A unit on Langfuse is one trace, one step inside a trace, or one score. A span is one step inside a trace. Opik, Phoenix and Braintrust count spans or gigabytes, so their free plans are not directly comparable with LangSmith's 5,000 traces. Closed source means the vendor publishes no server code. An MCP server lets a coding assistant, such as Claude Code or Cursor, work with the platform's data.

If you want the self-hostable tools at a glance, the comparison page puts AcruxCore next to six self-hostable platforms, feature by feature.

Why teams look for a LangSmith alternative​

• Every seat costs money. The Plus plan costs $39 per seat a month. Three people pay $117 a month before they pay for extra traces.
• Traces are kept 14 days. Base traces on Free and Plus are deleted after 14 days. Extended traces are kept 180 days for an extra fee.
• Self-hosting needs Enterprise. LangSmith self-hosting is an add-on to the Enterprise plan. Self-hosted LangSmith runs on Kubernetes with ClickHouse, PostgreSQL and Redis. It needs a license key from sales.
• The platform is not open source. The SDKs are MIT. The server ships only as container images that need a license key.

LangSmith does not need LangChain for tracing. Its traceable decorator and its OpenTelemetry endpoint work with any code.

Sources: LangSmith pricing, billing, self-hosting and OpenTelemetry pages.

How we picked​

• The tool does LangSmith's core jobs: tracing, and at least one of prompts or evals.
• The tool is active today. Each tool had a release or a commit in October 2026.
• The price is public, or the tool is free to self-host.
• Every fact comes from the vendor: the pricing page, the docs, the license file or the repository's compose file, read on 17 October 2026. The MCP servers were checked on 18 October 2026. We do not use other comparison lists as sources.

Open-source LangSmith alternatives (free to self-host)​

These five tools use a license approved by the Open Source Initiative. You can run every one of them on your own servers without paying.

1. Langfuse​

Langfuse vs LangSmith: Langfuse is open source and free to self-host, and LangSmith is neither.

Best for: teams that want the open-source tool with the largest community for tracing, prompts and evals.

Where Langfuse differs

• Langfuse charges one flat price for the whole team on Core. LangSmith charges per seat.
• Langfuse keeps data 90 days on Core. LangSmith keeps base traces 14 days on Plus.
• Langfuse has the largest community here: 35,957 GitHub stars.
• Datasets, experiments, LLM-as-judge on live traces and review queues are all in the free self-hosted version.
• Langfuse's MCP server creates prompts, moves prompt labels, and reads traces, datasets and scores.

Limits of Langfuse

• Langfuse has no gateway. The Langfuse docs use LiteLLM as the gateway.
• A self-hosted install runs four data stores: Postgres, ClickHouse, Redis and S3.
• Audit logs, project-level roles and retention controls need an enterprise license, even when you self-host.

We compared both tools on one app in Langfuse vs LangSmith.

Sources: license, pricing, self-host pricing, compose file, ClickHouse acquisition.

2. AcruxCore​

AcruxCore vs LangSmith: both cover prompts, tracing and evals. AcruxCore is Apache-2.0 and free to self-host, and its gateway sends your model calls to the provider.

Best for: small teams that want prompts, a gateway, traces and evals in one self-hosted app.

Where AcruxCore differs

• The gateway is part of the product. It holds your provider keys, caches responses, falls back to a second model, and stops calls at a spend budget or a rate limit.
• You define the tools a model can call and keep versions of them. AcruxCore runs a tool when your code calls its API.
• Roles and the audit log are free in every AcruxCore install. LangSmith keeps custom roles for Enterprise.
• Self-hosting needs two data stores, Postgres and Redis, with no ClickHouse.
• The MCP server lets a coding assistant run a stored prompt through the gateway and read the trace that the run produced.

Limits of AcruxCore

• The AcruxCore repository went public in August 2026 and has 11 GitHub stars.
• AcruxCore has no SAML or OIDC single sign-on. Sign-in is email or Google.
• The gateway supports OpenAI, Anthropic, Gemini and OpenAI-compatible APIs. The SDKs are Python and TypeScript only.

Our longer comparison is AcruxCore vs LangSmith.

Sources: license, compose file, pricing.

3. Opik​

Opik vs LangSmith: Opik is Apache-2.0 from end to end and has the cheapest paid plan here.

Best for: teams that want open-source evals and automatic prompt optimization.

Where Opik differs

• Opik ships an optimizer package that rewrites prompts and agents for you, with six optimization algorithms.
• Opik has guardrails for PII, topics and prompt injection.
• Opik's free cloud plan allows 10 members. LangSmith's free plan allows 1 seat.
• Opik's MCP server reads traces and saves new prompt versions.

Limits of Opik

• The self-hosted version has no user management.
• A self-hosted install runs five data stores: MySQL, Redis, ClickHouse, ZooKeeper and MinIO.
• The Opik docs describe the Opik gateway as beta, for development and testing.

Our comparison: AcruxCore vs Opik.

Sources: README and license, pricing, self-hosting, compose file, gateway, optimizer.

4. MLflow​

MLflow vs LangSmith: MLflow is an Apache-2.0 ML platform with tracing, a prompt registry, judges and a gateway in one server.

Best for: teams that already run MLflow for classic machine learning.

Where MLflow differs

• MLflow can run as one Python process with a SQLite file, with no Docker.
• The gateway splits traffic by weight, falls back to another model, and rejects calls over a USD budget.
• Judges can score live traces with no code change, and review queues assign traces to people.
• The same server holds your training runs and model registry.
• MLflow's MCP server reads traces and runs judges on them. The MLflow docs call the MCP server experimental.

Limits of MLflow

• MLflow has no hosted plan of its own. The managed version needs a Databricks workspace.

Our comparison: AcruxCore vs MLflow.

Sources: license, GenAI docs, self-hosting, compose file, managed MLflow.

5. Laminar​

Laminar vs LangSmith: Laminar is open-source tracing built for agents, with SQL over your traces.

Best for: teams that run agents with many steps, or agents that use a browser.

Where Laminar differs

• Laminar replays a browser agent's session as video, next to the matching steps of the trace.
• You can query traces, spans and metrics with SQL.
• Laminar's MCP server gives a coding assistant read-only access to traces, and the assistant can query them with SQL.
• Signals are rules written in plain English, such as "agent is stuck in a loop". Laminar checks every trace against these rules.

Limits of Laminar

• Laminar has no prompt management, so you need another tool to version prompts.
• Laminar's free cloud plan keeps data 7 days, and Starter keeps data 30 days.

Our comparison: AcruxCore vs Laminar.

Sources: repository, compose file, pricing.

Source-available: Arize Phoenix​

6. Arize Phoenix​

Phoenix vs LangSmith: Phoenix is free to self-host with no feature limits. Its license, the Elastic License 2.0, is not approved by the Open Source Initiative.

Best for: engineers who want free self-hosted tracing and evals built on OpenTelemetry.

Where Phoenix differs

• The Elastic License forbids selling Phoenix to others as a hosted service. Running Phoenix for your own team is allowed.
• Self-hosting takes one container, or pip install arize-phoenix and phoenix serve, with SQLite.
• Tracing uses OpenInference, Arize's OpenTelemetry conventions, so any OpenTelemetry language can send spans.
• Prompts have versions and tags, and the playground replays a traced model call.
• Phoenix's MCP server creates prompts, tags prompt versions and reads traces.

Limits of Phoenix

• The hosted plans are small: 25,000 spans on Free and 50,000 on Pro.
• On the hosted Arize AX plans, single sign-on and audit logs need Enterprise.

Our comparison: AcruxCore vs Phoenix.

Sources: license, self-hosting, compose file, pricing.

Hosted, closed-source alternatives​

Braintrust and W&B Weave publish no server code, like LangSmith. Self-hosting either one needs an enterprise contract.

7. Braintrust​

Braintrust vs LangSmith: both are closed source and hosted. Braintrust charges a flat price with unlimited users, and LangSmith charges per seat.

Best for: teams whose main work is evals and experiments.

Where Braintrust differs

• Loop is an agent inside Braintrust. It reads your logs and experiments, then edits prompts, scorers and datasets after you approve.
• The gateway caches responses for up to 7 days and falls back to another provider on errors.
• Braintrust gives Pro free for 6 to 12 months to startups that qualify. LangSmith gives up to $10,000 in credits to startups with venture funding.
• Braintrust's MCP server creates prompts, queries logs in SQL and runs evals.

Limits of Braintrust

• Braintrust's free plan keeps data 14 days.
• Prompt environments need Pro.
• The Braintrust gateway has no uptime guarantee (SLA) while it is in preview.

Our comparison: AcruxCore vs Braintrust. For more options starting from Braintrust, see 10 Braintrust alternatives.

Sources: pricing, self-hosting, gateway, Loop.

8. W&B Weave​

Weave vs LangSmith: Weave puts LLM traces and evals next to Weights & Biases experiment tracking.

Best for: teams that already use Weights & Biases.

Where Weave differs

• One subscription covers W&B experiment tracking and Weave traces.
• Monitors run LLM-judge checks, such as hallucination, on every production trace.
• The playground compares models side by side, using provider keys stored as team secrets.
• Weave's MCP server reads traces, evaluation results, runs and reports.

Limits of Weave

• Pro includes 1.5 GB of traces a month.
• The pricing page does not state how long traces are kept.

Sources: pricing, README, self-managed Weave, monitors, CoreWeave acquisition.

Tools other lists still recommend​

Older lists of LangSmith alternatives still include these five tools. Their status on 17 October 2026:

• Helicone was acquired by Mintlify in March 2026. The service runs in maintenance mode: it gets security fixes, bug fixes and new models, but no new features. (source)
• Portkey was acquired by Palo Alto Networks in May 2026. The open-source gateway repository has had no merged code since 25 May 2026. (source)
• Humanloop shut down on 8 September 2025. (source)
• Traceloop announced in March 2026 that it is joining ServiceNow. Its OpenLLMetry library stays open source and active. (source)
• Galileo was acquired by Cisco and is now sold as Splunk Agent Observability. It is closed source. (source)

Which LangSmith alternative should you pick?​

Start at the top and stop at the first question you answer yes to.

Best LangSmith alternative for a startup​

• AcruxCore if you want one self-hosted app for prompts, gateway, traces and evals, and the hosted plan is free to use.
• Opik if you want a paid cloud plan: $19 a month for up to 50 members.
• Langfuse if you want the largest open-source community and $29 a month for unlimited users.

Best free LangSmith alternative​

• AcruxCore, MLflow or Arize Phoenix to self-host without ClickHouse. AcruxCore runs on Postgres and Redis. MLflow and Phoenix each run as one Python process on SQLite. Phoenix's license forbids selling it as a hosted service.
• Opik for a free cloud plan that keeps data 60 days for up to 10 members.

Self-hosted LangSmith alternatives for regulated teams​

• Langfuse and Opik keep every trace on your servers. Both also need ClickHouse, an analytics database you must run and back up.
• AcruxCore keeps every trace on your servers on Postgres and Redis.
• MLflow keeps every trace on your servers on Postgres and an S3-compatible store.

LangSmith alternatives for enterprises​

• Braintrust Enterprise adds SAML single sign-on, custom roles and an SLA.
• Langfuse Enterprise adds audit logs, automatic user provisioning (SCIM) and an uptime SLA.

FAQ​

Is LangSmith open source?​

No. The LangSmith SDKs are MIT-licensed, but the platform has no public source code. The self-hosted version ships as container images that need a license key.

Is LangSmith free?​

The Developer plan is free for 1 seat and 5,000 base traces a month, kept for 14 days. The Plus plan costs $39 per seat a month. (pricing)

Can you self-host LangSmith?​

Only on the Enterprise plan. Self-hosting is an add-on to that plan and needs a license key from sales. It runs on Kubernetes with ClickHouse, PostgreSQL and Redis.

Does LangSmith need LangChain?​

Not for tracing. The traceable decorator and the OpenTelemetry endpoint work without LangChain.

What is the best open-source LangSmith alternative?​

Langfuse has the most GitHub stars of the open-source alternatives: 35,957. AcruxCore and MLflow add a gateway to tracing, prompts and evals. Opik adds automatic prompt optimization.

Related posts​

• Langfuse vs LangSmith: one app run end to end on both.
• 10 LLM observability tools tested hands-on: ten tools tested hands-on, with screenshots of each.
• AcruxCore vs Langfuse vs LangSmith: the same app run on all three.
• LangSmith hands-on walkthrough: every LangSmith screen, step by step.
• 8 best Langfuse alternatives for startups: the same comparison, starting from Langfuse.
• Quickstart: sign up for AcruxCore and send a first trace.
• Self-hosting LLM observability: seven open-source tools measured: memory, disk and trace storage for seven self-hosted stacks on one machine.
• Self-hosting AcruxCore: the docker compose setup.

Open-source LLM observability tools describe self-hosting the same way. You clone the repo, run one docker compose up, and open the dashboard. Most self-host pages say nothing after that step.

The compose file behind that command fixes several things the self-host page does not list. The file sets how many containers you run and which databases come with them. Those containers set how much memory the server needs before the first trace arrives. The file's volume mounts decide whether your data survives a restart.

We ran the official self-host setup of seven tools in October 2026: Langfuse, Opik, Phoenix, MLflow, Laminar, Agenta and AcruxCore. Each ran alone on the same machine. Each received the same 1,000 traces.

Short answer​

• Idle memory runs from 0.39 GiB to 3.69 GiB. AcruxCore and Phoenix use the least. AcruxCore and Phoenix store traces in Postgres, as MLflow and Agenta do. Opik uses the most, and after 1,000 traces it holds 4.5 GiB.
• The databases use memory before any trace arrives. Three of the seven ship a column store called ClickHouse. ClickHouse holds 0.6 to 1.1 GiB in each of those stacks before any trace arrives.
• Trace data is small, and the disk grows for other reasons. The 1,000 traces take 0.8 to 1.1 MiB in the three ClickHouse stacks and 4.6 to 13.5 MiB in the four Postgres stacks. Opik's disk still grew by about 170 MiB, because ClickHouse logs its own activity by default. Laminar ships a config that turns those logs off.
• Only Langfuse documents hardware. Langfuse asks for 4 cores, 16 GiB of memory and 100 GiB of disk. The other six publish no hardware numbers.
• Agenta has the largest install. It runs 14 containers from 12.5 GB of images, and 6.9 GB of that is the image for its agent runner.

At a glance​

"Containers" counts the containers that keep running. Opik also starts three short jobs that finish and exit, and MLflow and Agenta start one each. We ran Langfuse twice, so its row shows both runs.

Image sizes are uncompressed and summed per image, so a layer that two images share counts twice.

How we measured​

The machine was a 24-core Linux workstation with 62 GiB of memory, Docker 29.8 and Compose 5.6. We followed each project's own self-host page and used the files that page points to. We pinned every image to the newest published release and changed host ports that another program on the machine already used. The only other change was one line in MLflow's database URL, which its compose file needs to start on the current release. All images were downloaded before we started timing, so the ready times leave out download time. The scripts, every change we made to each setup and every raw reading are listed under Scripts and raw readings.

Every tool went through the same five steps:

1. Start the stack and time how long the app takes to answer its health check or home page.
2. Leave the stack idle for five minutes. Then read memory from docker stats twelve times, ten seconds apart, and keep the median.
3. Send 1,000 traces. Each trace is one agent turn with three spans: the agent run, a chat-model call carrying about 1.6 KB of prompt and answer, and a tool call. The content is identical for every tool. The traces go over OTLP, the OpenTelemetry wire format, which all seven accept.
4. Check inside each tool's database that all 1,000 traces arrived. All 1,000 arrived in every tool.
5. Read memory and disk two minutes after the load. Read disk again about fifteen minutes after the load.

The databases use memory before any trace arrives​

The app process is only part of the memory. Here is where the memory sits inside each stack at idle:

• Opik (3.69 GiB): Java backend 1.18 GiB, ClickHouse 1.11 GiB, MySQL 0.61 GiB, Python backend 0.44 GiB, ZooKeeper 0.29 GiB, then MinIO, Redis and the web server.
• Langfuse (2.60 GiB): web app 0.92 GiB, ClickHouse 0.88 GiB, worker 0.74 GiB, then MinIO, Postgres and Redis.
• AcruxCore (0.39 GiB): Node API 0.20 GiB, Node worker 0.08 GiB, Postgres 0.07 GiB, then Redis and an nginx web server.
• Laminar (1.11 GiB): ClickHouse 0.61 GiB, then a Next.js frontend, a Rust server, Postgres and Quickwit, a search engine.
• Phoenix (0.47 GiB): one Python server at 0.43 GiB and Postgres at 0.04 GiB.
• Agenta (2.62 GiB): API 0.85 GiB, services 0.43 GiB, two workers at 0.31 and 0.27 GiB, the SuperTokens login service 0.28 GiB, then two web apps, the agent runner, Postgres, a SeaweedFS object store, Traefik, two Redis instances and the cron job.

ClickHouse is a column store, a database built to scan very large tables quickly. A trace UI runs that kind of query: it filters millions of spans by time, model or cost, then aggregates them. Langfuse, Opik and Laminar all ship ClickHouse for that reason. ClickHouse also costs 0.6 to 1.1 GiB of memory before you store any traces at all.

MLflow and Agenta have no ClickHouse and still use more than 2 GiB, because their Python servers run several processes each. Agenta's API container runs two copies of its Python web server, and each copy holds about 460 MB. The two copies share some memory, so the API container totals 0.85 GiB. MLflow's 2.2 GiB server is not one process either. Inside it we counted fifteen Python processes: four web workers, seven background job consumers and a few supervisors. Most of those processes hold around 230 MB. You can lower that number in the config, but fifteen is the default.

After 1,000 traces, six of the seven grew by less than 0.3 GiB. Opik grew the most. Its backend and ClickHouse together grew by about 0.8 GiB, and 0.7 GiB of that was still in use fifteen minutes later.

Trace data is small, and the disk grows for other reasons​

The table shows three figures for each stack. The first is the disk used right after setup. The second is the space the 1,000 traces take inside the tool's database. The third is how much the whole stack's disk grew, read fifteen minutes after the load:

The trace space counts each tool's trace tables and their indexes. We measured it in a second run of each stack with the same 1,000 traces. Phoenix's starting figure includes 90 MiB of Python bytecode that the server writes into its own container on first start. Phoenix has no fifteen-minute reading, so its growth figure is the two-minute reading.

The second column grows with the number of traces you send. The third column also holds costs that do not depend on traces, and those costs differ between ClickHouse and Postgres.

ClickHouse stacks store traces compactly and log their own activity​

ClickHouse stores compressed columns, so Laminar, Opik and Langfuse each hold the 1,000 traces in about 1 MiB. Langfuse also keeps 3.9 MiB of raw event files in MinIO.

The rest of Opik's and Langfuse's growth came from ClickHouse's own system tables. By default ClickHouse keeps system tables that log every query, its internal metrics every second, and its own text log. Those tables grow with time, even when no traces arrive. On Opik's stack the system tables held 26 MiB of live data, and another 97 MiB were old pieces waiting to be cleaned up. ClickHouse writes each insert as a separate "part" on disk. After parts are merged, ClickHouse deletes the old ones about eight minutes later. Right after the load, Opik's trace table had 1,045 such old parts on disk. A disk reading two minutes after a load therefore shows more disk use than remains later, so we took a second reading fifteen minutes later.

Laminar's compose file mounts a config that turns off sixteen of these system tables and keeps the query log for three days. So the only growth left on Laminar's disk was the traces themselves. The same delayed cleanup removed old files from setup at the same time, so Laminar's ClickHouse volume ended slightly smaller than it started.

Postgres stacks store more per trace and write a journal in fixed steps​

Postgres stores rows, not compressed columns, so the four Postgres stacks need 4.6 to 13.5 MiB for the same traces. Indexes explain most of the range. AcruxCore's indexes take 1.3 MiB. Agenta's take 9.8 MiB, mostly for a full-text search index and a JSON index on span attributes, which let Agenta search inside span content.

Postgres also writes a write-ahead log, the journal it writes before it changes any table. Postgres writes this log in files of 16 MiB and reuses old files instead of deleting them. AcruxCore, Phoenix and MLflow each started with one log file, and the load opened a second one, so each stack's disk grew by 16 MiB at once. Agenta's setup runs many database migrations and had already created two log files, so the load reused them. That is why Agenta's total growth is the smallest of the four, although its traces take the most space. By default Postgres aims to keep this log under 1 GB, however many traces you store.

What we found only by running each setup​

Agenta: fourteen containers and a 6.9 GB runner image​

Agenta's quick start runs 14 containers and one migration job from 12.5 GB of images, the most of the seven. The agent runner, which executes the agents you build in Agenta, has a 6.9 GB image on its own. The API answered 47 seconds after up, and the stack held 2.62 GiB of memory at idle. Agenta keeps traces in Postgres, where the 1,000 traces took 13.5 MiB, most of it in search indexes on span content. The example env file leaves several secrets at replace-me, including the admin API key, so set them before anyone else can reach the stack.

Laminar: slow first boot, fast after that​

Laminar's frontend applies the whole database migration history on first start. In two runs the UI took 4.7 and 4.8 minutes to answer, far longer than any other stack here. Later boots skip the migrations that already ran.

The compose file sets pull_policy: always on Laminar's own two images and tags neither image with a version. Every up checks the registry, and you get whatever image was pushed last. Pin a digest if you want to run the same software twice. The repository also commits a .env file with default passwords. The compose file hard-codes the frontend's NEXTAUTH_SECRET, the key that signs login sessions. Change the passwords and the secret before anyone else can reach the stack.

AcruxCore: read the quickstart file before you keep it​

AcruxCore's README offers two compose files. We measured the quickstart, docker-compose.local.yml. It runs Postgres, Redis, a Node API, a Node worker and an nginx web server, and it is healthy 12 seconds after up. It has no column store, which explains most of its low memory. Without a column store, its traces take more disk per span than in ClickHouse, as the disk section shows.

The quickstart runs with no values filled in, because it ships fixed defaults. The Postgres password is acruxcore. Email is off, and new accounts count as verified. The GATEWAY_ENCRYPTION_KEY has a default value committed in the file. That key is the AES key that encrypts every model-provider API key you store. The file's own comments say the default is fine on your laptop and never on a server others can reach. Set your own key before you store a real provider key, because changing the key later makes every stored provider key unreadable. Login is on by default, but sign-up stays open, so anyone who reaches the port can create an account. A new account gets its own team and cannot see yours.

The production file, docker-compose.yml, brings no database. You supply Postgres and a reverse proxy, and the file publishes no ports of its own. The production file also joins a Docker network named mynetwork that Compose does not create. The README's production steps run docker network create mynetwork before docker compose up for that reason.

Langfuse, Opik and Phoenix​

• Langfuse is the only one of the seven whose docs state hardware: at least 4 cores, 16 GiB of memory and 100 GiB of disk on a VM. Langfuse marks 14 lines in its compose file with # CHANGEME, which are its default secrets. Its docs say the compose setup has no high availability, no scaling and no backups, and point to Kubernetes for production.
• Opik loads a demo project with 156 traces on first start. Its docs say the compose setup "is not meant for production deployments." Opik has no login screen in self-hosted mode, so anyone who can reach the port can read every trace.
• Phoenix has authentication off by default, and you turn it on with PHOENIX_ENABLE_AUTH. Phoenix's docs also show a single-container option on SQLite, which we did not measure. We measured the Postgres setup, because that is the setup you would keep running.

Which tool fits which server​

On memory alone, AcruxCore and Phoenix fit on a server with 1 GB. Laminar fits on 2 GB, with little memory left over. MLflow, Langfuse and Agenta need 4 GB once you allow for the operating system. Opik needs 8 GB, given how much Opik grows under load. Agenta's images also need 12.5 GB of disk before it stores anything.

The larger stacks bring a column store, an object store and background workers. Those parts use their memory and disk from the moment the stack starts, before any trace arrives.

Two questions help you choose between the seven:

1. How much server do you want to run? A stack that keeps traces in Postgres, such as AcruxCore, Phoenix, MLflow or Agenta, needs no column store, and Postgres is simpler to back up than ClickHouse. AcruxCore and Phoenix also fit on the smallest server.
2. Who maintains the stack? Langfuse, Opik and Laminar run ClickHouse, and each needs someone who can watch ClickHouse's disk use and limit its logs. Laminar's log settings are a starting point for Langfuse and Opik.

If you want to compare features rather than hosting at a glance, the comparison page puts AcruxCore next to six self-hostable platforms, feature by feature.

Scripts and raw readings​

The harness is in scripts/blogs/self-hosting-llm-observability on GitHub. bench.sh starts each stack, sends the load, and takes the memory and disk readings. load.py sends the OTLP traces. space_clickhouse.sh and space_postgres.sh measure the space the traces take inside each database. The overrides/ folder holds every change we made to each official setup. The results/ folder holds every raw reading, and python3 summarize.py prints the timing, memory and disk figures used in this post.

Six of these tools also have a hands-on comparison with AcruxCore that covers features rather than hosting: Langfuse, Opik, Phoenix, MLflow, Laminar and Agenta.

For license, price and features across more tools, read 8 Langfuse alternatives and 8 LangSmith alternatives. 10 LLM observability tools tested hands-on shows screenshots of each tool, and Langfuse vs LangSmith runs one app on both.

FAQ​

Which open-source LLM observability tool is the lightest to self-host?​

AcruxCore used the least memory of the seven: 0.39 GiB at idle and 0.44 GiB after 1,000 traces. It runs five containers from 3.0 GB of images and answered its health check 12 seconds after start. Phoenix came second, with 0.47 GiB at idle.

Can I self-host LLM observability on a 1 GB server?​

On memory alone, AcruxCore and Phoenix fit on a 1 GB server. Laminar needs about 2 GB. MLflow, Langfuse and Agenta need about 4 GB, and Opik about 8 GB.

Do I need ClickHouse to self-host LLM tracing?​

No. AcruxCore, Phoenix, MLflow and Agenta keep traces in Postgres, which is simpler to back up than ClickHouse. ClickHouse also holds 0.6 to 1.1 GiB of memory before the first trace arrives.

How much disk do 1,000 LLM traces take?​

0.8 to 1.1 MiB in the three ClickHouse stacks, and 4.6 to 13.5 MiB in the four Postgres stacks. AcruxCore needs the least of the Postgres stacks, at 4.6 MiB. The disk usually grows by more than the traces take, because of ClickHouse's own logs and Postgres's write-ahead log, as the disk section explains.

Why does self-hosted Langfuse or Opik use so much disk?​

ClickHouse logs its own queries, metrics and server log into system tables by default, and those tables grow every second. Opik's disk grew by about 170 MiB in fifteen minutes, while its 1,000 traces took 0.9 MiB. Laminar's ClickHouse config turns those logs off, and its settings are a starting point for Langfuse and Opik.

How do I self-host AcruxCore?​

Clone the repository, copy .env.local.example to .env, and run docker compose -f docker-compose.local.yml up. The stack starts Postgres, Redis, the API, the worker and the web app. Set your own GATEWAY_ENCRYPTION_KEY before you store a real model-provider key.

This post compares AcruxCore, Langfuse and LangSmith on the things a developer checks before choosing one. We also ran one app on all three platforms, with the same code, data, prompts and model, and measured each step. The app is a small agent that reads a client's email, checks each request against a signed project scope, prices the extra work, and writes a reply.

Differences at a glance​

Sources: the Langfuse license, pricing, self-hosting and alerts pages; the LangSmith pricing, self-hosting, regions, gateway and alerts pages; the Langfuse MCP and LangSmith MCP pages; and acruxcore.com/pricing. The prices were read on 16 October 2026, the MCP pages on 18 October 2026, and everything else on 17 October 2026.

AcruxCore​

Pros

• The hosted service is free during the beta, with no trace or seat limit.
• Every feature is open source under Apache 2.0, and self-hosting is free, with no paid features.
• The gateway sits in the request path. It has fallbacks, a cache, budgets, rate limits and virtual keys (keys you give an app in place of your provider key). It traces every call with no wrapper in your code. Our test did not use it, to match the other two legs.
• The audit log, which records who changed what, is on every plan, hosted or self-hosted.
• The comparison of two experiment runs opens on the test messages whose score changed.
• Prompts use Jinja2-style templates with if and for, and the server can fill them in.
• It has a versioned tool catalog that the gateway runs, and a built-in prompt optimizer.
• Its MCP server lets a coding assistant run a stored prompt through the gateway and then read the trace that the run produced.

Cons

• It has no review queue. In our test, reviewing one reply by hand took 4 clicks, and 2 on Langfuse and LangSmith.
• The Traces list shows each trace's name, not the user's message, so finding one message means opening rows.
• The SDK has no experiment runner. Your code loops over the test messages and stores the results with 3 REST calls.
• It has no SSO, and the hosted service does not state its data region.
• No price is published for after the beta.
• The community is small: 11 GitHub stars.
• Alerts arrive by email only, and there is no Vercel AI SDK integration.
• The SDK has no wrapped OpenAI client, so the test app needed 4 more lines than on Langfuse.
• A model call links to its prompt version but does not print the prompt's name or version number.

Langfuse​

Pros

• Our app needed the fewest lines of tracing code: 11.
• It has a review queue: 2 clicks per reply.
• The core is MIT-licensed and free to self-host.
• It has the largest community of the three: 35,935 GitHub stars.
• For 100,000 traces a month and 3 people, it costs $98.36 a month and keeps data for 90 days.
• A model call shows its prompt name and version as a link.
• Its MCP server works with prompts, traces, datasets, scores and review queues.

Cons

• It has no gateway.
• Its comparison of two experiment runs cannot hide the test messages whose score stayed the same.
• It bills each step and each score, so the bill grows with the number of steps in a trace.
• SSO and project roles need the paid Teams add-on, and the audit log needs Enterprise.
• A promotion asks for no confirmation, and the prompt page does not show past promotions.
• An experiment with a plain Python function runs one test message at a time.

LangSmith​

Pros

• It is the only one of the three with both a review queue and a filter for the test messages whose score changed. AcruxCore has the filter but no queue. Langfuse has the queue but no filter.
• It has SDKs for Go and Java as well as Python and TypeScript.
• It has a gateway, in beta.
• The prompt page keeps a rollback history.
• Alerts reach Slack, PagerDuty and webhooks.
• Its MCP server works with prompts, traces, deployments and sandboxes.

Cons

• For 100,000 traces a month and 3 people, it costs $567 a month, more than five times the Langfuse price, and keeps data for 14 days.
• The platform is not open source, and self-hosting needs the Enterprise plan.
• Pushing a prompt from code needs the langchain-core package.
• A prompt read with no tag returns the newest commit, not the production one, so a draft can reach users. Langfuse returns the production version.
• The SDK caches a prompt for 5 minutes, so a promotion takes up to 5 minutes to arrive.

Which one to choose​

Five features separate the three.

• AcruxCore if you want a free hosted service or a free self-host with every feature, including a gateway and an audit log, and you can work without a review queue or SSO.
• Langfuse if you want an open-source platform with a large community, a review queue and a low, known price.
• LangSmith if you want a review queue and the filter for changed scores in one product, or need Go or Java, and $567 a month fits your budget.

How we tested​

• The app: a small agent in four steps (extract, classify, price, reply), in one file, pipeline.py, with no tracing code in it. More about the app.
• The data: 30 client messages with the expected items and total, three scorers, and two versions of the prompts. All three platforms got the same files.
• The model: every call, the judge included, went to z.ai glm-5.3 with reasoning_effort="low", straight from the app.
• The platforms: Langfuse and LangSmith on their hosted EU regions, on the free plans. AcruxCore on its hosted service at acruxcore.com.
• The AcruxCore leg: a clean environment with Python 3.12, acruxcore 0.18.0, openai 2.48.0, httpx and python-dotenv.

Tracing​

These are all 15 lines of the AcruxCore instrumentation:

from types import SimpleNamespace
from acruxcore import AcruxCoreSync

hub = AcruxCoreSync(ACX_KEY, ACX_BASE_URL, timeout=180,
                    provider={"base_url": BASE_URL, "api_key": API_KEY})

def _create(*, model, messages, acx_prompt_version=None, **params):  # a z.ai call, traced
    result = hub.gateway.chat(model, messages, prompt_version_id=acx_prompt_version, **params)
    return SimpleNamespace(choices=[SimpleNamespace(message=SimpleNamespace(content=result.content))])

client = SimpleNamespace(chat=SimpleNamespace(completions=SimpleNamespace(create=_create)))

def step(name: str, kind: str):
    return hub.traces.step(name, kind=kind)

def make_app(alias: str = "production", only: set[str] | None = None):
    def get_prompt(name: str):  # `only` limits `alias` to some prompts; the rest stay on production
        version = hub.prompts.get_by_alias(name, alias if not only or name in only else "production")
        return version.messages, {"acx_prompt_version": version.id}
    return build_app(get_prompt, client, step)

• ACX_KEY, ACX_BASE_URL, BASE_URL and API_KEY come from environment variables. build_app is the shared app in pipeline.py, the same for all three platforms.
• AcruxCoreSync is the SDK's blocking client, for code that is not async.
• The provider setting gives the SDK our z.ai URL and key, so each model call goes straight to z.ai, as on the other two platforms.
• traces.step is the decorator. Each step becomes a span, and each model call becomes a span under its step.
• _create and the client line are the 4 lines that Langfuse does not need. The AcruxCore SDK has no wrapped OpenAI client, so these lines give the SDK's chat call the OpenAI shape that the app expects.
• The full file is acruxcore_sdk_leg.py.

All three platforms stored the same tree: the agent, scope-check, at the root, one span per step, one model call inside each step, and the price step marked as a tool.

The AcruxCore gateway is an OpenAI-compatible endpoint in the request path:

• You point the OpenAI client at AcruxCore, and each call arrives as a priced span. Your code needs no wrapper.
• The gateway passes reasoning_effort on to the model, so a call through the gateway runs the model with the same settings. One extract call used 3 reasoning tokens with "low" and 149 without it.
• We did not use the gateway for this test. The other two legs called z.ai directly, and the extra network hop would have added to AcruxCore's latency only.

Prompts, promotion and rollback​

• AcruxCore listed each prompt's variables from its {{ }} placeholders.
• AcruxCore created a production and a staging alias on version 1 of each prompt. Committing version 2 did not move them.
• hub.prompts.get_by_alias(name, alias) returns the version that the alias points to. AcruxCore can also fill in the variables on the server and return the finished messages.

Experiments over the dataset​

An experiment answers one question: after you change a prompt, which messages got better and which got worse? We tested it like this:

• The dataset is 30 client messages for a fictional yoga-studio website, in messages.json. Each message has the expected answer: the extra-work items and the total price.
• Three scorers check each answer: are the items exactly right, is the total exactly right, and does an LLM judge accept the reply.
• v1 is the first version of the prompts. v2 is a rewrite of the extract and classify prompts, made after reading v1's mistakes.
• On each platform, we ran the app over the 30 messages once with v1 and once with v2, and then opened the platform's comparison of the two runs.

On AcruxCore, v2 fixed 2 messages and broke 2 others. The comparison has to show those 4 messages. The table shows how each platform handled that.

• We do not compare the scores across platforms. All three ran the same prompts on the same model, and glm-5.3 gives a slightly different answer each time, so score gaps between platforms are noise. The scores of every run are in the run log.
• AcruxCore also has a server-side experiment, which runs one stored prompt over a dataset with an LLM judge. That kind cannot run this app, because the app has four steps and two of its scorers are Python code.

Reviewing replies by hand​

We asked one question about each of the 30 replies: "Would you send this reply to the client as is?"

• AcruxCore's Traces list shows each trace's name, not the client's message, so finding one message means opening rows.
• We would send 28 of the 30 AcruxCore v1 replies unchanged.

Live traffic​

We sent the 30 messages to all three platforms during the same 5 minutes, one at a time, and read the numbers back through each platform's API.

• The latency is almost all model time, because all three called the same z.ai endpoint.
• We sent this traffic on 16 October 2026, 17:18–17:22 UTC. The model's speed changes during the day. At 06:03 UTC the same app had a p50 of about 16.9 s on Langfuse and LangSmith (Langfuse vs LangSmith), so compare the columns with each other, not with latencies from another hour.
• The model cost is z.ai's bill for one message, not a platform fee. It differs because the model wrote replies of different lengths.
• On all three platforms, we entered z.ai's prices for glm-5.3: $1.40 per million input tokens, $0.26 per million cached input tokens, and $4.40 per million output tokens.
• On AcruxCore, 85% of the input tokens were cached. At the full input price, the 30 traces would have cost $0.088 instead of $0.048.
• AcruxCore's Dashboards page splits requests, errors, tokens, cost and latency by day, model, session, prompt version or step.

Price for 100,000 traces a month​

• Prices were read on 16 October 2026 from acruxcore.com/pricing and the Langfuse and LangSmith pricing pages.
• On all three, you pay the model provider yourself.
• Langfuse bills every step in a trace, so its price grows with the number of steps. LangSmith bills per trace.

Run it yourself​

• This post's scripts on GitHub: the app, the AcruxCore leg, the dataset, both prompt versions, and RUNLOG.md with the raw output of every step.
• The Langfuse and LangSmith legs, with their own run log.
• One platform at a time: AcruxCore vs Langfuse and AcruxCore vs LangSmith.
• More tools compared: 10 LLM observability tools tested hands-on, 8 LangSmith alternatives and 8 Langfuse alternatives.

This post compares Langfuse and LangSmith on the things a developer checks before choosing one. We also ran one app on both hosted platforms, with the same code, data, prompts and model, and measured each step. The app is a small agent that reads a client's email, checks each request against a signed project scope, prices the extra work, and writes a reply.

Differences at a glance​

Sources: the Langfuse license, pricing, self-hosting and alerts pages; the LangSmith pricing, self-hosting, regions, gateway and alerts pages; and the Langfuse MCP and LangSmith MCP pages. Prices were read on 16 October 2026, the MCP pages on 18 October 2026, and everything else on 17 October 2026.

Langfuse​

Pros

• Our app needed fewer lines of tracing and prompt-fetching code: 11, against 18 on LangSmith.
• The core is MIT-licensed and free to self-host.
• It has the larger community: 35,935 GitHub stars.
• 100,000 traces of our app cost $98.36 a month, less than a fifth of the LangSmith price.
• It keeps data for 90 days at that price.
• A model call shows its prompt name and version as a link.
• A prompt read with no label returns the production version, so a draft does not reach users.

Cons

• It has no gateway, so it cannot route model calls or stop them at a spend limit.
• Its comparison of two experiment runs cannot hide the examples whose score stayed the same.
• An experiment with a plain Python function runs one example at a time.
• The bill grows with the number of steps in each trace.
• A promotion asks for no confirmation, and the prompt page does not show past promotions.
• SSO needs a paid add-on, and the audit log needs Enterprise.

LangSmith​

Pros

• Its comparison of two runs filters to the examples whose score changed, in one click.
• An experiment with a plain Python function runs several examples at a time.
• It has SDKs for Go and Java as well as Python and TypeScript.
• It has a gateway, in beta.
• A promotion asks for a confirmation, and the prompt page keeps a rollback history.
• Alerts reach Slack, PagerDuty and webhooks.

Cons

• 100,000 traces of our app cost $567 a month.
• It keeps data for 14 days on that plan.
• The platform is not open source, and self-hosting needs the Enterprise plan.
• Pushing a prompt from code needs the langchain-core package.
• A prompt read with no tag returns the newest commit, so a draft can reach users.
• The SDK caches a prompt for 5 minutes, so a promotion takes up to 5 minutes to arrive.

Which one to choose​

• Langfuse if you want an open-source platform that you can self-host for free, a large community, and a low price with 90 days of data.
• LangSmith if you compare experiment runs often, need Go or Java, or want a gateway from the same vendor, and $567 a month fits your budget.

The setup​

The tables above come from vendor pages. The sections below come from running one real app on both platforms, so you can see how each platform behaves on a typical job.

The app's job​

A freelance developer builds a website for a yoga studio at a fixed price of $4,800. The signed scope lists what the price includes: five pages, a class schedule, a contact form, and 30 days of bug fixes. After the contract is signed, the client keeps emailing for more. The app reads each email, decides which requests the price already covers, and quotes the rest from a price list.

An example message and the answer the app should give:

The app answers in four steps:

1. Extract splits the message into separate requests.
2. Classify puts each request in one of four groups:
   • covered by the signed price;
   • extra work with a price on the price list;
   • extra work with no listed price, which needs an estimate;
   • unclear.
3. Price adds up the extra work. This step is plain Python, with no model call.
4. Reply writes the email to the client.

Why this app​

We picked this app because each feature in the comparison has a real job in it:

How we ran it​

• The code: the app is one file, pipeline.py, with no tracing code. Each platform's script passes three things to the app: a function that fetches a prompt, an OpenAI client, and a decorator that records each step in the trace. So the two runs differ only in these two scripts.
• The data: 30 client messages, each with its expected items and total.
• The model: every call, the judge included, went to the model glm-5.3 from the provider z.ai, through the OpenAI Python SDK.
• The platforms: the hosted EU region of each, on the free plans (Langfuse Hobby and LangSmith Developer).
• The environments: each platform's script ran in its own Python 3.12 environment, with openai 2.48.0 and either langfuse 4.14.1 or langsmith 0.10.10. Neither environment had a LangChain package.
• One exception: the script that pushes prompts to LangSmith needs langchain-core, so it ran in a separate environment.

Tracing​

• Both platforms stored the same tree: the agent at the root, one span per step, and the price step marked as a tool. Every step except price holds one model call.
• Nine of LangSmith's 18 lines read a prompt from LangSmith without LangChain.
• On Langfuse, you pass the prompt object to chat.completions.create, and the call links to that prompt version.
• LangSmith has no argument for that link when you call the model directly. We set the metadata keys lc_hub_repo and lc_hub_commit_hash, as LangSmith's own pull_prompt does, so each model call shows the prompt name and commit as plain text.

Do you need LangChain to use LangSmith?​

• LangSmith stores a prompt in LangChain's own JSON format, not as plain role and content messages.
• Without langchain-core, push_prompt raises ImportError: The client.create_commit function requires the langchain-core package. The prompt is still created, empty, before the error.
• Our push script runs in a separate environment with langchain-core 1.6.10. It builds a ChatPromptTemplate with template_format="mustache", so {{variable}} placeholders stay as they are.
• Langfuse stores a chat prompt as a list of {"role", "content"} messages, the shape the OpenAI SDK takes. Our JSON prompt files came back from Langfuse unchanged.

Prompts, promotion and rollback​

• After we pushed a draft v2, pull_prompt_commit("scope-extract") returned the draft. Code on LangSmith must ask for "scope-extract:production".
• The app code did not change for the promotion, and the next run used v2 on both platforms.
• The cache times are the defaults in the SDK code installed by pip.

Experiments over the dataset​

An experiment runs your app on every example in a dataset and scores each result. An example is one test message with its expected answer.

• The scorers: the items match the expected items exactly, the total matches exactly, and an LLM judge passes the reply.
• v1 and v2: v1 is the first set of prompts. v2 rewrites the extract and classify prompts from v1's mistakes. The reply and judge prompts stayed on v1, so both versions were scored the same way.
• The question: after a prompt change, how fast can you find the messages that the change fixed or broke?

• Langfuse calls a plain function inside its event loop, so max_concurrency has no effect.
• Each example waits for the examples queued before it, and Langfuse counts that wait in the example's trace latency.
• This async task runs 4 at a time on Langfuse:

async def task(*, item, **kwargs):
    return await asyncio.to_thread(app, **item.input)

• On the items score, v2 fixed 2 messages and broke 2 on LangSmith. On Langfuse, v2 fixed 4 and broke none. The screenshots below show how we found those messages.
• glm-5.3 does not return the same output twice, so the same prompts scored differently on each platform. These scores compare the two prompt versions, not the two platforms.
• Both comparison views measure v1 against v2. LangSmith shows two counts per score: ↑ for messages where v1 scored higher, ↓ for messages where v1 scored lower.
• Langfuse shows the same two counts, as "↗0 ↘4" for the items score. Clicking a count does not filter the table.
• The Langfuse averages above the table cover only the 20 rows on the first page.

Human review​

We put the 30 traces of each v1 run into a review queue and asked one question: "Would you send this reply to the client as is?"

• On LangSmith, create the feedback configuration first. create_annotation_queue names the configuration by its key and does not check that the configuration exists.
• Across both v1 runs, the items scorer marked 10 messages wrong. For 5 of those messages, we would send the reply unchanged, so the items scorer is stricter than a person.
• The total scorer agreed with us. Every reply with a wrong price failed it, and we marked each of those replies as needing a fix.

Live traffic​

We rolled both platforms back to v1 and sent the 30 messages to each, one at a time, during the same 10 minutes.

• The latency is almost all model time, because both called the same z.ai endpoint.
• We sent this traffic on 16 October 2026, 06:03–06:13 UTC. The model's speed changes during the day. At 17:18 UTC the same app had a p50 of 7.2 s on Langfuse and 7.6 s on LangSmith, so compare the two columns with each other, not with latencies from another hour.
• The model cost is z.ai's bill for one message, not a platform fee. It differs because the model wrote replies of different lengths.
• We entered z.ai's prices: $1.40 per million input tokens, $0.26 per million cached input tokens, and $4.40 per million output tokens. Both platforms priced cached tokens at $0.26.
• Both platforms have dashboards that work with no setup. Langfuse's home page shows latency percentiles per trace, per model call and per step. LangSmith's trace table shows latency and cost in each row.

What 100,000 traces a month costs​

• Prices come from langfuse.com/pricing and langchain.com/pricing, read on 16 October 2026, using each page's own calculator.
• Langfuse calls each step and each model call an observation. Our live traces averaged 9.67 units: 1 trace and 8.67 observations. 100,000 traces are 967,000 units.
• Our live traces had no scores. Scores were added only in the experiments.
• The Langfuse bill grows with the observations in a trace. An agent with 30 observations per trace would use about 3.1 million units for the same 100,000 traces.
• Free plans: Langfuse Hobby has 50,000 units a month, 2 users and 30 days of data. LangSmith Developer has 5,000 traces a month and 1 seat.

Run it yourself​

• The scripts folder on GitHub: the agent, both platform scripts, the dataset, both prompt versions, a README with the commands, and RUNLOG.md with the raw output of every step.
• Both platforms against AcruxCore, on the same app: AcruxCore vs Langfuse vs LangSmith.
• Each platform against AcruxCore: AcruxCore vs Langfuse and AcruxCore vs LangSmith.
• More tools compared: 10 LLM observability tools tested hands-on, 8 LangSmith alternatives and 8 Langfuse alternatives.
• What self-hosting Langfuse takes next to six other self-hosted tools: Self-hosting LLM observability: seven open-source tools measured.
• At a glance: the comparison page puts AcruxCore next to Langfuse and five other self-hostable platforms, feature by feature.

Braintrust is not open source, and self-hosting needs the Enterprise plan. There is no free self-hosted install to test against, so everything below ran on their hosted Starter tier.

At a glance​

Full breakdown, screenshots, and the verdict below.

Evaluation and datasets​

Evaluation is what Braintrust is built for, and it shows most clearly here.

An evaluation declares the data, the task and the scorers in one place, and the CLI runs it:

Eval(
    "My Project",
    data=lambda: DATASET,
    task=task,
    scores=[mentions_every_ticket, Factuality],
)

mentions_every_ticket is ordinary Python that checks whether the reply names each of the row's open ticket numbers. Factuality is an LLM judge from autoevals, Braintrust's own scorer library. Both kinds of scorer are declared the same way in the same list, so the cheap check costs no more effort to add than the paid judge does.

Running braintrust eval prints a summary that compares the run against the previous experiment. You do not ask for the comparison; it is the default output:

vip-support-triage-a64bd2f5 compared to vip-support-triage-55989b5e:
100.00% (+53.33%) 'Factuality'            score	(2 improvements, 0 regressions)
100.00% (+66.67%) 'mentions_every_ticket' score	(2 improvements, 0 regressions)

In the dashboard every run of this eval appears on one trend line, and the table below it carries every scorer as its own column.

Opening a run shows the per-row scores beside a score distribution for each scorer. The column header carries the delta against the base experiment, +53% and +67% here.

Scorers are also a resource with their own page, and a scorer can be created in the UI, either as an LLM judge or as code, with six templates ready to use. Tools cannot, as the next section shows.

AcruxCore runs evaluations server-side against a stored dataset, not from a CLI on your machine. AcruxCore also covers one case Braintrust's experiment model does not: online rules score a sample of live production traffic instead of a fixed dataset. Scoring live traffic and iterating on a dataset are two different jobs.

How tools are handled​

The Tools page answers one question before you have created anything, which is where a tool can be written:

So the tool is declared in a file and pushed:

project.tools.create(
    handler=search_flights,
    name="search_flights",
    slug="search-flights",
    parameters=SearchFlightsArgs,   # a pydantic model
    if_exists="replace",
)

braintrust push bundles the module and uploads it. What lands is not a schema pointing at your service, but the code itself:

"function_data": {
  "data": {
    "type": "bundle",
    "runtime_context": { "runtime": "python", "version": "3.13" }
  }
}

And the code executes. POST /v1/function/<id>/invoke with the schema's fields runs the handler on Braintrust's infrastructure and returns the rows:

{ "origin": "Amsterdam", "destination": "Lisbon", "count": 3,
  "flights": [ { "flight_no": "KL1693", "airline": "KLM", "price_eur": 189 }, … ] }

The tool page shows the uploaded source. Two lines on that page say what running on someone else's infrastructure costs you.

This function was bundled and uploaded, so you cannot edit it here.

Code runs in a sandboxed environment with restrictions on imports and filesystem access

That sandbox has two consequences. The first is that the flight inventory had to be inlined into the pushed file instead of read from ../data/flights.json, because the handler does not run where the data is. In the tutorial version, the same function reads that file without trouble, because the function runs in your process.

The second consequence is the pinned runtime, and the push is refused outright:

Unsupported python version "3.14". Supported versions: 3.8, 3.9, 3.10, 3.11, 3.12, 3.13

The refusal is not a bug: if Braintrust runs the code, Braintrust picks the interpreter.

One thing did not work. The tool page has a Run panel with a Test button, and that button could not call search_flights at all. Two different YAML shapes for the arguments were tried, both returned the same error, and the REST API accepted the identical payload and returned the three flights.

search_flights() missing 2 required positional arguments: 'destination' and 'departure_date'

The panel passes the whole input object as one positional argument. The API spreads that same object into keyword arguments. A test button that calls the handler differently from the real path is worse than no test button, because a green result would prove nothing.

AcruxCore stores the contract and leaves the choice of runner to you. A client executor keeps both the code and the data in your process, and the platform holds only the name, description and schema. An http executor calls an upstream service server-side instead. The catalog then reports per-tool volume, error rate and latency.

Neither choice is wrong. Braintrust's choice is better when you want a tool to be callable by anything that can reach the platform, with nothing of yours deployed. AcruxCore's choice is better when the tool needs your database, your filesystem, your VPC or your Python version, which is the usual case for a tool built in-house.

Prompt authoring and versioning​

Braintrust's authoring UI is ahead of ours. Braintrust's Python SDK then renders the result with a different engine from the one the prompt declares.

Each prompt picks its own template engine: Mustache, Jinja, or no templating at all. The editor notes that the web runtime uses nunjucks wherever a prompt says Jinja. The "no templating" option is one we do not have. In AcruxCore, a prompt full of JSON examples or LaTeX has to have its braces escaped by hand.

Because the engine is real nunjucks, the vip-support-triage prompt went in verbatim: the conditionals and the {% for %} loop over tickets, unchanged. No competitor in this series has taken that prompt unchanged before. Every previous one forced the template to be flattened into plain text first.

The version history is better than ours in two specific ways. The diff covers the entire prompt object: the model, the params, endpoint_name and template_format all appear in it, not only the message text. And you can compare any two versions, not only adjacent ones, then restore one with "Set as editor contents".

A model change alters behaviour without changing a word of the message, so our message-text diff would miss that edit. The whole-object diff and the any-two-versions comparison are both on the list of things to borrow from Braintrust.

Then the Python SDK renders that prompt with a different engine. Loading the same stored prompt through the Python SDK and calling build() produces this:

You are a support triage agent for Acme Corp.
{% if is_vip %}This customer is VIP — prioritize them…{% else %}…{% endif %}
{% if tickets and tickets.length %}Open tickets:
{% for ticket in tickets %}- #:
{% endfor %}{% else %}No open tickets.{% endif %}

{{ company }} resolved, because mustache and nunjucks agree on that syntax. Every {% %} block passed through as literal text, so the model received both branches of every conditional. And {{ ticket.id }} and {{ ticket.title }} became empty strings, so the ticket numbers and titles vanished while the "Open tickets:" heading above them stayed. Braintrust raised no exception and printed no warning, and the model answered plausibly.

The cause is in the source. In braintrust==0.39.0, template_format appears only inside generated type stubs, and Prompt.build() calls render_mustache() unconditionally.

Their Node SDK reads the same field and refuses to render rather than guessing:

Error: Nunjucks templating requires @braintrust/template-nunjucks.
Install and import it to enable templateFormat: 'nunjucks'.

Refusing is better than rendering with an engine the prompt never asked for. The package that message names currently 404s on npm, so the instruction it gives cannot be followed today.

AcruxCore renders nunjucks server-side in prompts.render(), so there is one renderer and no client-side engine that can disagree with it. The diff is narrower than Braintrust's, covering message text between adjacent versions.

Sending a live call​

The prompt page runs the stored prompt with values taken from a variables panel, and Braintrust builds that panel from the template itself. The panel inferred tickets.0.id, tickets.0.title and tickets.length without being told. Filling the panel in and sending produces a real completion, and in this screenshot the web runtime renders the nunjucks correctly, naming both tickets:

That reply and the SDK output above came from the same stored prompt with the same variables.

The playground does not show what the call cost. Tokens, cost and duration are one click away behind "View trace". AcruxCore shows all three on the run itself.

Tracing and observability​

Braintrust's tracing is explicit. Nothing is recorded until you both name a destination and instrument the client:

logger = init_logger(project="My Project")
client = wrap_openai(OpenAI(base_url="https://api.braintrust.dev/v1/proxy", api_key=KEY))

With those two lines the model call is captured in full: the system message, the user message, the tokens, the cost and the duration.

A parent span you create yourself behaves differently. Wrapping the call in logger.start_span() produces a root span whose input reads null, because a manual span records only the fields you write to it. The child LLM span is complete; the root span above it carries nothing but a name.

The log view carries a duration histogram over time, p50 columns for duration, LLM duration and time to first token, and failures highlighted in red. The failed search_flights test from earlier appears in the log list as a trace of its own.

AcruxCore writes the span from the gateway, server-side, so the script contains no tracing code. The span also carries a link back to the prompt version that produced it.

Where the platform sits​

A platform can sit in the request path, or beside it and only receive what the application reports. Braintrust offers both and makes you choose. Its AI proxy at api.braintrust.dev/v1/proxy sits in the request path, and that proxy resolved gpt-4o-mini against the org's OpenRouter credential while the script never knew which provider would serve the call. But wrap_openai() around a direct client works too, and then Braintrust only receives.

AcruxCore is always in the request path, and for completions there is no beside-it option. Routing, caching, budgets and virtual keys apply because the call goes through the gateway. The cost of always being in the path is a hop you cannot skip. The benefit is that those controls apply whether or not the application remembered to ask for them.

SDK and developer experience​

Both platforms' Python SDKs completed every task in this post. The Braintrust CLI does more: braintrust push bundles Python and uploads it, and braintrust eval runs experiments and prints a comparison. Those two commands are also where the friction was.

braintrust push needs the [cli] extra for uv. The same command refuses to run from a path whose folder names contain hyphens, because it turns that path into a dotted module name. And it refuses Python 3.14. Each of those three is small, and together they cost about twenty minutes.

The templating mismatch above is the larger cost, and it is a developer-experience problem as much as a rendering one: the surface you author on and the surface you deploy from disagree.

The AcruxCore script is shorter, because rendering and tracing both happen server-side:

rendered = await hub.prompts.render("vip-support-triage", "production", VARIABLES)
result = await hub.gateway.chat(rendered.model, rendered.messages, prompt_version_id=rendered.version_id)

All four scripts are committed and runnable: bt_sdk_run.py, bt_eval_run.py, bt_tool_push.py and acx_sdk_run.py.

Latency overhead​

There is no benchmark number in this post.

Every previous benchmark in this series compared a self-hosted competitor on localhost against AcruxCore on localhost. Braintrust cannot be self-hosted on a Starter account, so any timing would measure the round trip to their datacenter against a loopback call. Such a timing measures the distance between the two machines, not either product's overhead. Interleaving the runs does not remove that distance; it only spreads it evenly.

Their Docker data plane does run locally, but it cannot be attached to a Starter organization. The next section shows the setting that would attach it. So a Starter reader could not reproduce a benchmark here even if this post published one.

Licence, self-hosting and plans​

Braintrust is not open source. Parts of the tooling around it are published: the JavaScript SDK is Apache-2.0 and autoevals is MIT. The hosted platform itself is not something you can read or fork. AcruxCore is Apache-2.0 in full.

Self-hosting is Enterprise-only, and the product says so itself. The containers are not the obstacle. braintrustdata/braintrust-deployment brings up Redis, Postgres and a standalone-api container, all healthy in about 25 seconds, with no licence key and no S3 bucket, and GET :8000/status returns a real Braintrust ForbiddenError, so the API is live.

The deployment is still unusable, because those containers are only the data plane. The UI, authentication and metadata stay on braintrust.dev. The only way to attach your own data plane is to set a custom API URL on the organization, and on a Starter account that setting is not a field at all:

Their docs state the restriction twice more: "Self-hosting is only available on the Enterprise plan" and "Configuring custom data plane URLs is only available on the Enterprise plan."

Access control starts at $249/month. The Starter tier has no roles at all. RBAC appears as "Basic roles" on Pro, and as custom roles on Enterprise. Audit logs are Enterprise-only, which the dashboard states plainly:

Environments is Pro and above. Environments lets you tag a prompt version as production, staging or development and then pull it by name, which is the same job AcruxCore's production and staging aliases do. Our aliases are in the free self-hosted product.

Sources: braintrust.dev/pricing and the organization settings pages above, all checked 2026-09-13.

What Braintrust does that AcruxCore doesn't​

A SQL sandbox over your own data​

Braintrust ships a query editor over experiments, logs and datasets. This is real SQL against the experiment created earlier, returning in 0.41s:

SELECT input.customer_message AS message, scores.mentions_every_ticket AS ticket_score,
       scores.Factuality AS factuality
FROM experiment('a14e0c95-5d81-411b-b2d6-b3856df5bcf7')
WHERE scores.Factuality IS NOT NULL

Our analytics pages answer a fixed set of questions, and you cannot write your own query against the underlying data. Braintrust's schema does take learning, though. Scores and inputs sit on different rows of the same experiment, so a query that selects both at once returns half-empty columns until you know that.

Loop, an agent inside the dashboard​

An earlier attempt at that query failed, and the error came with a Fix with Loop button. Loop took the broken query as context and said what it intended to do. Loop then called its own tools to inspect the project and proposed a rewrite as a red/green diff, with Accept, Skip and an auto-accept toggle.

The answer was wrong. Loop's rewrite dropped the FROM clause, and the query it produced failed in a new way. Braintrust's own error message then named the BTQL syntax the query needed, which made the error more useful than the agent. The interaction is still worth copying: propose, show a diff, ask before applying. AcruxCore has nothing that works that way.

And several more, not driven here​

Their project sidebar also carries Patterns, Topics, a human Review queue, Automations, Remote evals, Preprocessors, Span iframes and a project-level MCP page. AcruxCore has an equivalent for two of the eight. Automations overlaps our online evaluation rules, and AcruxCore has its own MCP server. AcruxCore has nothing for the other six. All eight are listed rather than described, because none of them was driven hands-on for this post.

What AcruxCore does that Braintrust doesn't​

Braintrust's proxy and AcruxCore's gateway both sit in the request path, so the fair question is what each one is allowed to do while it is there. Two controls are in AcruxCore and not in Braintrust. Both were checked against Braintrust's own documentation rather than against our feature list, which would have turned up only what we already build.

Budgets that stop the call​

An AcruxCore budget is a cap in dollars, set per virtual key and per period. The gateway reserves the estimated cost of a call before the provider is contacted, and a request that would cross the cap is refused with 402 BUDGET_EXCEEDED rather than sent.

Braintrust has spend alerts, and their documentation says plainly what those do:

Spend alerts are informational-only and do not interrupt or limit your usage.

Their alerts also watch a different number. They fire on the Braintrust invoice for the whole organization, at up to three thresholds, by email or Slack. An AcruxCore budget caps model spend on one key.

Per-key request and token limits​

A virtual key in AcruxCore carries an optional requests-per-minute and tokens-per-minute limit, and the gateway checks both against a trailing 60-second window before the call goes out. A caller over either limit gets a 429 with Retry-After. That window lives in the API process today, so a deployment running several API instances enforces the limit per instance rather than across all of them.

Braintrust rate-limits its own API endpoints per organization, which protects their service rather than your spend. Their organization settings expose no rate limit you can set on model calls.

What was checked and ruled out​

Four controls that look like ours alone are not. Braintrust's AI proxy caches responses, and their documentation puts it as "The proxy automatically caches results and reuses them when possible". The same proxy issues temporary credentials that do the job virtual keys do, and it load balances across several keys for one model. Braintrust organizations hold provider credentials and per-function environment variables. None of those four belongs in this section.

Sources: the AI proxy guide, spend alerts and organization settings, all checked 2026-09-13.

Verdict​

If the job is to iterate on prompt quality against a dataset, Braintrust does it better than we do. Braintrust's evaluation model is the part of this comparison we learned most from, and its prompt authoring UI is ahead of ours as well.

One thing holds that authoring win back. Most calls reach a platform through its SDK, and Braintrust's Python SDK renders a stored prompt with an engine the prompt does not declare, deleting data as it goes and raising no error. Until the SDK reads template_format, the better editor only helps on the screen where the prompt was written.

Tools are the row where the two products differ most and neither is behind. Braintrust's answer costs you a sandbox, a chosen interpreter and a copy of your data, and buys a tool anything can call with nothing of yours deployed. Ours costs you the work of running the tool, and buys a tool that still reaches your own database and filesystem.

For the six open-source alternatives we have compared against, see the compare page.

To compare more tools, see 10 LLM observability tools tested hands-on, 8 LangSmith alternatives, 8 Langfuse alternatives and 10 Braintrust alternatives.

Start here if you want to try AcruxCore: Quickstart.

At a glance​

Full breakdown, screenshots, and the verdict below.

Tracing an agent run​

This is what Laminar is for, so it goes first. Laminar.initialize() patches the OpenAI client through OpenTelemetry, and an @observe() decorator nests the model call underneath a parent span — so the run's shape lands in the trace, not just the call.

from lmnr import Laminar, observe

Laminar.initialize(project_api_key=os.environ["LMNR_PROJECT_API_KEY"], base_url="http://localhost:8000")

@observe(name="vip_support_triage")
def triage(message: str) -> str:
    return client.chat.completions.create(model="gpt-4o-mini", messages=[...]).choices[0].message.content

The span view carries the full system and user messages, token counts, cost, and an "Experiment in playground" button that lifts the call straight into their playground.

AcruxCore has two ways in. The gateway writes a span server-side, with no tracing code in the script at all, and links it back to the prompt version that produced it.

For the run's shape there is an OTLP/HTTP receiver — POST /api/v1/traces/otlp, protobuf or JSON, gzip accepted — that takes nested OTel spans from your process, the same way Laminar's does. Both SDKs collapse the provider, processor and exporter wiring into one register() call, which can also switch on a framework's own OpenInference instrumentor:

from acruxcore.otel import register

register(api_key=KEY, base_url=BASE_URL, instrument=["crewai", "openai"])

Five frameworks are named today — OpenAI, OpenAI Agents SDK, CrewAI, LangChain and LlamaIndex — and any other OTel or OpenInference instrumentor can be wired to the same provider by hand. A CrewAI crew traced this way arrives as a real tree: a Crew.kickoff root, one agent span per crew member, with tool and LLM spans nested underneath.

So the difference is breadth and reading, not whether the run is visible. Laminar ships three times the integrations and reads them back as a transcript, a cost heatmap and a realtime feed. We ship five named frameworks, a span tree with a latency waterfall, and prompt-version lineage on the gateway span — and unlike theirs, ours exists even when the caller sends no spans at all.

Where the platform sits — in the request path, or beside it​

Laminar sits beside the request path. Its Rust app-server receives spans; it never receives your model call. We checked the server's routes: the only /chat/completions handling in the codebase is its outbound client for its own AI features, not an inbound proxy.

That is a real design choice with real advantages. Any provider works immediately, there is no proxy to configure or keep up, and nothing Laminar does can fail your production call. The cost is that a budget, a cache hit or a virtual key has no call to act on — by the time Laminar sees the request, the money is already spent.

AcruxCore takes the other side. The call goes through us first, which is what makes a spend cap enforceable: reserveBudgets() runs before the provider is contacted and returns 402 BUDGET_EXCEEDED, and a per-virtual-key RPM/TPM limit returns 429 RATE_LIMITED on the same pre-call path. The cost is the hop, measured in the next section, and the fact that we have to support your provider.

Latency overhead — measured​

100 rounds per run, interleaved in rotating order so a network blip lands on all three paths equally, three warm-up rounds discarded, every leg ending at gpt-4o-mini on api.openai.com with the same key and the same body. A gateway model is only a public name pointing at a credential, so the script resolves ours before it starts and refuses to run unless it refers to a native OpenAI credential — that way all three legs provably share one upstream.

A single run is not a result here: the median gap moves enough between runs that one number would be misleading. So it ran three times.

Laminar's overhead lands between −8 ms and +18 ms, AcruxCore's between +7 ms and +27 ms. In two of the three runs every confidence interval crosses zero, which means neither path is statistically distinguishable from calling the provider directly at this sample size. Laminar's median gap was the lower of the two in all three runs, by roughly 10 ms — consistent enough to be real, small enough that no application would notice it.

Both are cheap, and they are different kinds of cost: Laminar's is client-side instrumentation that batches spans off the hot path, ours is a real extra network hop that buys routing, budget enforcement, virtual keys and caching.

Full script: latency_bench.py.

Querying your trace data​

Laminar's SQL editor is the feature we most wish we had. It is not a filter builder — it is SQL over the ClickHouse span store, with table, JSON and chart output, saved queries, an "Ask AI" button that writes the query for you, and CSV export.

The dashboard is composable in the same spirit: drag-and-drop cards, resizable, each one backed by a metric or a custom SQL query. After the benchmark run it had 135 spans to chart.

AcruxCore has neither. Our analytics page groups by exactly one of day, model, session or prompt_version and renders a fixed set of tiles and charts; the state lives in the URL, so you can share a view as a link, but you cannot compose one. There is no way to ask our trace store an arbitrary question.

Prompt authoring and versioning​

Here the shapes diverge completely. Laminar has no prompt registry — we checked the database schema rather than only the sidebar, and there is no prompts table at all. The nearest thing is playgrounds, one row per playground holding prompt_messages as a mutable JSONB blob. Editing it overwrites it. There are no version numbers, no aliases, no diff, and no template variables: their only templating dependency, mustache, is used for rendering span previews, not prompts.

So the fixture prompt had to be flattened by hand for Laminar — the {% if is_vip %} branch resolved to its VIP text, and the {% for ticket in tickets %} loop expanded into a literal list of two tickets. That flattened string is what lm_trace_run.py inlines, because there is nothing on the platform to fetch it from.

On AcruxCore the same prompt is three immutable versions with production on v3 and staging on v2, a unified diff between any two, and the conditionals and loop rendered by nunjucks at request time.

Sending a live call​

Laminar's playground stores messages, a model, tools and an output schema, and can be opened directly from a traced span — a genuinely nice loop for debugging a call you just saw. It needs a provider key saved in project settings first, and that page offers a fixed list: OpenAI, Gemini, Groq, Anthropic, Mistral, Azure and Bedrock, with no field for a custom base URL. We confirmed the list in their source (MODEL_PROVIDER_TO_API_KEYS in frontend/lib/env/utils.ts); keys are stored encrypted per project rather than read from the environment.

With an OpenAI key saved, it runs. Below is the flattened prompt returning a real completion on GPT-4o mini. Two things to notice. The messages are typed in flat, because a playground row has no variables — the VIP sentence and the ticket list are pre-rendered by hand, the same flattening described above. And the footer reports input, output and total tokens, but no cost — no latency and no cache status either.

The limitation that stands is the provider list: seven named vendors and no custom base URL, so your own gateway, a relay or a local model server cannot be called from it at all.

AcruxCore's playground runs the stored prompt against any model the gateway knows, and shows cost, cache status and latency inline with the response.

How tools are handled​

Laminar records tool calls as spans and stores a tool schema as a tools JSONB field on a playground row. There is no tool catalog page, no version history for a tool, and nothing in Laminar executes one — your agent does, and Laminar watches. That is consistent with sitting beside the request path.

AcruxCore's Tool Catalog is the opposite: a tool is a first-class, versioned resource that the gateway itself calls, with a page measuring per-tool volume, error rate and P50/P95.

Evaluation and datasets​

Laminar's evaluations are unopinionated and code-first: you supply the data, an executor function and scorer functions, and it runs them locally or in CI, tracing every call and tracking scores across runs. We ran one over three triage cases.

evaluate(
    data=DATA,
    executor=executor,
    evaluators={"mentions_topic": mentions_topic, "under_five_sentences": under_five_sentences},
    name="vip-triage-v1", group_name="triage",
)

The results view puts the per-row scores beside the trace of each datapoint, with the evaluator functions themselves showing up as spans under the executor.

Datasets are built from the traces you already have — "Add to dataset" on a span turns it into a Data/Target row with no manual copying.

AcruxCore runs evaluations server-side instead — datasets, LLM judges and experiment runs from the dashboard, plus rules that score live production spans automatically. A rule is plain-English criteria plus a judge model, matched against incoming llm spans by prompt, alias, model or tags, sampled (10% by default) and capped at a daily limit, producing a 0–100 score with a reason and emailing owners when a score drops below a threshold.

The honest difference: Laminar's scorers are arbitrary code and can assert anything, but somebody has to run them. Ours run themselves against real traffic, but they are LLM judges producing a sampled numeric score, not arbitrary assertions.

SDK and developer experience​

Laminar's onboarding is the most coding-agent-first we have seen. The empty traces page hands you a prompt to paste into your coding agent; npx lmnr-cli setup authenticates, writes a project key to .env and installs a "Laminar skill"; and the CLI can query your traces in SQL so the agent can verify its own instrumentation.

AcruxCore's SDK is aimed at a different job — the prompt lives on the platform, so the script fetches and fills it, sends it through the gateway, and never mentions tracing:

async with AcruxCore(api_key=KEY, base_url=BASE_URL) as hub:
    rendered = await hub.prompts.render("vip-support-triage", "production", VARIABLES)
    result = await hub.gateway.chat(rendered.model, rendered.messages, prompt_version_id=rendered.version_id)

Both scripts are committed and were run before this post linked them: lm_trace_run.py and acx_sdk_run.py.

Both MCP servers let a coding agent read your traces. Through Laminar's server, a coding agent can also run SQL over your spans. AcruxCore's server has no SQL tool. Through AcruxCore's server, a coding agent can commit a prompt version, promote an alias and run the stored prompt through the gateway. Laminar's server is read-only, so a coding agent cannot change anything in Laminar through it.

What Laminar does that AcruxCore doesn't​

The two biggest ones — SQL over spans and the custom dashboard builder — have their own section above. Four more, each checked against our own codebase before being written down.

Labeling queues. A span can be pushed into a named queue where a reviewer works items one at a time against a defined annotation schema, then pushes the labelled result to a dataset. AcruxCore has trace feedback — a human can score a trace from −1 to 5 with a label and comment — but there is no queue: no assignment to a reviewer, no next-unlabelled-item workflow, no rubric, no completion tracking.

Signals. You describe a behaviour in plain English, give it a structured output schema, and an LLM watches traces for it — with templates for failure, logic, task, user friction, hallucination and intent, a trigger on trace or span completion, filters, and sampling. Matching events are grouped into clusters and can page you in Slack.

This one needs a caveat we will not soften. AcruxCore's online evaluation rules cover much of the same ground — plain-English criteria, an LLM judge, automatic matching against live spans, sampling, daily caps and email alerts — so this is not a capability we lack. What we lack is the event-and-cluster model on top: their output is a structured event that gets clustered into behavioural patterns, ours is a 0–100 score with a reason. Also worth saying plainly: on the lite self-hosted stack, after enabling Signals and running a fresh trace through it, no event was produced within our observation window. The builder is real and we screenshotted it; the end-to-end result we could not confirm.

The debugger. LMNR_DEBUG=1 lets a coding agent run your agent, read the resulting trace, change the code and re-run with cached state — the iteration loop owned by the agent rather than by you. AcruxCore has nothing equivalent. AcruxCore's MCP server lets a coding agent read a trace and run a stored prompt, but that server cannot re-run your agent with cached state.

PII redaction. A project-level toggle runs every ingested span through a PII redactor before storage, replacing detected names, emails and phone numbers. AcruxCore always scrubs a fixed set of secret patterns — API keys, bearer tokens, AWS access keys and email addresses — and has a team-level switch to stop capturing payloads at all, but nothing that detects names, phone numbers or addresses.

Two more worth naming without a screenshot, because they are structural: Laminar delivers alerts to Slack, while AcruxCore's only notification channel is email; and Laminar records browser-agent sessions for Browser Use, Stagehand and Playwright, which we have no answer for at all. Our "Sessions" means traces grouped by a caller-supplied session id — a conversation thread, not a recording.

Is AcruxCore a Laminar alternative?​

For part of what Laminar does, no — and that is the most useful thing this comparison found. Both of us take nested OTel spans, so seeing an agent run is not the dividing line. Digging into one is: SQL over your own span store, a composable dashboard, labeling queues, a transcript view, and three times the framework integrations. We do not replace that depth, and this post does not pretend otherwise.

Where we are a genuine alternative is everything that happens before the call. Laminar has no prompt registry, no request path, and therefore no way to enforce a spend cap or serve a cached response. If those are the things going wrong for you, that is the swap worth making.

They also compose, which is the answer most people actually want: point your client at our gateway and keep Laminar's SDK instrumenting the run. Nothing about the two conflicts.

Verdict​

These two tools barely overlap, which is the most useful thing this comparison found. Laminar starts at the trace and builds upward — SQL, dashboards, labeling, signals, a debugger — and it beats us clearly on querying trace data, and on the breadth and reading of traces — both platforms take nested OTel spans, but they have three times the integrations and far more ways to read them back. On measured latency the two are close — both sit within tens of milliseconds of a direct call, with Laminar consistently the lower of the two by about 10 ms. AcruxCore starts at the prompt and the request — versions, aliases, templates, a gateway that can refuse a call that would blow a budget — and Laminar has no answer for any of that, by design.

If you already run Laminar and your prompts are drifting across a team, the two compose fine: point your client at our gateway, keep Laminar's SDK instrumenting the run.

Licence, pricing, team model, security and community stats for both — with a source link and a checked-on date for every fact — are on the compare page.

To compare more tools, see 10 LLM observability tools tested hands-on, 8 LangSmith alternatives and 8 Langfuse alternatives.

Start here if you want to try the other side: Quickstart.

At a glance​

License, pricing, team structure, security, and community stats: see AcruxCore vs MLflow on the compare page.

Full breakdown, screenshots, and the verdict below.

Where the platform sits — in the request path, or beside it​

Every other observability-first platform we've compared sits beside the request path: your own client calls the provider, and the tool ingests a trace after the fact. MLflow doesn't. Its AI Gateway is a real proxy: you create a named endpoint, pick a provider and model, and get back an OpenAI-compatible URL. Every call through it is usage-tracked and traced server-side, the same shape as AcruxCore's own gateway.

We created the endpoint, then called it with a plain curl — no SDK, no auth header beyond what the endpoint already stores — and got a real completion back with "provider": "openrouter" in the response. AcruxCore's own request-path gateway is one sentence here; see the walkthrough for the picture.

The endpoint also ships an aggregate Usage dashboard — requests, p50/p90/p99 latency, token usage, and cost broken down by model — that neither of the other four platforms we've compared has an equivalent of.

Spend controls — both platforms stop the call​

Sitting in the request path is what makes a spend cap enforceable rather than advisory, and both platforms use it. MLflow's Budgets page (under AI Gateway) creates a policy per endpoint: a reset period, an action for when it's exceeded, and a spending window, tracked against real current spend. No policy existed on this build, so this is a comparison of the two configuration surfaces, not of two enforcement runs.

AcruxCore's cap is scoped to a team or to a single virtual key, over a day, week, month, or the key's whole lifetime. Enforcement happens before the provider is called: the request reserves its estimated cost against the cap, and one that would cross it comes back 402 BUDGET_EXCEEDED instead of being billed. Owners and admins get an email once at 80% of the cap and again when it's exhausted. Separately, a virtual key can carry RPM and TPM limits, which return 429 on the same pre-call path.

Prompt authoring & versioning​

MLflow's Prompt Registry has no "create" button anywhere in the UI — the empty state literally offers "Get help registering a prompt" and "Copy for coding agent." Every prompt starts life as a mlflow.genai.register_prompt() call.

What that call can hold was the real surprise here: MLflow's templates are full Jinja2, not flat {{variable}} substitution. We registered the fixture's actual {% if is_vip %}...{% else %}...{% endif %} and {% for ticket in tickets %}...{% endfor %} logic verbatim — no flattening needed, the first competitor in this series where that's been true.

MLflow also has real @production / @staging aliases — set_prompt_alias() — the same functional idea as AcruxCore's aliases, just set via SDK call rather than a UI button. AcruxCore's own prompt editor is one sentence here; see the walkthrough for the picture.

Sending a live call​

We sent the rendered prompt through the Gateway endpoint's OpenAI-compatible /gateway/mlflow/v1/chat/completions route. The trace shows tokens, latency, and the full request/response — but no cost figure on the individual trace, even though the aggregate Usage dashboard above computes cost just fine. Cost lives at the dashboard level, not the per-call level.

Tracing & observability​

Every Gateway call is traced automatically — that part is a genuine, automatic side effect, same as AcruxCore. What's not automatic is the link back to the prompt version: MLflow's trace-to-prompt link needs an explicit client.link_prompt_versions_to_trace() call after the request completes. We wrote a small script (mlflow_gateway_run.py) that renders the prompt locally, calls the Gateway, then makes that separate linking call — and it shows up correctly under the trace's Linked prompts tab.

How tools are handled — schema registry, or execution​

MLflow's answer here is an MCP Registry (Beta): you register an external Model Context Protocol server by pasting its server.json manifest — name, source repo, icon, tags. It's a discovery catalog for MCP servers your agents can connect to, not a schema-and-execution catalog for individual tools.

Neither model is strictly better — they answer different questions. MLflow's registry answers "which MCP servers exist and are they reachable?" AcruxCore's catalog answers "what did this specific tool call cost, and how often does it fail?" Nothing in MLflow's MCP Registry executes a tool call or records its latency; nothing in AcruxCore's catalog discovers external MCP servers.

Evaluation & datasets​

MLflow ships built-in LLM-as-judge and custom code judges from an empty state with two clear entry points, plus dataset creation from the UI or from any trace via "Add to dataset."

One real bug hit hands-on: after creating the dataset and naming it, the Datasets list page kept showing its empty "Create an evaluation dataset" state — reloading didn't fix it. The dataset existed the whole time (confirmed via mlflow.genai.datasets.search_datasets(), and its own detail page rendered fine when linked directly) — the list view just never picked it up in this build.

AcruxCore's dataset — built from a real feedback row — is one sentence here; see the walkthrough for the screenshot.

SDK & developer experience​

MLflow's SDK has dedicated surfaces for each step — load_prompt(), start_span(), and a separate link_prompt_versions_to_trace() call. AcruxCore's two calls (hub.prompts.render(), hub.gateway.chat()) do the same three things — render, call, trace-and-link — in one round trip.

prompt = mlflow.genai.load_prompt("prompts:/vip-support-triage@production")
rendered = prompt.format(**VARIABLES)

with mlflow.start_span(name="vip-support-triage-gateway-call") as span:
    resp = requests.post(f"{TRACKING_URI}/gateway/mlflow/v1/chat/completions",
                          json={"model": "vip-support-triage", "messages": rendered})
    trace_id = span.trace_id

mlflow.MlflowClient().link_prompt_versions_to_trace(
    trace_id=trace_id, prompt_versions=[prompt]
)

rendered = await hub.prompts.render("vip-support-triage", "production", VARIABLES)
result = await hub.gateway.chat(
    rendered.model, rendered.messages,
    temperature=0, max_tokens=256,
    prompt_version_id=rendered.version_id,
)

Real output, both platforms:

{
  "content": "I apologize for the inconvenience. I'll prioritize your request and have our support team review your open ticket #4821 regarding the billing export button...",
  "model": "openai/gpt-4o",
  "prompt_tokens": 95,
  "completion_tokens": 45,
  "elapsed_s": 2.232,
  "trace_id": "tr-50862fac52f978588cca2bc167fc3aa2"
}

{
  "content": "I apologize for the inconvenience with the billing export button. I will escalate this issue immediately...",
  "model": "openai/gpt-4o-mini",
  "prompt_tokens": 100,
  "completion_tokens": 51,
  "cost_usd": 0.0000456,
  "trace_id": "a5a2324c-4c48-4e40-8bfb-2b8379772d67"
}

Full scripts: mlflow_gateway_run.py and acx_sdk_run.py.

Latency overhead — measured​

We timed the identical call three ways, interleaved in rotating order over 100 rounds so a network blip hits all three equally: a raw direct call to the provider (baseline), the same call through MLflow's AI Gateway endpoint, and the same call through AcruxCore's gateway. Both gateways were pointed at a native OpenAI provider on gpt-4o-mini, and the script refuses to start unless AcruxCore's model resolves to an OpenAI credential — so all three legs provably end at the same host. It ran three times.

This is the one row in this series where the two gateways are not close. MLflow's overhead lands between +135 ms and +225 ms and no confidence interval crosses zero in any run, so it is a real cost rather than measurement noise. AcruxCore's lands between +21 ms and +43 ms — also real, and roughly five times smaller. Both are doing comparable work on the request path: resolve an endpoint, forward the call, record usage.

Full script: latency_bench.py.

Friction hit during this run​

Real friction hit while doing this comparison, not a symmetric wish list.

MLflow

• 👍 No login wall at all on self-host — straight into a working instance.
• 👍 Full Jinja2 templates — no flattening needed, the first competitor in this series where that's true.
• 👍 A real Gateway endpoint with a working curl snippet shown right on its overview page.
• 👎 The OpenRouter model picker doesn't include gpt-4o-mini, even though OpenRouter's own live catalog still serves it — the picker is a curated static list, not a mirror of the provider's actual catalog.
• 👎 The Datasets list page didn't show a dataset that demonstrably existed server-side.
• 👎 Linking a trace to its prompt version needs a separate, easy-to-forget SDK call.

AcruxCore

• 👍 One gateway call, and the trace and its prompt-version link both happen for free.
• 👍 Cost is on every gateway response, not just an aggregate dashboard.
• 👎 A model with no registered per-1M rate shows a blank — cost instead of $0 or an estimate, until you register one.

What MLflow does that AcruxCore doesn't​

Sourced from MLflow's own dashboard and docs, not from this post's earlier aspects — those carry the same bias any of our own feature list would.

Guardrails — Safety and PII detection, built into every Gateway endpoint. Each endpoint has its own Guardrails tab, and creating one offers three types: a Safety guardrail ("detects harmful, offensive, or toxic content"), a PII Detection guardrail ("detects personally identifiable information such as names, emails, and phone numbers"), or a fully custom guardrail with your own instructions. AcruxCore has no equivalent: nothing inspects a call's input or output for unsafe content or personal data before or after it reaches the model.

An MCP Registry, and a built-in AI assistant. Beyond guardrails: MLflow's MCP Registry (covered in "How tools are handled") has no AcruxCore equivalent at all, and every page ships a docked "MLflow Assistant" chat panel that can answer questions about your own experiments, traces, and evaluations, plus a one-click "Detect Issues" button on the experiment overview that uses AI to flag latency and correctness problems across recent traces. AcruxCore has no in-product assistant or automated issue detection.

AcruxCore has no equivalent to any of the three — no guardrails, no catalog of external MCP servers, and no in-product AI assistant.

Is AcruxCore an MLflow alternative?​

Partly, and the split is clean. MLflow is much more than an LLM-ops tool — it is an experiment tracker with years of history, tens of thousands of GitHub stars and hundreds of contributors. If you already run MLflow for classical ML, nothing here suggests removing it.

As an LLM gateway, it is the closest structural match to us of anything we have compared, which is what makes the measured numbers the interesting part: its AI Gateway added +135 to +225 ms across three 100-round runs and never once crossed zero, where ours added +21 to +43 ms in the same runs. Two identical designs, about 5× apart.

What you would give up is real. MLflow has Safety and PII guardrails on every gateway endpoint and enforced spend Budgets with a reset period and an on-exceeded action. We have spend caps and rate limits, but no content guardrail at all.

Verdict​

This is the closest structural match to AcruxCore of any competitor we've compared: both run a real gateway in the request path, and both trace as a side effect of that same call. Both enforce a spend cap on that path (Spend controls above). Because both sit in the request path, the cost of that hop is directly comparable, and it is the one place they diverge sharply: MLflow's gateway adds +135 to +225 ms where ours adds +21 to +43 ms (Latency overhead above). Otherwise they diverge in what sits around the call — MLflow layers on guardrails that AcruxCore doesn't have (What MLflow does that AcruxCore doesn't above); AcruxCore keeps cost and prompt lineage automatic and inline where MLflow makes you dashboard-hunt for cost or make a second SDK call for lineage (Sending a live call and Tracing & observability above). See the full picture, including license, pricing, and team structure, on the compare page.

To compare more tools, see 10 LLM observability tools tested hands-on, 8 LangSmith alternatives and 8 Langfuse alternatives.

Want to run this yourself? The Quickstart gets you from sign-up to a traced, gateway-routed call in about ten minutes.

So this one measures the full cycle, on real OpenAI billing, across all six platforms plus both of AcruxCore's calling modes — eight paths in one interleaved run. We also ran it four independent times, because a single 100-round run against a live third-party API turned out to not be enough to trust a single number, and that's worth showing rather than hiding.

The setup​

All eight paths hit the same upstream — real api.openai.com, model gpt-4o-mini — with the same prompt and parameters (temperature: 0, max_tokens: 5). Each path does whatever that platform's stored-prompt feature actually requires (fetch a prompt, resolve a template, or nothing, if the platform has no such concept), then completes it, timed together as one number:

1. OpenAI direct — no platform in the loop. The baseline.
2. Opik tracked SDK** — Opik's track_openai() wrapper around a direct call.
3. MLflow AI Gateway** — routed through a local MLflow AI Gateway endpoint.
4. Langfuse OTel SDK — langfuse.get_prompt(), then a call instrumented with Langfuse's OpenTelemetry integration.
5. Helicone AI Gateway — routed through Helicone's AI Gateway.*
6. Phoenix OTel SDK — a call instrumented with Arize Phoenix's OpenTelemetry integration.
7. AcruxCore gateway — hub.prompts.render(), then hub.gateway.chat() routed through AcruxCore's gateway.
8. AcruxCore BYOK (gateway-free) — the same render call, then hub.gateway.chat() with a provider config that posts to OpenAI directly, skipping the gateway. See build a RAG agent without the gateway for the full BYO path.

Each run is 100 rounds plus 3 discarded warm-up rounds, with the eight paths interleaved in rotating order — the running order shifts every round, so no path is always first and a passing network blip hits all eight equally. Connections are pooled. We report medians with a 95% bootstrap confidence interval (5,000 resamples) on the gap against the OpenAI baseline: if that interval crosses zero, the gap isn't distinguishable from noise at this sample size, and we say so rather than quoting the point estimate as if it were exact.

Full script: full-cycle-latency-bench.py.

Median full-cycle latency, all eight paths​

All eight paths land within about 220ms of each other on the median — small next to the ~600–800ms the model call itself takes. The tail tells a fuller story:

Notice the baseline's own p99 (6402ms) is worse than every platform's median and most of their p99s — a reminder that raw OpenAI, with nothing in front of it, has the widest tail here. That's normal live-network variance, not evidence any platform is "faster than the provider it calls."

Median, p95, and p99 are still only three numbers standing in for 100 real measurements each. Here's every one of those 800 rounds, plotted:

The boxes overlap heavily — that's the point. Opik's box sits visibly lower than the rest, which is what the negative gap in the next section is measuring; every other platform's box straddles the baseline's dashed median line.

Is any of this gap real?​

A median alone doesn't tell you whether a platform costs something or whether you're looking at noise. Bootstrapping the gap against the OpenAI baseline, per round (so a shared network blip cancels out), gives a confidence interval:

Only Opik's gap clears the zero line here, and it's negative — faster than the raw baseline. Everything else, AcruxCore included, is statistically indistinguishable from calling OpenAI with nothing in front of it, in this particular 100-round sample.

Opik's negative gap isn't a one-run fluke: it shows up in all four independent runs (−140ms to −168ms), and it's not a few outliers dragging the median either — plotting every round against the same-round baseline shows Opik coming back faster in 72–77% of individual rounds, every run:

That's a real, repeatable pattern. What it isn't, yet, is a fully explained one: an isolated side-by-side test of the two candidate causes — Opik's prompt rendering into a single merged message versus the baseline's separate system/user messages, and the track_openai() wrapper itself — reproduced only a small and inconsistent fraction of the gap, nowhere near its full size. The honest state of this one: real effect, confirmed cause not yet found. We're not going to publish a tidy explanation we haven't verified.

*Helicone's leg times completion only, not a real prompt fetch — its number is directly comparable to the baseline's shape, not the other platforms' two-round-trip cycles.

How stable is that gap, run to run?​

Here's the part worth showing rather than skipping past: we ran the full eight-path benchmark four separate times, on different occasions, with identical code. Tracking AcruxCore's two modes against the OpenAI baseline across all four runs:

The gateway's point estimate moves from +194ms down to 0ms and back to +63ms across four runs of unchanged code — only two of the four even clear statistical significance. BYOK's gap crosses zero in every run it was measured.

That difference is itself informative. A 100-round bootstrap on top of a highly variable, 600–800ms live-network signal has real estimation width — commonly ±100–150ms per run at this sample size — which is enough to swallow a true underlying effect of a few tens of milliseconds. A small, real, and stable cost will still look like it's bouncing between 0 and 190ms across separate runs, purely from sampling noise, even though nothing about the code changed between them. BYOK skips the gateway's synchronous work — a rate-limit check, a budget-reserve transaction, and the request-persist step — while still recording a trace asynchronously, and that's the one number here that came back the same every time we asked.

The honest takeaway: if you need a single number for "how much does the gateway cost," don't take one run's point estimate — the range across four runs is closer to the truth than any one of them. We're publishing all four runs' raw data alongside the script for exactly that reason.

What this means for you​

• All eight paths cluster within ~220ms of each other on the median — small next to the ~600–800ms the model call itself takes.
• Only one gap in this benchmark is consistently distinguishable from noise across runs: Opik's client-side wrapper measured faster than raw OpenAI. Part of that leg also isn't a strict apples-to-apples comparison — Opik sends one merged message instead of the split system/user shape every other leg uses (see the ** note above) — but isolated testing found the message shape explains only a small part of the gap, not the cause.
• AcruxCore's gateway adds a small cost that this method can't pin to an exact number — somewhere in the tens of milliseconds, from synchronous DB work on the request path, but not stable enough across independent runs to quote as a fixed figure.
• AcruxCore's BYOK mode is the one result that held up identically across every run: statistically indistinguishable from calling OpenAI directly, while still reporting a trace. If you want the gateway's tracing and versioned prompts without its synchronous request-path work, that's the trade BYOK makes — see route your calls through the gateway for the other side of that choice.
• Real third-party network latency is the dominant, most variable term in every path here — hundreds of milliseconds and swings of a similar size, next to a local cost measured in the tens.

Reproduce it​

The script needs OPENAI_API_KEY, a personal AcruxCore API key, and local instances of whichever of Opik / MLflow / Langfuse / Phoenix / Helicone you want to include — it skips a platform cleanly if its env vars aren't set. Run it with python full-cycle-latency-bench.py; it prints the summary table and the bootstrap gap analysis to stdout, then writes results.json. Run it more than once before trusting any single number, for the reason above.

Full script and all four runs' raw data: scripts/comparison/shared/ on GitHub. Both charts above are generated straight from results.json by plot_latency_charts.py in the same folder — point it at any run's results.json to regenerate them.

For the narrower question of what AcruxCore's own gateway costs against a BYO call on the same machine, see how much overhead does an LLM gateway add?

For license, price and features of the same tools, see 10 LLM observability tools tested hands-on.

At a glance​

License, pricing, team structure, security, and community stats: see AcruxCore vs Helicone on the compare page.

Full breakdown, screenshots, and the verdict below.

Where the platform sits — in the request path, or beside it​

Unlike most competitors in this series, Helicone genuinely is a request-path product by design: its Providers page lets you register 20+ providers for its "AI Gateway", and its Cache and Rate Limits pages document real header-controlled response caching (Helicone-Cache-Enabled) and rate limiting. Hands-on, routing a non-native provider through it failed two different ways: POST /v1/gateway/oai/v1/chat/ completions forwards the Authorization header straight to api.openai.com regardless of the OpenRouter key registered in Settings, and the generic multi-provider path (/v1/gateway/gateway/...) returns a flat HTTP 501 "Not implemented" — confirmed by reading the self-hosted jawn service's own compiled source, where that handler is a stub. AcruxCore's gateway sits in the request path too, and routed our OpenRouter call successfully on every one of the 100 benchmark rounds below.

Prompt authoring & versioning​

AcruxCore's editor uses nunjucks — real {% if %} and {% for %} logic, so the VIP branch and the ticket list live in the template itself. Helicone's Playground templating is flat {{ hc:var:type }} substitution with a type annotation per variable, so we had to flatten the VIP note and the ticket list into plain text before saving the prompt — the same compromise a competitor forced in an earlier comparison. Saving created a single v0, auto-labeled production; a second version never got created because the next step (a live Run) is the one that's broken on this build.

AcruxCore's own prompt editor — real nunjucks conditionals and loops rendering VIP status and the ticket list inline — is one sentence here; see the walkthrough for the picture.

Sending a live call​

This is where the run stopped producing new evidence and started producing bugs. Clicking Run in Helicone's Playground returns {"error":"Invalid session","trace": "isAuthenticated.error"} every time, on every model, after a fresh login, and after clearing local storage. Network inspection shows why: the request to /v1/playground/generate carries a helicone-authorization header with an empty JWT token ("token":""). AcruxCore's Playground, by contrast, ran the same prompt and rendered the completion inline with cost, cache, and latency.

Tracing & observability​

With the Playground blocked, we fell back to Helicone's other documented path: call the provider directly, then POST the request/response pair to its manual-logging endpoint (/v1/trace/custom/log self-hosted; api.worker.helicone.ai/custom/v1/log on Helicone Cloud). The direct OpenRouter call succeeded. The log call returned HTTP 500 {"details":"Region is missing"} — the self-hosted docker-compose ships its jawn service without an S3_REGION env var, so its S3 client fails to construct before the trace ever reaches ClickHouse. AcruxCore's trace, meanwhile, is written automatically as a side effect of the gateway call — nothing to log by hand at all.

How tools are handled — schema registry, or execution​

We checked every nav section — Segments (Sessions, Properties, Users, Cache, HQL), Improve (Prompts, Datasets, Playground), Monitor (Rate Limits, Alerts) — and found no tool-catalog concept anywhere in self-hosted Helicone: no schema builder, no execution record, nothing. AcruxCore has a real Tool Catalog — callable functions, versioned like prompts, with their own analytics page, wired into the gateway so a call is a real, traced execution.

AcruxCore's Tools page — a dozen real, versioned tools — is one sentence here; see the walkthrough for the screenshot.

Evaluation & datasets​