Skip to main content

LLMOps comparisons and benchmarks

Hands-on LLMOps platform comparisons, latency benchmarks against real providers, self-hosting LLM observability, and research findings on AI agents.

LLM-as-a-Judge: How Reliable Are LLM Judges, and How to Evaluate an LLM App or Agent

Published · 22 min read
AI engineer, author of 23 peer-reviewed papers in applied machine learning

How reliable are LLM judges: three judge modes (pairwise comparison, single-answer grading, reference-guided grading), five measured biases (position, verbosity, self-preference, leniency, empty answers) and five fixes (swap answer order, control for length, reference answer, clear rubric, different model family)

LLM as a judge means you use one language model to grade the output of another. A strong judge can agree with people about as often as people agree with each other, and it also fails in ways that are now well measured. In the MT-Bench study, GPT-4 agreed with human experts on 85% of non-tie votes, while two experts agreed with each other on 81%. In the same study, GPT-4 kept its verdict on only 65% of answer pairs when the order of the two answers was swapped.

This article collects what peer-reviewed research and the OpenAI, Anthropic and Google evaluation guides say about how reliable LLM judges are, which biases have been measured, and what reduces them. It then covers how to evaluate an LLM app, a RAG pipeline and an agent.

15 LiteLLM alternatives: self-hosted gateways, hosted gateways and libraries (2026)

Published · 33 min read
LLM-ops platform

Fifteen LiteLLM alternatives in a colour-coded grid, each with what it is best for. Self-hosted open source: AcruxCore (ours), MLflow AI Gateway, Bifrost, Apache APISIX, Agent Router, Portkey, Kong AI Gateway, Helicone. Hosted: Cloudflare AI Gateway, Vercel AI Gateway, TrueFoundry, OpenRouter. Library: any-llm, LangChain, Vercel AI SDK

This post compares fifteen LiteLLM alternatives on license, self-hosting, gateway controls and price. It sorts them into gateways you run yourself, hosted gateways and libraries, so you can tell which kind of tool replaces the part of LiteLLM you use.

We build AcruxCore, one of the tools in this post. We do not rank it.

13 OpenRouter alternatives: gateways, resellers and local models (2026)

Published · Updated · 30 min read
LLM-ops platform

OpenRouter alternatives in four columns: Your own keys (LiteLLM, Kong AI Gateway, Helicone, Portkey, AcruxCore (ours)), One bill (Vercel AI Gateway, Cloudflare AI Gateway, Requesty, Eden AI, NanoGPT), Cloud catalogs (Amazon Bedrock, Together AI, Hugging Face) and Local models (Ollama)

This post compares thirteen OpenRouter alternatives on fees, bring-your-own-key support, self-hosting and routing. After it, you can tell whether you need a reseller, a gateway you run yourself, a cloud catalog or a local runtime.

We build AcruxCore, one of the tools in this post. We do not rank it.

AcruxCore vs Agenta: built-in evaluators vs a gateway

Published · Updated · 22 min read
LLM-ops platform

Header card for AcruxCore vs Agenta, listing each product's main strengths side by side

The short answer: Agenta is stronger at evaluation, and AcruxCore is stronger at controlling the model call. Agenta has 11 built-in evaluator types, plus five regex presets, and annotation queues. AcruxCore has a gateway with budgets, caching and virtual keys, and a catalog of versioned tools that it runs.

Agenta is an open-source platform from Agentatech UG. Its README describes a workspace where you build agents by chatting with them. You then talk to those agents in Slack, Telegram or WhatsApp. Behind the agent pages, Agenta also ships a prompt registry, a playground, evaluation, annotation queues and tracing. Agenta documents those parts in its v1.0 docs, which it marks as no longer actively maintained. Those parts still ship in version 0.122.7, the version we ran. This post compares those parts with AcruxCore, because those are the parts the two products share.

Each section below shows what we saw on Agenta, then what AcruxCore does with the same prompt.

AcruxCore is our product. The Agenta and AcruxCore scripts are linked where they are used.

7 best Langfuse alternatives for startups: self-hosted, cheaper and open source (2026)

Published · Updated · 25 min read
LLM-ops platform

Langfuse alternatives in a grid, each with what it is best for, coloured by license: MLflow, OpenLIT, Laminar and Opik open source, Arize Phoenix source-available, LangSmith and Braintrust closed source, and AcruxCore (ours) open source

This post compares seven alternatives to Langfuse on self-hosting effort, license, price and the features that need an enterprise plan. For each Langfuse alternative, it lists what the tool adds, what it lacks, and what changes when you move from Langfuse to it.

We build AcruxCore, one of the tools in this post. We do not rank it.

7 LangSmith alternatives for startups: open source, self-hosted and cheaper (2026)

Published · Updated · 24 min read
LLM-ops platform

LangSmith alternatives in a grid, each with what it is best for, coloured by license: Langfuse, Opik, MLflow and Laminar open source, Arize Phoenix source-available, Braintrust and W&B Weave closed source, and AcruxCore (ours) open source

This post compares seven LangSmith alternatives on license, self-hosting, price and features. For each alternative to LangSmith, it lists what the tool adds, what it lacks, and how it changes the per-seat price and the 14-day trace retention that push teams off LangSmith. A section at the end shows how to move your traces.

We build AcruxCore, one of the tools in this post. We do not rank it.

Self-hosting LLM observability tools: seven measured, six open source

Published · Updated · 17 min read
LLM-ops platform

Bar chart of idle memory for seven self-hosted LLM observability stacks: AcruxCore 0.39 GiB, Phoenix 0.47, Laminar 1.11, MLflow 2.33, Langfuse 2.60, Agenta 2.62, Opik 3.69

Open-source and source-available LLM observability tools describe self-hosting the same way. You clone the repo, run one docker compose up, and open the dashboard. Most self-host pages say nothing after that step.

The compose file behind that command fixes several things the self-host page does not list. The file sets how many containers you run and which databases come with them. Those containers set how much memory the server needs before the first trace arrives. The file's volume mounts decide whether your data survives a restart.

We ran the official self-host setup of seven tools in October 2026: Langfuse, Opik, Phoenix, MLflow, Laminar, Agenta and AcruxCore. Each ran alone on the same machine. Each received the same 1,000 traces.

We build AcruxCore. We ran every tool ourselves, and the scripts are linked below.

AcruxCore vs Langfuse vs LangSmith: one app tested end to end on all three

Published · Updated · 19 min read
LLM-ops platform

AcruxCore vs Langfuse vs LangSmith, one app tested on all three. AcruxCore: open source with every feature, gateway built in, no review queue. Langfuse: open core with a big community, review queue, gateway in beta. LangSmith: closed platform, review queue, Go and Java SDKs

This post compares AcruxCore, Langfuse and LangSmith on the things a developer checks before choosing one. We also ran one app on all three platforms, with the same code, data, prompts and model, and measured each step. The app is a small agent that reads a client's email, checks each request against a signed project scope, prices the extra work, and writes a reply.

We build AcruxCore, one of the three tools compared here. We ran all three ourselves, and the scripts for every leg are linked below.

Langfuse vs LangSmith: one app tested end to end on both

Published · Updated · 20 min read
LLM-ops platform

Langfuse vs LangSmith, one app tested on both. Langfuse: open core and free to self-host, draft-safe prompt reads, a gateway in beta with no spend caps. LangSmith: closed with Enterprise-only self-hosting, a changed-score filter, a gateway in beta

This post compares Langfuse and LangSmith on the things a developer checks before choosing one. We also ran one app on both hosted platforms, with the same code, data, prompts and model, and measured each step. The app is a small agent that reads a client's email, checks each request against a signed project scope, prices the extra work, and writes a reply.

Pick Langfuse if you want an open-source platform that you can self-host for free, a large community, and 90 days of data at a low price. Pick LangSmith if you compare experiment runs often, need Go or Java, or want a gateway that enforces spend and rate limits. At our profile of 100,000 traces a month and 3 people, LangSmith Plus cost $567 a month and Langfuse Core cost $98.36.

We build AcruxCore, a third tool in this space that this post does not test. We ran Langfuse and LangSmith ourselves, and the scripts are linked below.