5 posts tagged with "LLM latency"

Agenta is an open-source platform from Agentatech UG. Its README describes a workspace where you build agents by chatting with them. You then talk to those agents in Slack, Telegram or WhatsApp. Behind the agent pages, Agenta also ships a prompt registry, a playground, evaluation, annotation queues and tracing. Agenta documents those parts in its v1.0 docs, which it marks as no longer actively maintained. Those parts still ship in version 0.122.7, the version we ran. This post compares those parts with AcruxCore, because those are the parts the two products share.

We ran Agenta 0.122.7 on one machine with its own Docker Compose file. The prompt, the customer message and the model were the same on both sides. Each section below shows what we saw on Agenta, then what AcruxCore does with the same prompt.

Laminar (lmnr) is a Y Combinator S24 project that calls itself observability purpose-built for AI agents — a Rust ingest server, ClickHouse for spans, Quickwit for search, and an OpenTelemetry SDK that instruments fifteen-plus agent frameworks from one line of code — Vercel AI SDK, Claude Agent SDK, OpenAI Agents SDK, LangChain DeepAgents, Mastra, Pydantic AI, Browser Use, Stagehand, Playwright, LiteLLM, OpenCode and OpenHands among them. It is the most differently shaped tool we have compared AcruxCore against. Its project sidebar has dashboards, traces, evaluations, datasets, labeling, a SQL editor, playgrounds, a debugger and settings — and no prompt registry anywhere. That is not an oversight; it is the product being about agent runs rather than about the calls inside them. Both projects are Apache 2.0, so this is a comparison of shape, not of who is open.

Every latency number we'd published before this measured one thing: the completion call. But a real request to any of these platforms is usually two round trips — fetch or resolve a stored prompt, then complete it — and most of them were benchmarked against OpenRouter, not the provider whose name is on the model.

So this one measures the full cycle, on real OpenAI billing, across all six platforms plus both of AcruxCore's calling modes — eight paths in one interleaved run. We also ran it four independent times, because a single 100-round run against a live third-party API turned out to not be enough to trust a single number, and that's worth showing rather than hiding.

A gateway virtual key can be given a cache window: set cacheTtlSeconds and an exact-match repeat of a call — same model, same messages, same temperature/max_tokens/top_p/stop — is served from Postgres instead of the provider. We ran the same fixed set of prompts through it repeatedly to see what a hit is actually worth, in milliseconds and in dollars.

Putting a gateway in front of your model providers buys you a lot: one endpoint for every provider, cost accounting, caching, virtual keys, and budgets. But it raises an obvious worry — am I paying for that with latency? Every request now takes an extra hop, and for a user-facing app, milliseconds matter.

So we measured five ways to reach the same model — through our gateway, around it with your own key, and raw — to see exactly where the milliseconds go.