Langfuse AI Gateway vs LiteLLM vs AcruxCore: a self-hosted gateway benchmark
Langfuse opened its AI Gateway in public beta on 4 November 2026. We tested it against the LiteLLM proxy and AcruxCore. All three ran on the same machine and called the same OpenAI model. For each gateway, we measured the time it adds to a call, how many streams it handles at once, how much memory it needs and what it can do.
-
Added latency, without streaming: LiteLLM added 61 to 67 ms to each call in all three runs. AcruxCore added 15 to 24 ms, and its 95% interval stayed above zero in two of the three runs. Langfuse added 8 ms or less, and its interval included zero in every run. For the first token of a stream, no gateway added a delay in all three runs.
-
At 1,000 simultaneous streams on 2 CPUs, AcruxCore and Langfuse with a raised open-file limit finished all 1,000. Langfuse as shipped failed 739 of them, because its gateway process keeps Docker's default limit of 1,024 open files. One line in its Compose file fixes it. LiteLLM used both CPUs from 100 streams on, and its median time to the first token reached 19 seconds at 1,000.
-
Idle memory: the Langfuse gateway process uses 13 MiB, but the Langfuse stack it needs uses 3.5 GiB. LiteLLM with its database uses 810 MiB, and AcruxCore's whole stack uses 405 MiB.
Before any of that, one finding about OpenAI itself changed our first results by more than any gateway did.
The setup
Every path ends at the same model: OpenAI gpt-6-luna with reasoning_effort: "none",
the same API key, the same prompt and a cap of 5 output tokens. The model's own time is the
same on every path, so what is left is the path.
| Path | Version | How it ran |
|---|---|---|
| OpenAI direct | — | A Node.js client calling api.openai.com. This is the baseline. |
| Langfuse AI Gateway | 4.77.0 | Langfuse's own docker-compose.yml: web, worker, ai-gateway, Postgres, ClickHouse, Redis and MinIO |
| LiteLLM proxy | 1.108.4 | Its Docker image with a Postgres database. Calls use a virtual key, so each one is checked against the database and its spend is written back. |
| AcruxCore | 2026.11.4 | The Docker Hub images with NODE_ENV=production: API, worker, web, Postgres and Redis |
The machine is an AMD Ryzen 9 5900X (12 cores, 24 threads) with 62 GB of RAM, Fedora Linux and Docker 29.9. Everything ran on it at once, and nothing ran on a hosted service.
The OpenAI address that moved results by 250 ms
api.openai.com resolves to more than one address. From our network it returned two
Cloudflare addresses, and they are not the same distance away:
| Address | TCP + TLS setup | Call on an open connection (median) | Call on a new connection (median) |
|---|---|---|---|
| 162.159.140.245 | 33 ms | 852 ms | 865 ms |
| 172.66.0.243 | 263 ms | 863 ms | 1,117 ms |
On a connection that is already open, both addresses answer in the same time. On a new connection, the far address adds about 250 ms to the call, because the connection takes that much longer to set up.
This matters for any gateway comparison. A gateway keeps its connection to OpenAI open, so it
pays the setup once and then stays on whichever address it reached first. A client that opens
new connections pays it again and again, and how much it pays depends on which address DNS
hands it each time. Node's built-in fetch closes an idle connection after about 4 seconds,
so a client that calls OpenAI every few seconds reconnects on almost every call.
Our first run did not control for this. In that run Langfuse came out 43 ms faster than calling OpenAI directly, which a gateway cannot be. The direct path was paying for new connections that the gateway was not.
For every number below, each gateway container has api.openai.com pinned to the near
address, and the direct path keeps its connection open with the same address. To pin a
gateway, we added one extra_hosts line to its Compose file. Docker writes that line into the
container's /etc/hosts, so the container skips DNS for that name:
extra_hosts: ['api.openai.com:162.159.140.245']
The direct path in latency-bench.mjs uses an undici Agent with a 60-second keep-alive and a
DNS lookup that always returns the same address. We checked each container's open connections
to confirm that every path used that address. If you benchmark a gateway yourself, run the
address check first:
OPENAI_API_KEY=… node openai-edge-check.mjs 40
The same advice applies outside benchmarks. If your app calls OpenAI directly, keep the HTTP connection open between calls, or some of your calls will pay this cost.
Added latency
Each run had 5 warm-up rounds and then 150 measured rounds. In each round, we called OpenAI directly and through each of the three gateways, once without streaming and once with streaming. The order changed every round, so no path always went first.
For each gateway, we subtracted the direct call's time from the gateway's time in the same round. OpenAI's own response time changes a lot from one call to the next, much more than any gateway adds. Calls in one round happen close together, so most of that change cancels out in the subtraction. We report the median of the 150 differences, with a 95% bootstrap interval that shows how certain that median is.
A gateway can keep a log of every call it passes on to OpenAI. Writing that log takes time, which could make the gateway slower. So we ran the test three times: with no log ("Off"), with the log each gateway writes when you install it and change nothing ("Defaults"), and with everything logged, including the full question and answer ("Full"). The table shows which option we set in each product to get each level:
| Recording level | Langfuse | LiteLLM | AcruxCore |
|---|---|---|---|
| Off | Ingestion mode NONE | No callback | Trace without request and response |
| Defaults | Ingestion mode USAGE | No callback | Trace with request and response |
| Full | Ingestion mode FULL | Langfuse callback with request and response | Trace with request and response |
LiteLLM writes a spend row to Postgres for every call at all three levels, because the calls use a virtual key. AcruxCore always writes a trace, so its "Defaults" and "Full" are the same configuration. The "Off", "Defaults" and "Full" columns in the next two tables are these three runs.
Time to the full response, without streaming
| Added to the direct call | Off | Defaults | Full |
|---|---|---|---|
| Langfuse | −9 ms (−38 to 22) | +3 ms (−14 to 29) | +8 ms (−9 to 23) |
| LiteLLM | +67 ms (32 to 96) | +65 ms (29 to 86) | +61 ms (41 to 80) |
| AcruxCore | +15 ms (−14 to 30) | +20 ms (4 to 42) | +24 ms (2 to 46) |
The direct call's median was 781 to 831 ms across the three runs. No call failed.
LiteLLM is the only gateway whose interval stays above zero in every run, at 61 to 67 ms. The Langfuse callback did not change its cost. AcruxCore measured 15 to 24 ms, and its interval stays above zero in the two runs that record the request and response. Langfuse measured close to zero in every run. AcruxCore's median is 16 to 24 ms higher than Langfuse's, but the two intervals overlap in every run, so this test does not rank those two.
Time to the first token, with streaming
| Added to the direct call | Off | Defaults | Full |
|---|---|---|---|
| Langfuse | −5 ms (−27 to 19) | 0 ms (−24 to 19) | +5 ms (−8 to 31) |
| LiteLLM | +17 ms (2 to 30) | +43 ms (20 to 72) | +9 ms (−6 to 38) |
| AcruxCore | +8 ms (−7 to 28) | +17 ms (−13 to 41) | +14 ms (−2 to 37) |
No gateway adds a delay to the first token in all three runs. LiteLLM's interval stays above zero in two of them, at 17 and 43 ms, but not in the third. Langfuse and AcruxCore stay within the noise in every run. LiteLLM's cost without streaming, 61 to 67 ms, is larger than any of its first-token numbers, which suggests most of its extra time is spent after the model has answered. We did not profile it to confirm this.
1,000 streams on 2 CPUs
Each gateway container was limited to 2 CPUs. The test opened 10, 100, 250, 500 and then 1,000 streaming calls at the same moment through one gateway, each asking the model to count to 60, and waited for all of them to finish. CPU and memory are read from each container's cgroup every 250 ms.
As shipped
| Simultaneous streams | Langfuse | LiteLLM | AcruxCore |
|---|---|---|---|
| 250 | 250 ok, first token 1.4 s | 250 ok, first token 9.8 s | 250 ok, first token 1.5 s |
| 500 | 110 ok, 390 × 502 | 249 ok, 251 × 503 | 500 ok, first token 2.2 s |
| 1,000 | 261 ok, 739 × 502 | 403 ok, 597 failed | 1,000 ok, first token 3.5 s |
First-token times are medians of the streams that succeeded.
Langfuse's failures were 502 Upstream request failed, and they repeated on a second run
(125 of 500 succeeded). The gateway's own logs split them into two kinds: failed connections
to OpenAI, and failed connections to Langfuse's web service, which the gateway asks to check
each key. LiteLLM's failures at 500 were 503 the authentication database is temporarily unreachable, and at 1,000 they were connection errors. The 503 errors have a separate cause,
explained below.
The cause: 1,024 open files
Linux counts every network connection as an open file, and it caps how many open files one process may hold. The cap has two numbers. The soft limit is the cap the process has now. The hard limit is the highest value the process may raise its own soft limit to.
Each stream holds two connections: one from the client and one to OpenAI. So 500 streams need about 1,000 open files, and 1,000 streams need about 2,000.
On our machine, Docker started every container with a soft limit of 1,024 and a hard limit of 524,288. All three gateways got these same two numbers. The difference is what each program does when it starts. Node.js raises its own soft limit to the hard limit, so the AcruxCore API runs with 524,288. The Langfuse gateway, written in Rust, and LiteLLM, written in Python, keep the soft limit of 1,024, and neither Compose file sets a higher one.
We read the soft limit each gateway process ended up with from /proc/<pid>/limits:
| Gateway process | Open-file limit |
|---|---|
Langfuse ai-gateway (Rust) | 1,024 |
| LiteLLM (Python) | 1,024 |
| AcruxCore API (Node.js) | 524,288 |
To test this, we raised the limit to 65,536 for all three gateways and ran again. For a self-hosted Langfuse,
that is these lines on the langfuse-ai-gateway service in its Compose file:
ulimits:
nofile: { soft: 65536, hard: 65536 }
For LiteLLM we also ran 2 workers, one per CPU, as its production guide recommends for a single machine.
Limit increased
| Simultaneous streams | Langfuse | LiteLLM | AcruxCore |
|---|---|---|---|
| 250 | 250 ok, first token 1.2 s | 250 ok, first token 4.7 s | 250 ok, first token 1.5 s |
| 500 | 500 ok, first token 1.4 s | 380 ok, 120 × 503 | 500 ok, first token 2.2 s |
| 1,000 | 1,000 ok, first token 1.7 s | 1,000 ok, first token 19.2 s | 1,000 ok, first token 3.8 s |
AcruxCore ran with the same limit of 65,536 here. Its results are within 0.3 s of the run with its own limit of 524,288.
With the limit raised, the Langfuse gateway finished all 1,000 streams with the fastest first token of the three. At its peak it used 1.65 of its 2 CPUs and 168 MiB. AcruxCore used both CPUs from 100 streams on, and its first token at 1,000 streams was 3.8 s at the median and 9.7 s at p95, against Langfuse's 1.7 s and 9.6 s. LiteLLM used its whole CPU allowance from 100 streams on with either setting. In this run, 120 of 500 streams failed with the same database error, and then all 1,000 succeeded at the next level.
The gateway process is not the whole cost. While the Langfuse gateway served 500 streams, its whole stack used up to 3.1 CPUs, because the web service checks keys and the worker and ClickHouse store usage. AcruxCore's whole stack peaked at 4.0 CPUs, since its API writes a trace for every call to Postgres. The per-level numbers for both are in the results files.
LiteLLM's 503 errors
LiteLLM checks each virtual key in its Postgres database and keeps the result in memory for
60 seconds. When many calls arrive while the key is not in memory, each call looks the key up
in the database. Each LiteLLM worker opens at most 10 database connections, and LiteLLM rejects
a call with 503 when its key lookup takes more than 10 seconds. LiteLLM's log gave that reason,
key lookup did not answer within 10s, for every one of the 840 failed streams in the runs below.
We ran LiteLLM with one worker and only the open-file limit raised, to separate the two causes:
| Run | 500 streams |
|---|---|
| Key not in memory, run 1 | 339 ok, 161 × 503 |
| Key not in memory, run 2 | 416 ok, 84 × 503 |
| Key not in memory, run 3 | 500 ok |
| One call made first, so the key is in memory | 500 ok |
Each "not in memory" run started after 75 seconds with no calls. How many lookups miss the
10-second limit changes from run to run. That is why the 503 errors appeared at 500 streams in
one ladder and at 1,000 in another. In the full ladder with this setting, all 500 streams
succeeded and 595 of 1,000 failed with 503.
The connection errors that LiteLLM as shipped had at 1,000 streams did not appear once the open-file limit was raised.
Memory
Measured with the stacks idle, after all the tests in this post had run:
| Gateway process | Whole stack | |
|---|---|---|
| Langfuse | 13 MiB | 3,584 MiB (ClickHouse 1,810, web 917, worker 654, MinIO 103, Postgres 75, Redis 12) |
| LiteLLM | 727 MiB | 810 MiB (with Postgres 83) |
| AcruxCore | 200 MiB | 405 MiB (worker 105, Postgres 66, web 18, Redis 16) |
The Langfuse gateway is a small Rust process, but it cannot run alone. It asks Langfuse's web service to check every key, and it records calls into the rest of the Langfuse stack. If you already run Langfuse for tracing, the gateway adds 13 MiB. If you do not, it brings the 3.5 GiB stack with it.
At 1,000 streams, the gateway processes peaked at 168 MiB for Langfuse, 553 MiB for AcruxCore and 2,748 MiB for LiteLLM with 2 workers.
Features
From each project's documentation, checked on 10 November 2026, and AcruxCore's source.
| Langfuse AI Gateway (beta) | LiteLLM proxy | AcruxCore | |
|---|---|---|---|
| Providers | OpenAI and Anthropic | 100+ | OpenAI, Anthropic, Gemini and any OpenAI-compatible API |
| Accepts one API format and calls another | No. The route decides the format. | Yes, OpenAI ↔ Anthropic both ways | OpenAI format in, converted for Anthropic and Gemini |
| Spend caps that block a call | No, on its roadmap | Yes, needs Postgres | Yes |
| Rate limits per key | No | Yes, needs Postgres | Yes |
| Fallbacks between models | No, on its roadmap | Yes | Yes |
| Response cache | No | Yes | Yes |
| A key per person or service | Yes | Yes, needs Postgres | Yes |
| Model allow-list | Per provider credential | Per key | Per key |
| Joins the caller's trace | Yes, W3C traceparent | Yes, with its OpenTelemetry callback | Yes, with its own x-trace-id header |
| Custom upstream URL | No, official provider URLs only | Yes | Yes |
| License | MIT, except the ee/ folders | MIT, except the enterprise/ folder | Apache 2.0 |
Sources: Langfuse AI Gateway docs, Langfuse gateway self-hosting, LiteLLM proxy docs, LiteLLM budgets, LiteLLM virtual keys, AcruxCore gateway guide.
Which one fits
Langfuse AI Gateway fits a team that already self-hosts Langfuse and wants every OpenAI or Anthropic call recorded there, with keys per person. Its gateway process is the lightest of the three and handled 1,000 streams best, once the open-file limit is raised. It is a beta with two providers, and it has no spend caps, fallbacks or cache yet.
LiteLLM fits a team that needs many providers and format conversion between them. Plan for CPU. In this test it used both CPUs from 100 simultaneous streams on, and the first token took 4.7 s at 250 streams, so a few hundred streams need more CPUs or more replicas than this.
AcruxCore fits a team that wants budgets, rate limits, fallbacks and a cache in front of the model, plus prompts, tools, traces and evaluation in the same self-hosted stack. It finished every stream without tuning. At high concurrency its single Node.js process uses its whole CPU allowance, so give it more CPU or more replicas before Langfuse would need them.
Key point: Two settings outside the gateways' code decided this test. The OpenAI address a new connection reaches can add about 250 ms, and Docker's open-file limit of 1,024 made two of the three gateways fail at high concurrency.
What this test does not show
- One machine, one network. The OpenAI address finding depends on where you are. Run
openai-edge-check.mjsfrom your own servers. - The gateways' own code is measured through OpenAI. Langfuse's gateway cannot point at a custom upstream URL, so we could not use a local mock model to measure each gateway to the millisecond. Differences smaller than the confidence intervals above are not measured.
- A short prompt. Each latency call sends a few tokens and returns one word. Large requests or responses may change the picture, especially with full payload recording.
- One version each. Langfuse's gateway is a beta and changes weekly.
All scripts, the Compose overrides and the raw results:
scripts/comparison/langfuse-gateway-vs-litellm-vs-acruxcore