17 posts tagged with "LLM tracing"

Most comparisons of the best LLM observability tools are written from docs and marketing pages. We didn't do that here. We ran ten LLM-ops platforms ourselves — LangSmith, Langfuse, PromptLayer, Arize Phoenix, Opik, MLflow, Helicone, Laminar, and Agenta, plus our own AcruxCore as the baseline — and did the same thing on each one: create a prompt, version it, run it live with a real model key, inspect the resulting trace, and try to build an eval. Then we wrote a small script against each platform's own SDK and ran that too, and timed every one of them against a direct call to the provider.

Most people who search for LLM observability tools want tracing and monitoring, and one section below covers both. Prompt management, evals, guardrails, and tool calling get their own sections too. For open source LLM observability tools that you can self-host, a separate post measures the memory and disk use of seven open source observability tools on one machine.

The previous post compares how LangSmith and AcruxCore trace the same tool-calling agent.

This post compares the code you write when the tool is a function in your own app. LangChain's create_agent runs against the AcruxCore SDK's run_tool_loop. Same tool, same question, same model (openai/gpt-4o-mini via OpenRouter), both run live.

A tool call is the point where an agent stops producing text and calls code or an API. When an agent gives a wrong answer, the first question is usually what the tool returned and how long it took. A trace of the run has to answer that.

We built the same agent twice and ran both live. The two agents use the same tool, the same question and the same model: openai/gpt-4o-mini through OpenRouter. LangSmith traces one run and AcruxCore traces the other. Then we ran both again with streaming.

Every number and every output below comes from those runs. The scripts for both platforms are at the end of the post in full.

This is the AcruxCore leg of a hands-on comparison series. We ran the exact same steps — create a prompt, version it, generate a trace, try to build a dataset — against the hosted product ourselves, so the LangSmith, Langfuse, and PromptLayer write-ups have a fair baseline to compare against. No marketing framing here — just what the dashboard actually does. The main comparison post pulls the findings from all four platforms together.

Langfuse is one of the best-known open-source LLM engineering platforms — it covers prompt management, tracing (recording what an LLM app actually did, step by step), evaluation, and datasets. This is the Langfuse leg of a hands-on comparison series: we logged into Langfuse's hosted EU cloud with a real account and clicked through the whole loop ourselves — open a prompt, version it, inspect a real trace, and set up a dataset for testing. This post is a plain, factual account of what we found — the good and the rough edges. The main comparison post pulls the findings from Langfuse, LangSmith, PromptLayer, and AcruxCore together.

We logged into the real, hosted version of LangSmith (US region) and worked through the same four steps we're using across every platform in this comparison: create a prompt, run it and inspect the trace, and build a small eval dataset. This post is just the LangSmith leg — screenshots, actual UI, no marketing copy. The main comparison post pulls the findings from LangSmith, Langfuse, PromptLayer, and AcruxCore together.

If you're building on LLMs, you eventually need four things: prompts you can change without redeploying, a reliable path to model providers, visibility into what each call did, and a way to tell whether a change helped. LangSmith and Acrux Core both address this space from different starting points. This is an honest look at where they overlap and where they differ.