LangChain published guidance on Oct. 15, 2026, detailing how developers can use its LangSmith platform to evaluate agent behavior through a combination of offline datasets and online production monitoring.
The company asserts that agents are harder to test than traditional software because outputs vary between runs and correct results can be achieved through multiple different paths, according to the LangChain Blog.
LangChain describes a workflow where developers start by analyzing traces—records of agent runs including inputs, outputs, and tool calls—to identify specific behaviors to test. Developers then build small datasets of 10 to 20 examples from these traces to run baselines and compare different agent versions via "experiments" in LangSmith, the company said.
The company distinguishes between offline evals, which run against fixed datasets before shipping to catch regressions, and online evals, which score real production traces to monitor quality. Failures identified by online evals are intended to be fed back into the offline datasets to create an improvement loop, according to the LangChain Blog.
LangChain identifies four types of evaluators: deterministic code evaluators, LLM-as-a-judge, human review, and pairwise evaluators that compare two outputs. To manage the cost of online evaluations, LangSmith allows developers to score only a percentage of traffic or filter for specific criteria, such as negative user feedback, the company said.
For complex, multi-step tasks, LangChain highlights the use of benchmarks—offline evals involving isolated containerized environments. The company mentions Harbor, an open-source eval harness that powers Terminal-Bench and integrates with LangSmith to log results as experiments, according to the LangChain Blog.