LangChain said in a Nov. 3 blog post that LangSmith annotation queues can present a full multi-turn conversation, called a thread, as one review item LangChain Blog1. The company outlined how domain experts can evaluate the complete context of an agent’s behavior rather than isolated steps.
The update addresses what LangChain identified as a limitation in previous workflows where reviewers examined single "runs" or spans within a larger session. LangChain stated that reviewing a single step often misses preceding messages and tool calls necessary to understand why an agent succeeded or failed. By grouping these elements into a thread, the platform presents the ordered sequence of interactions without nested run tree duplication, allowing reviewers to assess the conversation against the user's original intent.
LangChain described three principles for this workflow: defining quality through expert judgment, evaluating agents in context, and turning feedback into reusable data. When a reviewer scores a thread, they provide a structured score (such as pass/fail or a 1–5 scale) along with reasoning. This creates ground truth for that specific interaction. Builders can then use these labeled examples to align large language model judges by comparing their scores to human scores and updating prompts where they disagree. They can also update online evaluators if humans consistently catch misses that automated systems overlooked. Strong human-labeled examples can be promoted into golden datasets for offline testing when agent prompts change.
In a case study included in the announcement, Fetch Rewards described using annotation queues to evaluate its AI Rewards Assistant. Reviewers assessed individual turns but investigated trace metadata to verify tool usage and data grounding. For example, they confirmed that product-card images matched the actual products and that recommendations were available to the user. Engineers then used the reviewers' documented reasoning to tune automated judges, identifying where the system's logic diverged from human assessment. The post cites no independent testing of thread-based review against single-run review, and the Fetch Rewards example is LangChain's own case study.
The post does not state pricing, plan, or access limits for the feature. LangChain noted that a queue item can still be a single run, meaning the thread option is an alternative rather than a replacement for isolated step review.