Capture user feedback from your application to traces

Recommended Reading

Before diving into this content, it might be helpful to read the following:

• Conceptual guide on tracing and feedback
• Reference guide on feedback data format

In many applications, but even more so for LLM applications, it is important to collect user feedback to understand how your application is performing in real-world scenarios. The ability to observe user feedback along with trace data can be very powerful to drill down into the most interesting datapoints, then send those datapoints for further review, automatic evaluation, or even datasets. To learn more about how to filter traces based on various attributes, including user feedback, see this guide

LangSmith makes it easy to attach user feedback to traces. It's often helpful to expose a simple mechanism (such as a thumbs-up, thumbs-down button) to collect user feedback for your application responses. You can then use the LangSmith SDK or API to send feedback for a trace. To get the run_id of a logged run, see this guide.

note

You can attach user feedback to ANY intermediate run (span) of the trace, not just the root span. This is useful for critiquing specific parts of the LLM application, such as the retrieval step or generation step of the RAG pipeline.

Python

from langsmith import Client

client = Client()

# ... Run your application and get the run_id...
# This information can be the result of a user-facing feedback form

client.create_feedback(
    run_id,
    key="feedback-key",
    score=1.0,
    comment="comment",
)

TypeScript

import { Client } from "langsmith";
const client = new Client();

// ... Run your application and get the run_id...
// This information can be the result of a user-facing feedback form

await client.createFeedback(
    runId,
    "feedback-key",
    {
        score: 1.0,
        comment: "comment",
    }
);