Configuration for the judge agent.
Configuration for the judge agent.
The criteria that the judge will use to evaluate the conversation.
OptionalmaxTokens?: numberThe maximum number of tokens to generate.
Optionalmodel?: LanguageModelV1The language model to use for generating responses. If not provided, a default model will be used.
Optionalname?: stringThe name of the agent.
OptionalsystemPrompt?: stringA custom system prompt to override the default behavior of the judge.
Optionaltemperature?: numberThe temperature for the language model. Defaults to 0.
import { run, judgeAgent, AgentRole, user, agent, AgentAdapter } from '@langwatch/scenario';
const myAgent: AgentAdapter = {
role: AgentRole.AGENT,
async call(input) {
return `The user said: ${input.messages.at(-1)?.content}`;
}
};
async function main() {
const result = await run({
name: "Judge Agent Test",
description: "A simple test to see if the judge agent works.",
agents: [
myAgent,
judgeAgent({
criteria: ["The agent must respond to the user."],
}),
],
script: [
user("Hello!"),
agent(),
],
});
}
main();
Agent that evaluates conversations against success criteria.
The JudgeAgent watches conversations in real-time and makes decisions about whether the agent under test is meeting the specified criteria. It can either allow the conversation to continue or end it with a success/failure verdict.
The judge uses function calling to make structured decisions and provides detailed reasoning for its verdicts. It evaluates each criterion independently and provides comprehensive feedback about what worked and what didn't.