Configuration for the judge agent.
Configuration for the judge agent.
The criteria that the judge will use to evaluate the conversation.
OptionalmaxTokens?: numberOptionalmodel?: LanguageModelOptionalname?: stringThe name of the agent.
OptionalsystemPrompt?: stringA custom system prompt to override the default behavior of the judge.
Optionaltemperature?: numberimport { run, judgeAgent, AgentRole, user, agent, AgentAdapter } from '@langwatch/scenario';
const myAgent: AgentAdapter = {
role: AgentRole.AGENT,
async call(input) {
return `The user said: ${input.messages.at(-1)?.content}`;
}
};
async function main() {
const result = await run({
name: "Judge Agent Test",
description: "A simple test to see if the judge agent works.",
agents: [
myAgent,
judgeAgent({
criteria: ["The agent must respond to the user."],
}),
],
script: [
user("Hello!"),
agent(),
],
});
}
main();
Factory function for creating JudgeAgent instances.
JudgeAgent evaluates conversations against success criteria.
The JudgeAgent watches conversations in real-time and makes decisions about whether the agent under test is meeting the specified criteria. It can either allow the conversation to continue or end it with a success/failure verdict.
The judge uses function calling to make structured decisions and provides detailed reasoning for its verdicts. It evaluates each criterion independently and provides comprehensive feedback about what worked and what didn't.