Class ScenarioExecution

constructor

• new ScenarioExecution(
      config: ScenarioConfig,
      script: ScriptStep[],
  ): ScenarioExecution

  Creates a new ScenarioExecution instance.

  Parameters

  • config: ScenarioConfig

    The scenario configuration containing agents, settings, and metadata

  • script: ScriptStep[]

    The ordered sequence of script steps that define the test flow

  Returns ScenarioExecution

Readonlyevents$

messages

• get messages(): ModelMessage[]

  Gets the complete conversation history as an array of messages.

  Returns ModelMessage[]

  Array of ModelMessage objects representing the full conversation

result

• get result(): ScenarioResult | undefined

  Gets the result of the scenario execution if it has been set.

  Returns ScenarioResult | undefined

  The scenario result or undefined if not yet set

threadId

• get threadId(): string

  Gets the unique identifier for the conversation thread. This ID is used to maintain conversation context across multiple runs.

  Returns string

  The thread identifier string

addAgentTime

• addAgentTime(agentIdx: number, time: number): void

  Adds execution time for a specific agent to the performance tracking.

  This method is used internally to track how long each agent takes to respond, which is included in the final scenario result for performance analysis. The accumulated time for each agent is used to calculate total agent response times in the scenario result.

  Parameters

  • agentIdx: number

    The index of the agent in the agents array

  • time: number

    The execution time in milliseconds to add to the agent's total

  Returns void

  Example

  // This is typically called internally by the execution engine
  execution.addAgentTime(0, 1500); // Agent at index 0 took 1.5 seconds
  Copy

agent

• agent(content?: string | ModelMessage): Promise<void>

  Executes an agent turn in the conversation.

  If content is provided, it's used directly as the agent's response. If not provided, the agent under test is called to generate a response based on the current conversation context and any pending messages.

  This method is part of the ScenarioExecutionLike interface used by script steps.

  Parameters

  • Optionalcontent: string | ModelMessage

    Optional content for the agent's response. Can be a string or ModelMessage. If not provided, the agent under test will generate the response.

  Returns Promise<void>

  Example

  // Let agent generate response
  await execution.agent();
  
  // Use provided content
  await execution.agent("The weather is sunny today!");
  
  // Use a ModelMessage object
  await execution.agent({
    role: "assistant",
    content: "I'm here to help you with weather information."
  });
  Copy

execute

• execute(): Promise<ScenarioResult>

  Executes the entire scenario from start to finish.

  This method runs through all script steps sequentially until a final result (success, failure, or error) is determined. Each script step can trigger one or more agent interactions depending on the step type:

  • user() and agent() steps typically trigger one agent interaction each
  • proceed() steps can trigger multiple agent interactions across multiple turns
  • judge() steps trigger the judge agent to evaluate the conversation
  • succeed() and fail() steps immediately end the scenario

  The execution will stop early if:

  • A script step returns a ScenarioResult
  • The maximum number of turns is reached
  • An error occurs during execution

  Returns Promise<ScenarioResult>

  A promise that resolves with the final result of the scenario

  Throws

  Error if an unhandled exception occurs during execution

  Example

  const execution = new ScenarioExecution(config, script);
  const result = await execution.execute();
  console.log(`Scenario ${result.success ? 'passed' : 'failed'}`);
  Copy

fail

• fail(reasoning?: string): Promise<ScenarioResult>

  Immediately ends the scenario with a failure verdict.

  This method forces the scenario to end with failure, regardless of the current conversation state. It's useful for scenarios where you want to explicitly mark failure based on specific conditions or external factors.

  This method is part of the ScenarioExecutionLike interface used by script steps.

  Parameters

  • Optionalreasoning: string

    Optional explanation for why the scenario is being marked as failed

  Returns Promise<ScenarioResult>

  A promise that resolves with the final failed scenario result

  Example

  // Mark failure with default reasoning
  const result = await execution.fail();
  
  // Mark failure with custom reasoning
  const result = await execution.fail(
    "Agent failed to provide accurate weather information"
  );
  Copy

judge

• judge(content?: string | ModelMessage): Promise<ScenarioResult | null>

  Invokes the judge agent to evaluate the current state of the conversation.

  The judge agent analyzes the conversation history and determines whether the scenario criteria have been met. This can result in either:

  • A final scenario result (success/failure) if the judge makes a decision
  • Null if the judge needs more information or conversation to continue

  This method is part of the ScenarioExecutionLike interface used by script steps.

  Parameters

  • Optionalcontent: string | ModelMessage

    Optional message to pass to the judge agent for additional context

  Returns Promise<ScenarioResult | null>

  A promise that resolves with:

  • ScenarioResult if the judge makes a final decision, or
  • Null if the conversation should continue

  Example

  // Let judge evaluate current state
  const result = await execution.judge();
  if (result) {
    console.log(`Judge decided: ${result.success ? 'pass' : 'fail'}`);
  }
  
  // Provide additional context to judge
  const result = await execution.judge("Please consider the user's satisfaction level");
  Copy

message

• message(message: ModelMessage): Promise<void>

  Adds a message to the conversation history.

  This method is part of the ScenarioExecutionLike interface used by script steps. It automatically routes the message to the appropriate agent based on the message role:

  • "user" messages are routed to USER role agents
  • "assistant" messages are routed to AGENT role agents
  • Other message types are added directly to the conversation

  Parameters

  • message: ModelMessage

    The ModelMessage to add to the conversation

  Returns Promise<void>

  Example

  await execution.message({
    role: "user",
    content: "Hello, how are you?"
  });
  Copy

proceed

• proceed(
      turns?: number,
      onTurn?: (state: ScenarioExecutionStateLike) => void | Promise<void>,
      onStep?: (state: ScenarioExecutionStateLike) => void | Promise<void>,
  ): Promise<ScenarioResult | null>

  Lets the scenario proceed automatically for a specified number of turns.

  This method is a script step that simulates natural conversation flow by allowing agents to interact automatically without explicit script steps. It can trigger multiple agent interactions across multiple turns, making it useful for testing scenarios where you want to see how agents behave in extended conversations.

  Unlike other script steps that typically trigger one agent interaction each, this step can trigger many agent interactions depending on the number of turns and the agents' behavior.

  The method will continue until:

  • The specified number of turns is reached
  • A final scenario result is determined
  • The maximum turns limit is reached

  Parameters

  • Optionalturns: number

    The number of turns to proceed. If undefined, runs until a conclusion or max turns is reached

  • OptionalonTurn: (state: ScenarioExecutionStateLike) => void | Promise<void>

    Optional callback executed at the end of each turn. Receives the current execution state

  • OptionalonStep: (state: ScenarioExecutionStateLike) => void | Promise<void>

    Optional callback executed after each agent interaction. Receives the current execution state

  Returns Promise<ScenarioResult | null>

  A promise that resolves with:

  • ScenarioResult if a conclusion is reached during the proceeding, or
  • Null if the specified turns complete without conclusion

  Example

  // Proceed for 5 turns
  const result = await execution.proceed(5);
  
  // Proceed until conclusion with callbacks
  const result = await execution.proceed(
    undefined,
    (state) => console.log(`Turn ${state.currentTurn} completed`),
    (state) => console.log(`Agent interaction completed, ${state.messages.length} messages`)
  );
  Copy

step

• step(): Promise<void>

  Executes a single agent interaction in the scenario.

  This method is for manual step-by-step execution of the scenario, where each call represents one agent taking their turn. This is different from script steps (like user(), agent(), proceed(), etc.) which are functions in the scenario script.

  Each call to this method will:

  • Progress to the next turn if needed
  • Find the next agent that should act
  • Execute that agent's response
  • Set the result if the scenario concludes

  Note: This method is primarily for debugging or custom execution flows. Most users will use execute() to run the entire scenario automatically.

  After calling this method, check this.result to see if the scenario has concluded.

  Returns Promise<void>

  Example

  const execution = new ScenarioExecution(config, script);
  
  // Execute one agent interaction at a time
  await execution.step();
  if (execution.result) {
    console.log('Scenario finished:', execution.result.success);
  }
  Copy

succeed

• succeed(reasoning?: string): Promise<ScenarioResult>

  Immediately ends the scenario with a success verdict.

  This method forces the scenario to end successfully, regardless of the current conversation state. It's useful for scenarios where you want to explicitly mark success based on specific conditions or external factors.

  This method is part of the ScenarioExecutionLike interface used by script steps.

  Parameters

  • Optionalreasoning: string

    Optional explanation for why the scenario is being marked as successful

  Returns Promise<ScenarioResult>

  A promise that resolves with the final successful scenario result

  Example

  // Mark success with default reasoning
  const result = await execution.succeed();
  
  // Mark success with custom reasoning
  const result = await execution.succeed(
    "User successfully completed the onboarding flow"
  );
  Copy

user

• user(content?: string | ModelMessage): Promise<void>

  Executes a user turn in the conversation.

  If content is provided, it's used directly as the user's message. If not provided, the user simulator agent is called to generate an appropriate response based on the current conversation context.

  This method is part of the ScenarioExecutionLike interface used by script steps.

  Parameters

  • Optionalcontent: string | ModelMessage

    Optional content for the user's message. Can be a string or ModelMessage. If not provided, the user simulator agent will generate the content.

  Returns Promise<void>

  Example

  // Use provided content
  await execution.user("What's the weather like?");
  
  // Let user simulator generate content
  await execution.user();
  
  // Use a ModelMessage object
  await execution.user({
    role: "user",
    content: "Tell me a joke"
  });
  Copy