What it does

The Okareo MCP server exposes tools for creating scenarios, registering models, running evaluations, and executing multi-turn voice agent simulations from within your code editor. It is for developers building and testing voice agents who want to access Okareo's evaluation platform without leaving their editor.

Create and manage test scenarios by saving input/result data rows and organizing them into named datasets.
Run quality checks and evaluations against registered generation models to assess voice agent performance.
Execute multi-turn simulations and retrieve conversation transcripts to validate voice agent behavior.

Tools this server exposes

Tool	Does
save_scenario	Save a named scenario from rows of input/result data (idempotent).
list_scenarios	List all scenarios in the project with names, IDs, and row counts.
get_scenario	Retrieve a scenario's metadata and all data rows by name or ID.
create_scenario_version	Create a new version of an existing scenario with updated data.
preview_delete_scenario	Preview what will be deleted before removing a scenario.
delete_scenario	Permanently delete a scenario and all related test data.
list_available_llms	Browse available LLMs from the Okareo registry.
register_generation_model	Register a generation model for testing by selecting an LLM from the registry.
list_generation_models	List all registered generation models in the project.
get_generation_model	Read detailed information about a registered generation model.
update_generation_model	Change the LLM a registered generation model points to.
delete_generation_model	Remove a registered generation model and all its related test data.
list_checks	List available quality checks (built-in and custom) for evaluating model outputs.
run_test	Run a quality test that evaluates a model against a scenario using specified checks.
list_test_runs	List past test runs with optional filters (model, scenario, simulation-only).
get_test_run_results	Load detailed per-row results of a test run or simulation by ID or name.
get_conversation_transcript	Retrieve the full conversation transcript for a single data point.
reevaluate_test_run	Re-score a completed test run against a (possibly different) set of checks.
create_or_update_check	Create or update a quality check by name — model-based, code-based, or audio (upsert).
generate_check	Generate a check from a natural-language description, then save it.
get_check	Retrieve a check's full configuration, including its prompt template or code.
delete_check	Permanently delete a check by name.
list_tenants	List Okareo organizations you have access to (remote MCP only).
switch_tenant	Switch to a different Okareo organization (remote MCP only).

Before you enable it

An MCP server runs with whatever access you give it. Start with a development account rather than production, and check which tools the server exposes before you let an agent call them unattended.