mcpdirectory.dev
9,129 servers · updated 23 Sept

Evals MCP server

community Active last commit 18 d 
{
  "mcpServers": {
    "completionkit-evals": {
      "type": "http",
      "url": "https://completionkit.com/mcp"
    }
  }
}
file .mcp.json (in your project root)version 1.0.0

What it does

Evals is an MCP server that runs prompts against datasets and scores outputs 1-5 using an LLM judge. It lets you test and compare prompt versions, models, and parameters to measure performance changes numerically.

  • Run the same prompt against 200+ real-world inputs and get aggregate quality scores to validate prompt improvements.
  • Compare two versions of a prompt side-by-side by running both against your dataset and viewing score differences.
  • Experiment with model selection, temperature, or phrasing, then re-run evaluations to see which changes improve scores and which break existing behavior.

Tools this server exposes

ToolDoes
run_prompt_evaluationExecute a prompt against a dataset and score each output with an LLM judge.
compare_prompt_versionsRun multiple prompt versions on the same dataset to compare their scores.
upload_datasetLoad test data from CSV for prompt evaluation.

Before you enable it

An MCP server runs with whatever access you give it. Start with a development account rather than production, and check which tools the server exposes before you let an agent call them unattended.