AcademySkillsDevelopment

coder_eval

Engineering teams that build skills, MCP servers or CLIs and want to check how coding agents behave with them.

  • Recommendedour rating
  • 151gitHub stars
  • Apache-2.0licence
  • todaylast update
git clone https://github.com/UiPath/coder_eval.git
README.md

Loading the file...

What it does

An open-source framework that tests whether your skills, MCP servers and command line tools work when a coding agent uses them. You describe the checks in YAML files, run them in a sandbox, and use the pass or fail result as a gate in CI.

Use cases

  1. 01Check that a new skill works with a real agent
  2. 02Compare two agent setups in an A/B experiment
  3. 03Block a release in CI when agent tests fail

Questions about
coder_eval.

What is coder_eval used for?

Engineering teams that build skills, MCP servers or CLIs and want to check how coding agents behave with them. An open-source framework that tests whether your skills, MCP servers and command line tools work when a coding agent uses them. You describe the checks in YAML files, run them in a sandbox, and use the pass or fail result as a gate in CI.

How do I install coder_eval?

Run this in your terminal: git clone https://github.com/UiPath/coder_eval.git

Is coder_eval open source?

Yes. The code is on GitHub (UiPath/coder_eval) under the Apache-2.0 licence.

Is coder_eval safe to use?

It passed our automatic scan for credential theft, hidden instructions and risky install commands. Third party open source software. Sabemos AI does not maintain it. Check the code and permissions before you connect it to company data.