Evaluation supply: tasksets, graders and RL environments.

What this covers

Environments, tasksets, graders, and corpora.

• Environments. Runnable RL environments: scored, reproducible, and specific to a target skill rather than a generic benchmark.
• Tasksets. Structured task collections for benchmarking or fine-tuning, built to a written spec.
• Graders. Evaluation logic built to resist reward hacking — checking for the actual property you want, not pattern-matching on output shape.
• Corpora. Curated traces and examples, sourced under license or authored to spec.

Source or build

Two different jobs, both scoped to what you need.

Some evaluation supply already exists in a usable form and needs sourcing, cleaning, and licensing rather than authoring from scratch. Some has to be built from a written spec: a target skill, a scoring method, and a reference solution, evaluated the same way the screening method on this site evaluates engineers — against a rubric, not a vibe. We do both, and say up front which one a given request is.

What is published now

Nothing yet — the first one goes here.

There is no catalogue on this page, on purpose. When the first environment ships, it gets published here as an open artifact with its grader and a short write-up, so you can inspect the work before asking for more of it. That published artifact is the proof this page does not yet have.