Data for what’s next.

Given a problem statement and a held-out gold patch, the agent reconstructs a target program using only observed program behavior, following the official ProgramBench format. A task is solved only when every test passes; the grader awards no partial credit.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Sample

sharkdp__numbat

Listed production capacity

50 tasks / day

Capacity as listed in the September 2026 catalog.

The agent implements a real requirement in a live repository. A task is solved only when every PASS_TO_PASS and FAIL_TO_PASS test passes and none of the previously passing tests regresses.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Sample

fastify-content-negotiation

Listed production capacity

50 tasks / day

Capacity as listed in the September 2026 catalog.

No patch or test constraints are required. The agent investigates a real codebase and provides a written answer. Each criterion in a written rubric is independently verified, and all criteria must pass for the task to be solved. The assessment is qualitative but strict.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Sample

qna-nats-jetstream-consistency

Listed production capacity

50 tasks / day

Capacity as listed in the September 2026 catalog.

The agent optimizes for a measurable performance budget. There is no binary pass/fail threshold: the continuous score can fall between the negative control and reference implementation, or exceed the reference implementation.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Sample

fast-decimal-to-double

Listed production capacity

50 tasks / day

Capacity as listed in the September 2026 catalog.

Prioritizes coding and debugging across Debug, DevOps, SWE, and code implementation. Task formats include incomplete or buggy software repositories, environments for building GUIs and apps, and verifiable frontend visualization environments.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

A benchmark for terminal and command-line agents covering real terminal tasks, including environment setup, file and script operations, debugging, and DevOps.

Availability

Production is complete. Data can be delivered in the quantities required.

A software engineering agent benchmark built around issue-resolution tasks in real repositories. The catalog describes higher difficulty and longer context than SWE-bench, with coverage of all mainstream programming languages.

Availability

Production is complete. Data can be delivered in the quantities required.

Starting with screenshots and original assets from real products, the agent rebuilds the code, a running service, usable interactions, and responsive APIs from scratch. Evaluation happens after execution: a GUI agent drives a real browser and actual HTTP requests reach the live service. Assets are sourced from original sites, rather than synthesized.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Sample

gimmecoffee, longreads, spicewalla

Listed production capacity

80 tasks / day

Capacity as listed in the September 2026 catalog.

Image-to-code data pairing UI screenshots or design mockups with frontend code, including HTML/CSS and components. Includes image-code pairs and reconstruction-fidelity evaluation for image2code SFT and visual validation.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Evaluates large vision-language models (LVLMs) on reconstructing interactive webpages from videos of user interactions. Cases include trajectories.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

High-quality SVG, CSS, and Canvas work with source code, including SVG illustrations, CSS art, UI motion, and Canvas animation. Covers object drawing, animated effects, interactive components, data visualization, physics and chemistry simulations, mathematical demonstrations, and creative CSS elements.

Availability

Some finished data is available; the remainder can be produced to your specifications.

Frontend page and component annotations covering webpage elements, layout structures, component types, and interaction states.

Availability

Production is complete. Data can be delivered in the quantities required.

Simulated web environments built from real web applications. Each application includes realistic tasks at different complexity levels and task-specific verifiers, supporting more accurate browser-use agents.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Long-horizon computer-use tasks designed for everyday and professional workflows. Includes task descriptions, required asset files, and task-specific verifiers, targeting agent capability on tasks lasting more than 1.5 hours.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Skill-invocation trajectories.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Skill-invocation trajectories for RL.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Skills sandbox validation.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Tool generalization.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Tool policy optimization.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Multi-turn Python interaction trajectories for image-processing tasks.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Image-processing validation using function calling.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Multimodal search agent tasks.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Multimodal search validation.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Multi-image composition and editing tasks based on multiple input images, including stitching, style fusion, local edits, and consistency-aware generation. Each task includes an input image set, an instruction, and a target image for multimodal generation and image-editing capabilities.

Availability

Production is complete. Data can be delivered in the quantities required.

Uses a problem / solutions / answer format, covering STEM knowledge, logical reasoning, and precise comprehension.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Expert rubric annotation in the medical domain.

Availability

The production pipeline is ready and samples are available. Full datasets are produced to your specifications.

Game scripts from Chengguang (澄光, as named in the source catalog) and Steam.

Availability

Production is complete. Data can be delivered in the quantities required.

Some available datasets are not listed in the catalog. Contact us to discuss additional data needs.

Availability

Existing data or samples are available. Contact us to discuss the specific dataset.