Papers, technical reports and methodology notes published by the same teams running our production pipelines. Reproducible by design — what we write up is what we ran.
Evaluation · Paper
Rather than measuring success solely through final outcomes, this approach evaluates how effectively an agent performs each component of a task and integrates these components to achieve a goal.
Evaluation
Internal report
Quantifying how grading rubrics decay when applied to multi-step agent runs — and a calibration loop that recovers signal.
Jul, 2026.
R. Patel, J. Costa
Methodology
Comparing expert vs. crowd labelling across legal, medical and STEM corpora — and what closes the gap.
Jun, 2026.
M. Larsen, S. Okafor
Multimodal
EMNLP 2026
Methodology for blending image, video and audio instruction data in training mixes without collapsing modal-specific capabilities.
Jun, 2026.
S. Okafor, A. Reyes, R. Patel
Agents
ICML 2026
How synthetic execution traces lift downstream coding benchmark scores — with a recipe for keeping the traces from over-fitting.
Jun, 2026.
J. Costa, N. Adeyemi
Methodology
A practitioner guide to budgeting expert review against agreement floors — and where multi-layer review buys itself back.
May, 2026.
M. Larsen
Multimodal
Internal report
An internal protocol for annotating long-form video events with sub-second precision and inter-annotator agreement above 94%.
Apr, 2026.
A. Reyes, S. Okafor
Evaluation
ICLR 2026
A targeted red-team test bank for code-generation models — and what it reveals about benchmark headroom.
Mar, 2026.
R. Patel, J. Costa, N. Adeyemi
Data
How preference annotations drift between batches and what a publishable audit trail looks like in practice.
Feb, 2026.
S. Okafor, M. Larsen
01
Stream
Agent scaffolding, long-horizon planning and the production patterns that make them survive contact with deployment.
02
Stream
Sourcing, labelling and curation methodology — written by the same operators running the pipelines.
03
Stream
Cross-modal training mixes and annotation protocols across text, image, video and audio.
04
Stream
Benchmarks, rubrics and adversarial probes — designed to grade what production actually exposes.
Collaborations
Frontier Lab — undisclosed
Co-authored evaluation methodology shipped alongside a frontier model release.
Frontier lab
Sovereign AI initiative — EU
Methodology partnership covering multilingual annotation and audit standards.
Sovereign research
ETH Zürich — DS3Lab
Joint research on synthetic data provenance and reproducibility tooling.
Academic
An AI-native enterprise partner
Production deployment of evaluation harnesses against an internal coding-agent fleet.
Industry
Ongoing services
Multi-rubric grader for tool-use agents
Evaluation researchCross-lingual preference data benchmarks
Data researchLong-context multimodal training mix
Multimodal researchOpen methodology note on annotator calibration
Data researchI.
Every paper, report and methodology note is authored by the same teams running the production pipelines they describe. No detached commentary, no consulting layer.
II.
Reproduction artifacts — datasets, rubrics, harnesses, prompts — ship with every publication where the licensing allows. What we write up is what we ran.
III.
Methodology notes return upstream into Collection and Annotation rubrics before they reach a paper. The loop tightens with every release cycle.
Publications, YTD
Papers, technical reports and whitepapers across all streams.
Cadence
Public publication cadence; internal methodology notes monthly.
Co-authored
Publications co-authored with frontier labs or sovereign research.
Reproduction artifacts
Of publications shipping with open reproduction artifacts.
