Human Data Evaluations Lead

About the role

You will own Anyone AI’s data initiatives and proposals to AI labs, from initial concept through pilot delivery. This role leads the design and execution of high‑quality sample packages and benchmarks across reasoning, coding, agents, tool use and multimodal tasks.

Key responsibilities

• Transform public benchmarks and evaluation targets into proposals and sample packages that demonstrate capability and win work.
• Design and build expert‑verified sample packages with exact‑match ground truth, gold reasoning trajectories, and multi‑model headroom results.
• Recruit, brief, calibrate, and manage a pool of subject‑matter experts in coding, agentic/tool‑use, and STEM/reasoning.
• Serve as the primary contact for lab partners, handling Slack, calls, SOWs, staffing, production, QC and delivery of pilots.

Required profile

• 5+ years in technical delivery, quality or program management, with recent AI/ML data, model evaluation or benchmarking experience.
• Hands‑on experience delivering data or evaluation work to AI labs or enterprise ML teams, from scoping through delivery.
• Proven ability to lead people or vendor teams, recruiting and maintaining a high‑quality expert pool.
• Fluent English; Spanish is a plus.

Required skills

• LLM benchmarking and code‑model evaluation.
• Design and implementation of rigorous QC processes and artifact standards.
• Understanding of frontier model evaluation metrics such as benchmarks, rubrics, pass rates and headroom.