Freelance Agent Evaluation Engineer (AI Testing)

About the role

We are looking for a freelance Agent Evaluation Engineer to help build a high‑quality dataset for assessing AI coding agents. You will work remotely on a project basis, designing realistic developer environments and crafting tasks that challenge state‑of‑the‑art models.

Key responsibilities

• Create virtual company environments with codebases, infrastructure, tickets, documentation and conversation histories.
• Design evaluation tasks from intermediate states, write clear prompts and define success criteria.
• Develop automated tests that accept all correct solutions while rejecting incorrect ones.
• Iterate on tasks and tests based on QA feedback and analysis of agent performance.
• Collaborate with the Mindrift team to ensure the evaluation framework is robust and fair.

Required profile

• 5+ years of professional software development experience.
• Strong understanding of where current AI coding models succeed and fail.
• Experience writing functional and integration tests.
• Professional English proficiency (B2+).

Required skills

• Python (FastAPI)
• JavaScript / TypeScript (React)
• Docker
• PostgreSQL
• Kafka
• Redis
• Test design and implementation

What we offer

• Up to $30 USD per hour, based on experience and pace.
• Fully remote, flexible schedule.
• Opportunity to shape cutting‑edge AI evaluation datasets.