Multilingual Audio Evaluator

About the role

Anyone AI creates high-quality training and evaluation data for frontier AI models. It is looking for multilingual audio evaluators with strong spoken-language proficiency and technical literacy to assess conversational AI agents. The work takes place in live, real-time roleplay sessions, and the project runs for 2-3 months as an independent contractor; location is remote but subject to project restrictions.

Key responsibilities

• Prepare and confirm the required tools and access before each live session.
• Conduct live, real-time roleplay scenarios with conversational AI agents.
• Capture recordings, session IDs, and tool-call or console logs.
• Label conversation transcripts according to project guidelines.
• Evaluate interactions using structured rubrics and provide clear written explanations for your scores.
• Read, validate and troubleshoot JSON, identifying structural and formatting issues.

Required profile

• C1/C2 spoken fluency in at least one of the target languages (Welsh, Dutch, Greek, Italian, French, German, Portuguese).
• Ability to prove language proficiency through a speaking and listening assessment.
• Strong attention to detail and consistent adherence to evaluation guidelines.
• Clear written communication and sound judgment when assessing AI interactions.
• Nice to have: experience in AI evaluation, audio annotation or transcript labeling, and a background in technical support, QA or software development.

Required skills

• Strong JSON skills: reading, validation, troubleshooting and identifying structural or formatting errors.
• Comfort using digital tools and collecting recordings and technical logs.
• Familiarity with conversational AI, APIs and structured data is a plus.