AI Instruction-Following Evaluator

Job Description

The AI Instruction-Following Evaluator is responsible for assessing how effectively artificial intelligence systems understand, interpret, and execute instructions provided by users. The position focuses on determining whether AI-generated outputs accurately reflect the user’s requested task, comply with specified requirements, and follow formatting, language, content, and procedural instructions.

The evaluator will test AI systems using different instruction scenarios, including multi-step requests, conditional instructions, formatting requirements, complex questions, and tasks involving multiple constraints. The role requires exceptional attention to detail, logical reasoning, and the ability to distinguish between responses that merely appear relevant and those that fully satisfy the original instructions.

Key Responsibilities:

Evaluate whether AI systems correctly follow user instructions.
Test model responses against predefined task requirements.
Assess compliance with formatting, language, and structural instructions.
Identify incomplete responses and deviations from user requirements.
Evaluate the handling of multi-step and complex instructions.
Examine whether AI systems maintain previously established constraints.
Identify situations where models misunderstand or ignore important details.
Compare responses based on instruction compliance and task completion.
Prepare detailed feedback on instruction-following failures.
Collaborate with model training teams to improve instruction adherence.
Qualifications and Requirements:

Bachelor’s degree in Computer Science, Linguistics, Communications, English, or a related discipline.
Strong logical reasoning and analytical abilities.
Excellent comprehension of complex written instructions.
Experience in AI evaluation, content assessment, or quality assurance.
Ability to identify subtle differences between requested and delivered outputs.
Familiarity with generative AI systems and prompt-based workflows.