AI Critical Thinking Evaluator

Job Description

The AI Critical Thinking Evaluator is responsible for examining whether artificial intelligence systems demonstrate balanced judgement, sound analysis, evidence-based conclusions, and the ability to consider different perspectives when responding to complex questions. This role focuses on assessing the quality of AI-generated analysis rather than simply checking whether an answer is grammatically correct or factually plausible.

The successful candidate will review responses involving arguments, opinions, decisions, controversial subjects, hypothetical situations, and complex scenarios. They will determine whether AI models recognise relevant evidence, identify assumptions, distinguish strong arguments from weak ones, and consider alternative explanations before reaching conclusions.

The evaluator will also assess whether AI-generated responses demonstrate appropriate reasoning when dealing with incomplete information or competing viewpoints. They will identify overgeneralisation, biased interpretation, unsupported conclusions, inconsistent arguments, and failures to recognise important limitations.

This position is essential for improving the ability of AI systems to provide thoughtful, balanced, and well-reasoned responses across educational, professional, and general-use environments.

Key Responsibilities:

Evaluate AI-generated responses for critical-thinking quality and analytical depth.
Assess whether arguments are supported by relevant evidence.
Identify logical weaknesses, unsupported assumptions, and overgeneralisation.
Review how AI systems handle conflicting information and alternative viewpoints.
Determine whether conclusions are balanced and appropriately qualified.
Evaluate the ability of AI models to distinguish facts from opinions.
Identify one-sided reasoning and unjustified generalisations.
Assess whether responses recognise uncertainty and limitations.
Compare multiple answers based on analytical strength and objectivity.
Provide detailed feedback to support improvements in AI reasoning.
Qualifications and Requirements:

Bachelor’s degree in Philosophy, Psychology, Law, Political Science, Research, or a related discipline is preferred.
Excellent critical-thinking and analytical reasoning abilities.
Strong understanding of argument structure and evidence evaluation.
Ability to identify logical fallacies and reasoning weaknesses.
Excellent written communication and comprehension skills.
Ability to assess complex and competing perspectives objectively.
Familiarity with AI-generated content evaluation is an advantage.
Strong attention to detail and professional judgement.