Search Evaluation Specialist - Expert | Upto $30/hr Hourly

About The Job

Mercor connects elite creative and technical talent with leading AI research labs. Headquartered in San Francisco, our investors include

Benchmark

,

General Catalyst

,

Peter Thiel

,

Adam D'Angelo

,

Larry Summers

, and

Jack Dorsey

.

Position

Search Generalist Expert

Type:

Contract

Compensation

$10–$30/hour

Location:

Remote

Role Responsibilities

• Evaluate AI-generated search responses for factual accuracy, helpfulness, clarity, completeness, and overall quality.
• Assess whether models use search appropriately and whether search queries are well-formed and effective.
• Compare model responses side by side and provide concise, defensible rationales.
• Write and refine prompts, golden answers, rubric criteria, and edge cases for search-related evaluations.
• Apply project guidelines consistently across ambiguous, multi-step, and real-world search tasks.
• Identify recurring failure modes and escalate unclear cases or rubric gaps to project leads.

Qualifications

Must-Have

• Excellent written English and strong online research skills.
• Strong judgment when synthesizing information from multiple sources.
• Ability to distinguish factual accuracy from fluency, confidence, or style.
• High attention to detail and comfort following structured guidelines.
• Reliable, self-directed, and responsive in an asynchronous remote environment.

Preferred

• Experience in search quality, fact-checking, content evaluation, trust and safety, annotation, QA, or prompt/rubric writing.
• Familiarity with search evaluation concepts such as factuality, helpfulness, severity, side-by-side comparisons, or tool-use assessment.
• Experience working with LLM evaluation workflows or human data projects.
• Multilingual skills are a plus.
• Bachelor’s degree preferred; advanced degree or strong professional background is a plus.

Application Process (Takes 20–30 mins to complete)

• Upload resume
• AI interview based on your resume
• Submit form

Resources & Support

• For details about the interview process and platform information, please check: https://talent.docs.mercor.com/welcome
• For any help or support, reach out to: support@mercor.com

PS: Our team reviews applications daily. Please complete your AI interview and application steps to be considered for this opportunity.

,