About the role
Nouvells is seeking AI Evaluation Specialists who can assess the quality, accuracy, relevance, safety, and usefulness of AI generated outputs. You'll help organizations understand how effectively AI systems perform in real-world scenarios and identify opportunities for improvement.
As our AI Evaluation Specialist, you will own the quality, safety, and qualitative benchmarking of Nouvell’s models and agentic workflows. You will design and execute evaluation frameworks that measure nuanced LLM performance—detecting subtle hallucinations, evaluating multi-step reasoning, testing conversational quality, and running safety stress-tests. This role sits at the intersection of technical operations, linguistics, and machine learning engineering. You will translate qualitative business expectations into programmatic benchmarks and repeatable evaluation suites, bridging human evaluation with automated LLM-as-a-judge pipelines.
Key responsibilities
- Design and maintain comprehensive, domain-specific evaluation benchmark suites across core metrics: factual accuracy, hallucination rates, style adherence, latency, and context retention.
- Create curated "golden datasets" and adversarial test sets that challenge models on edge cases, multi-turn context drift, and ambiguity.
- Manage qualitative review cycles for new model versions, prompt iterations, and fine-tuned checkpoints before deployment to production.
- Build automated evaluation workflows (LLM-as-a-judge) and validate that automated scoring closely correlates with human consensus.
- Audit annotation outputs for inter-annotator agreement (IAA) using statistical measures (e.g., Cohen’s kappa) and calibrate scoring discrepancies.
Qualifications
- Bachelor’s Degree
- Equivalent practical experience: A degree from an accredited institution in a relevant quantitative, technical, or analytical field