Software Engineer in Test (AI Eval)

About the Role

This role involves working as a Lead Eval Engineering Consultant, where you will be embedded directly within a high-stakes AI-DLC delivery pod. Your primary responsibility will be to design, enforce, and execute verification contracts that make every delivered outcome provable before it leaves the engineering pipeline.

You will work closely with Spec Owners during the initial design phases to ensure that loose requirements are never allowed to pass into development. Your work will ensure that engineering pipelines emit bulletproof semantic evidence, preventing false-pass behaviors and structural gaps from escaping to production.

Key Responsibilities

• Own the Feature Eval Plan: Map every acceptance criterion to a dedicated evaluation path during the Define phase.
• Establish Sufficiency Thresholds: Define the quantitative and qualitative evidence bars required to mark a criterion as Confirmed, preventing the shifting of proof standards after code execution.
• Design and Calibrate Rubrics: Construct manual review frameworks, design-fidelity metrics, and LLM-as-judge graders, verifying their alignment and calibration against human judgment.
• Manage the Unverifiable-Criteria Log: Identify, document, and log features that cannot be programmatically or structurally proven, establishing named ownership and risk mitigations rather than allowing soft passes.
• Enforce False-Pass Controls: Design the algorithmic and semantic checks that catch illegitimate green results, serving as the human counterpart to grounding gates to eliminate trivial diffs and tautological test suites.
• Execute Phase Gate Sign-Offs: Audit task DAGs during planning, verify pipeline gate signals during execution, and deliver the final independent verdict (Confirmed, Missing, Divergent, or Unverifiable) at the Outcome Review stage.

Skills & Qualifications

• Production-Level Python: Deep engineering experience extending test harnesses, working with dashboard backends, manipulating pytest configurations, and writing clean GraphQL resolvers.
• Evaluation Methodology: Proven track record designing evaluation datasets, reference tests, and reproducible scoring harnesses for LLMs or agentic workflows using tools like Inspect, Promptfoo, or Braintrust.
• Statistical Honesty: Expert-level command of hypothesis testing, significance tracking, and false-pass control mechanisms with the ability to clearly demonstrate why passing test suites do not equate to verified outcomes.
• LLM-as-Judge Calibration: Practical experience building, testing, and optimizing model-based graders, including statistical proof of their alignment with human expert benchmarks.
• Technical Defensibility: Exceptional communication skills with a demonstrated ability to defend objective technical verdicts directly to client counter parties and engineering leads under timeline pressure.
• Bachelor's or Master's degree in Computer Science or related field.

What You'll Learn

In this role, you will have the opportunity to learn about the latest advancements in AI evaluation and verification techniques. You will work with cutting-edge tools and technologies, including LLMs and agentic workflows, and develop your skills in designing and implementing evaluation datasets, reference tests, and reproducible scoring harnesses.

You will also learn about the importance of statistical honesty and false-pass control mechanisms in ensuring the integrity of AI systems. Additionally, you will develop your skills in communicating complex technical concepts to non-technical stakeholders and defending objective technical verdicts under pressure.

Resume Tip

When applying for this role, make sure to highlight your experience with production-level Python, evaluation methodology, and statistical honesty. Emphasize your ability to design and implement evaluation datasets, reference tests, and reproducible scoring harnesses, as well as your experience with tools like Inspect, Promptfoo, or Braintrust.

Also, be sure to mention any experience you have with LLM-as-judge calibration, technical defensibility, and communication skills. Finally, highlight any relevant certifications or training programs you have completed in AI evaluation, verification, or related fields.