Production AI Eval Clinic

Bring an agent that works 70–90% of the time and 100 traces.

This is the difficult middle ground: the system is useful enough to deploy but unreliable enough that aggregate success rates and static benchmarks do not explain what users actually experience.

We work through one real system end to end. The output is not a report. It is an evaluation you can run, and a hypothesis you can test in production.

The process

User goal

Trace

Failure diagnosis

Evaluation design

Intervention hypothesis

Re-evaluation

A good fit if

The system is in production or a serious pilot.
There is real usage and there are real traces.
Failures are known but not explained.
Sanitized examples can be shared.
Someone can confirm what actually happened in a given trace.

Apply

Don’t fill this out if you’re human:

Name

Organization

What does the agent do?

Is it in production?

Roughly how many traces can you share?

What problem are you currently unable to diagnose?

Can traces be anonymized and shared for research?

Would you consider a sanitized public case study?

Please do not submit traces, proprietary code, credentials, personal data, or confidential customer information through this form. We will agree on how to share material securely after we reply.