Playbook · 5 min read
A/B testing AI agents: how many calls you actually need
Why 30 connected calls per variant is a floor, not a finish line.
Change one thing
Test a new greeting or a new qualification question—not both—so you know what moved the result.
Measure connected calls
Compare outcomes on calls that actually reached a person. Voicemails dilute the signal—calls handled by voicemail drop are recorded as voicemails, not conversations, so they stay out of the comparison.
Be patient
Under 30 connected calls per agent, differences are mostly noise. Revenue Relay's Ask AI warns you when samples are small.