TD · Product Owner
Experimentation Practice
1M+
Users Served
+18%
Engagement
-12%
Churn
Overview
Created a rigorous experimentation practice on a real-time decisioning platform, replacing unverified “wins” with pre-registered tests, holdouts and measurable outcomes.
The Challenge
The platform served more than one million users but had no structured experimentation process or defined success metrics. Earlier “wins” lacked holdout groups, and an expensive data-enrichment initiative was not addressing why analysts overrode recommendations.
The Approach
Interviewed analysts, identified interpretability as the issue, and redirected engineering toward an explainability layer while protecting the sub-500ms latency requirement. Co-authored experiment standards with data science and engineering, then ran the first test as a public proof of concept.
Key Outcomes
- Built pre-registration, holdout and significance standards for experiments.
- Ran 15+ experiments; engagement improved 18%, conversion 18% and churn fell 12%.
- The experimentation framework was adopted by three other product teams.
- Redirected engineering from a low-value data-enrichment initiative, avoiding about $2M in annual spend.
The Result
Fifteen-plus experiments produced attributable engagement, conversion and churn improvements. Analyst overrides fell from 40% to under 20% after the explainability work, and the practice was adopted by three other teams.