The decision
Test the candidate, not the correlation.
eReflect's Typesy VIP team had a candidate behavior that appeared before the business outcome: complete at least one course within the first four days.
The experiment asked two separate questions. Could an intervention move that behavior? If it did, would trial conversion move too?
Keeping those questions separate matters. A behavior can be responsive to treatment without causing or predicting the outcome a team cares about.
The experiment preserved the holdout.
Treatment received the activation intervention. Control retained the existing experience.
Both groups used the same candidate-indicator definition, four-day window, and trial-conversion outcome. The comparisons below are intent-to-treat.
We moved the behavior. Conversion did not move.
| Intent-to-treat result | Treatment | Control | Difference |
|---|---|---|---|
| Complete ≥1 course within first 4 days | 69.4% (129/186) | 57.7% (56/97) | +11.6pp (+20.1% relative) |
| Trial conversion | 76.8% (136/177) | 76.7% (66/86) | +0.1pp |
The indicator difference was +11.6 percentage points, or +20.1% relative. Its z-score was approximately 1.95 and p-value approximately 0.051: borderline, not clean statistical significance.
Trial conversion differed by +0.1 percentage points. There is no conversion lift in these results.
The leading explanation for the null is a ceiling effect. About 77% of trials converted with no intervention at all, leaving very little headroom.
The experiment supports only the two intent-to-treat comparisons above. It does not establish a revenue lift, a conversion lift, or that completing a course caused conversion.
The control arm challenged the candidate itself.
Inside the control arm, which was unexposed and therefore free of post-treatment contamination, indicator hitters converted at 75.5% (n=49). Non-hitters converted at 78.4% (n=37).
Hitting the indicator did not predict conversion in this clean comparison. The confidence interval is roughly −21 percentage points to +15 percentage points, so a moderate effect cannot be ruled out.
Method notes and limits.
| Candidate indicator | Complete at least one course within the first four days. |
|---|---|
| Indicator cohort | 283 users: 186 treatment and 97 control. |
| Trial-conversion cohort | 263 users with conversion outcomes: 177 treatment and 86 control. |
| Source | 2026-08-04 Supabase experiment_outcomes pull for eReflect's Typesy VIP experiment. |
| Excluded comparison | We do not report the treatment/no-hit subgroup cross-tab. pai_hit is post-treatment; conditioning on it breaks randomization through collider bias. |
The indicator result is borderline, the conversion result is null, and the control-arm confidence interval remains wide.
The operating lesson.
Moving a plausible behavior is not enough. The outcome must move under a real holdout before a team treats that behavior as a useful lever.
This null is the value of validation: it can stop a lifecycle team from scaling an intervention around the wrong signal.