1
$\begingroup$

I'm curious what the broader statistics community thinks of synthetic control in the context of experimentation. In my work, we often have a low sample size (50-100 total randomization units.) And no matter how we randomize, we often end up in a situation where prior to experiment treatment exposure, there is an imbalance in the target metric.

We've typically resorted to DiD, using the parallel trends assumption to produce a counterfactual of how the treatment group would have behaved in the post-start period and calculated the lift therein. However, when sample sizes are so small, this assumption is often not valid.

Hence, we are interested in something scalable. From what I've read about synthetic control, it should be able to do the job by dynamically weighting control group randomization units to produce a target metric mean (near) equal to the treatment group mean in the absence of treatment exposure.

What are the community's thoughts on this technique in context of the this low data problem with persistent target metric imbalance?

$\endgroup$
2
  • $\begingroup$ The post-experiment balancing of covariates leaves one with a weighted sample, which my intuition tells me has a lower effective sample size and hence bigger confidence intervals. Therefore, I think that it is better to do the balancing pre-experiment either through stratified sampling (as suggested by AdamO) or similar-purposed techniques such as individual matching or covariate-constrained optimization. $\endgroup$
    – svendvn
    Jun 8, 2023 at 22:14
  • $\begingroup$ Heads up- my shop is almost exclusively Bayesian. P-values are essentially taboo. Credible intervals from the posterior are typically used instead. $\endgroup$
    – jbuddy_13
    Jun 9, 2023 at 14:34

1 Answer 1

1
$\begingroup$

What do you mean by "target metric"?

The point of randomization is not to actually balance covariates, but to balance them in probability. You can convince yourself of this by simulation. If you actually want to balance covariates, you can perform stratified permuted block randomization. This of course takes longer and costs more - if the reason for so few subjects is that few if any subjects are eligible to participate in the study, then this kind of randomization scheme isn't feasible. If the cost of recruiting these patients is too much for a small pharma, that's another story.

Balance cannot be assessed by hypothesis testing. Just the same, the more such values you declare to be "target metrics" - the more likely you are to find one or more that is blatantly unbalanced (regardless of statistical significance). Once ten or so such metrics are identified, at least one is very likely to seem "imbalanced". This is just a feature of conducting clinical research.

In all the literature I've consulted, failure to balance a small study has not been (successfully) cited as a reason to use a synthetic control - not of which I'm aware.

$\endgroup$
5
  • $\begingroup$ Target metric is tech speak for outcome of interest, for example conversion rate. And we frequently find a 0.5-1.0% imbalance in conversion rate in experiments with <100 advertisers $\endgroup$
    – jbuddy_13
    Jun 9, 2023 at 7:08
  • $\begingroup$ Meaning you randomize and yet by chance the treat and ctrl group conversion rate means (w/o exposure) are different by almost 1%. This is huge a deal when the baseline is low (5-10%) and an increase in this rate by as little as 0.2% would be impactful for the business. $\endgroup$
    – jbuddy_13
    Jun 9, 2023 at 7:11
  • $\begingroup$ Fisher said, "As ye randomize so shall ye analyze". There is no possible way that randomization by virtue of what it is biases an analysis by a measurable factor. What you're describing is probably a Hawthorne effect. Consider a better design. Synthetic control is 100% sure to make things way worse. I've proposed studies with very long follow-up so that you can apply regression to the mean, that is if there's an observer effect maybe it tapers off over time and you can make some extrapolations. $\endgroup$
    – AdamO
    Jun 9, 2023 at 16:37
  • $\begingroup$ 2/2 what I mean about "regression to the mean" is that if there's an observer effect you might expect it to show some temporal trends and ameliorate over time. The real question is why you would randomize a condition and not expect to see a change in response - are you intentionally randomizing a non-effective condition? $\endgroup$
    – AdamO
    Jun 9, 2023 at 16:41
  • $\begingroup$ So paraphrasing and making potentially jumping to conclusions here— find the source of variability that’s messing up randomization, potentially use a block design or matched pairs design to mitigate this effect, randomize again then there shouldn't be any imbalance between treatment and control conversion rates. Fair? $\endgroup$
    – jbuddy_13
    Jun 9, 2023 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.