I am working on uplift model to predict the heterogeneous treatment effect on different experimental units. This model has some unrelated exogenous covariates that were collected before the experiment. So I would not expect these covariates to be significantly different between treatment and control units. This is true in the data I have that no covariate is significantly difference between treatment and control.

However, to validate this model I did a training and test split using sklearn. I am very surprised to find that some of the exogenous covariates become significantly different between treatment and control group in the training and test data.

I am confused by this for several reasons

  1. If training and test data is a random split, why would I be seeing that covariates confound with treatment assignment?

  2. training and test data is a smaller subset from the original data, If so it should be having smaller power to detect any significant differences, why am I only seeing significant differences in the covariate in the training and test data, not the original data?

  3. Is it possible that some covariates are just statistically significant by chance in the small subset from the original data set given I have 100+ covariates? Every time I split the train and test, it is usually a different set of covariates that is significant.


1 Answer 1

  1. A random subset of the data is not necessarily balanced for all covariates. If you were to repeatedly partition the data, partitions should be balanced on average, but this does not mean that an individual partition will be. You can demonstrate this with a really trivial example in R. You can see that in some cases the proportion of class "a" varies quite widely from the population proportion.

a <- sample(letters[1:2], 1000, replace=TRUE)
m <- mean(a == "a")
r <- replicate(10000, {
  x <- sample(a, 100)
  mean(x == "a")
hist(r, breaks = "FD")
abline(v = m, lty = "dashed", col = "dodgerblue")

  1. Low statistical power does not prevent spurious false positives.
  2. That is exactly what's happening. It's generally very difficult to partition test and training data while balancing for even a relatively low number of covariates (~10), even if you have a fairly large sample size.
  • $\begingroup$ thank you so much Alan! In cases where training and test data set are not balanced, I can see that for predictive problem, it may not be a big concern. That is also why I never looked into the bias in features before as no results seems too off from training to testing. but for causal inference models that predict the treatment heterogeneous effect, this seems really bad if covariates are confounded somehow by the split ? $\endgroup$
    – Chi Yuan
    Mar 2, 2020 at 17:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.