I am working on uplift model to predict the heterogeneous treatment effect on different experimental units. This model has some unrelated exogenous covariates that were collected before the experiment. So I would not expect these covariates to be significantly different between treatment and control units. This is true in the data I have that no covariate is significantly difference between treatment and control.
However, to validate this model I did a training and test split using sklearn. I am very surprised to find that some of the exogenous covariates become significantly different between treatment and control group in the training and test data.
I am confused by this for several reasons
If training and test data is a random split, why would I be seeing that covariates confound with treatment assignment?
training and test data is a smaller subset from the original data, If so it should be having smaller power to detect any significant differences, why am I only seeing significant differences in the covariate in the training and test data, not the original data?
Is it possible that some covariates are just statistically significant by chance in the small subset from the original data set given I have 100+ covariates? Every time I split the train and test, it is usually a different set of covariates that is significant.