0
$\begingroup$

When tuning causal models (e.g. uplift models) using cross validation, how important is the statistical significance of the measurement (e.g., the difference between group A and B, or Control and Treatment) in each training and validation fold? Can a model be well tuned if the folds are not statistically significant?

Update

I am building uplift models which are design to capture the causal effect between a control and treatment groups. I am using xgboost as the meta learner, xlearner from the causalml python package and the Transformed Outcome method from the pylift python package. The training data set has a statistical significant effect (i.e, the difference between the treatment and control is statistical significant); however, when using a CV to tune the model some of the training and validation folds are not exhibiting statistical significance any more. Therefore, I am wondering if this will void the quality of the validation process. In addition, should the effect measured in the overall training dataset be identical to the effect in each training and validation folds or is this not necessary?

$\endgroup$
6
  • $\begingroup$ CV will allow you to assess the performance of your model in an unbiased manner. But if you just want one model to determine which variables are significant, you can simply use all the data and just build one model. $\endgroup$ Jan 7, 2021 at 15:28
  • $\begingroup$ @NuclearHoagie the goal is not to determine which features/variables are statistically significant but rather to determine if the measures effect (the difference between group A and B) is statistically significant. $\endgroup$ Jan 7, 2021 at 17:45
  • $\begingroup$ I think CV only complicates matters here - what if you find that there is a difference in half the folds, and that there is no difference in the other half? An analysis of the full data will give you one single answer which has greater statistical power than any fold individually, since it's derived from more data. CV is only useful for evaluating the performance of a model, it doesn't help at all in building the model or measuring effect sizes. $\endgroup$ Jan 7, 2021 at 17:50
  • $\begingroup$ @NuclearHoagie The way that these casual models work to capture the treatment effect is by utilizing meta learners (such as linear models and tree-based models). Therefore, these models need to be tuning (i.e., determine hyperparameters) which can be done using CV. Are you suggesting a simple single train/test split for model tuning? $\endgroup$ Jan 7, 2021 at 19:06
  • $\begingroup$ One approach might be to take 80% of the data, perform cross-validation to determine the best hyperparameters, and then train the model with tuned hyperparameters on the 80% and test on the never-used 20%. The 20% is only useful for assessing the model, but not building it - you could instead use 100% of the data in CV to get the hyperparameters and build a model on that 100% at the end, but at the cost of not being able to evaluate it. $\endgroup$ Jan 7, 2021 at 20:18

1 Answer 1

2
$\begingroup$

Statistical significance of an effect or model parameter is not required in each fold of a cross validation. A meta-parameter is "well tuned" in a machine learning setting if it minimizes the cost function, regardless of any estimate of statistical significance. If, however, your effect of interest is not significant in any cross-validation fold, and presumably not significant overall, then your effect is likely not there. I can't give more specific help without more specifics on your model. What model are you training? How are you assessing significance? etc.

$\endgroup$
1
  • $\begingroup$ I updated the original question with additional clarifications $\endgroup$ Jan 7, 2021 at 15:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.