1
$\begingroup$

I have observational data with a "treatment" and I want to use propensity scores to weight the samples.

A predictive model will be built (including the case weights). I will want to include the covariates used in the propensity score in order to make predictions. Specifically I am planning on running various covariate patterns through the model with treatment = 1 and then treatment =0 to estimate the lift from the treatment, conditioned on the other X.

How can I validate the model on a hold out set or through k-fold validation that will take into account the weights? Lets say I hold out a test set from the predictive model. Do I need to use the weights to create a weighted response variable to measure accuracy or AUC or whatever criteria is used to determine if the predictive model is accurate?

$\endgroup$

1 Answer 1

1
$\begingroup$

If the holdout is representative of the population, your model should work just fine by scoring the dataset and comparing directly. The weights were used to account for over/under sampling and the estimates are created for application against a full dataset.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.