0
$\begingroup$

I have a large data frame with about 1100 columns containing integers and about 30'000 rows. The last column contains a binary random variable which attains values 0 and 1. 30% of the data frame entries miss this value and my task is to find a way to predict them. I have no a priori information on the problem: the binary variable has a general name "target", hence there is no way to make any modelling or to know which variables may have most influence on "target".

My first guess was to use a probit model, but variables are too many. Hence I elaborated the following method:

  • I select N random variables and compute a probit model, using the complete data I have
  • I compute this model's AIC score
  • I iterate this M times and choose the model with lowest AIC score to make my prediction on the incomplete data.

By testing this approach on the complete data I have, I find a success rate of about 70%.

I would like to ask if what I am doing is formally correct, if there would be any better approach in your opinion and if 70% is a good success rate.

$\endgroup$

1 Answer 1

0
$\begingroup$

I am not sure the exhaustive search would work given the # of variables you have: Even at N=3 you have ~2e8 combinations and so iterating over all the cases looks quite difficult. Besides how would you be able to tell which N to use?

A more feasible way may be to go through the variable selection processes. You can find numerous discussions online. For example, start with your 1100 columns and investigate their relationships. Some of them may be quite similar and you can find this out by using PCA. Next you can check how they are related to your target variables by looking at their IV or R2 etc. There are also some automatic variable selection algorithms like step-wise although their uses are often criticized.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.