1
$\begingroup$

I have a classification problem with about 10,000 records. I have twenty predictors and I have data for most of the predictors.

Some of the predictors provide valuable information, but I only have data on some of the rows. The model has to do with user behavior and the data is structurally missing, since it is information that has only been added in the last couple years (out of a sample set that references five years of behavior).

The missing data ranges from 1% missing on the low end to 85% missing on the high end. Most of the variables with missing data are missing on 50%-60% of records.

I figure that I can predict the data for records with less than 15% missing and that I can drop the variables for records with more than 80% missing.

What do I do about the variables in between? All the variables with missing data are categorical. Would I add bias if I made an "Unknown" category for each of them?

$\endgroup$
  • $\begingroup$ I know it's encouraged to add an example. Any suggestions for this type of problem? $\endgroup$ – Cauder Mar 4 '19 at 16:11
1
$\begingroup$

Adding "Unknown" is a valid way to model missingness explicitly. It would not break your data. Missing data is usually classified as Missing at Random (MAR), Missing Completely at Random (MCAR), or Not Missing at Random (NMAR). This is covered in many places, check this link for a gentle intro. Modelling missingness explicitly is applicable for any of the classifications including NMAR, which is the trickiest to deal with / impute.

By the way, there are some algorithms, like Bayesian Networks or XGBoost, that deal with missing values automatically so you do not need to treat them. Indeed Bayesian Networks are usually more practically suitable for use cases where all variables are categorical (which simplifies the calculation of conditional probabilities). Hope these help.

$\endgroup$
  • $\begingroup$ What other models do you recommend for cases where most variables are categorical? $\endgroup$ – Cauder Mar 4 '19 at 20:12
  • $\begingroup$ i would search this website for a thread on this, i am sure it exists. $\endgroup$ – Zhubarb Mar 5 '19 at 12:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.