0
$\begingroup$

I have a dataset of 54,681 rows and 10 features. I'm trying to predict whether or not a user will complete an a specific action, ie a binary target label. There are several features such as signup_date, item_added_date and action_completed_date. My problem is that all of these data are only recorded for users that complete the action, if they don't then these features are recorded as NaN. Therefore, I only have complete information for users who actually complete the action and introduces bias into the data. Estimation techniques such as Logistic regression, naive bayes, or random forests are all incompatible with the NaN data type.

I've been looking at scikit-learn's methods for data imputation. Of the 54,681 rows only 6,137 have complete data. Is it a reasonable technique to compute some imputation when the amount of missing data is nearly ten times the number of complete observations? Or should I rethink my entire approach?

$\endgroup$
2
$\begingroup$

By the explanation the missing data is not completely random and independent of the process, therefore they can add information to the model.

You can apply some transformations to the data to identify the missing values as a numeric value, using either the average value of the variable or, for example, $-1$ for positive variables.

You can also create an additional variable that counts how many missing values exist for each sample or just a flag indicating the existence or not of missing values.

Nonlinear models tend to deal well with the empty information encoded in a label, but some linear models can behave poorly, this will depend on analysis of the technique used.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.