1
$\begingroup$

In an R Bloggers post, the author suggests that if more than 5% of the observations in a sample feature is missing, you might want to consider dropping that sample feature entirely. My question is, is there a certain threshold where it's acceptable to drop an entire feature due to missingness? If so, what's an acceptable threshold (i.e. at what percentage should you drop an entire feature completely)?

I have an explanatory variable in my data set where 99% of the values for the last purchase date for a customer is missing. The goal of the analysis is essentially to identify who the top customers are and how they vary from the others. Sure, the last purchase date intuitively seems like an important feature that could explain whether or not a customer is a "Top Customer," but with 99% of the data missing, what are my options?

$\endgroup$
1
$\begingroup$

No, there is no such threshold. Considerations like that will always depend on the setting and the role of the variable in the analysis.

Some examples:

  1. You have a response variable in a model with 20% missing values. If you remove this column, there will be nothing to analyze. You would just remove the lines without response and mention that 20% rows without response had to be discarded.

  2. You want to compute the correlation between two key variables. Even if there are many missing values in the two columns, you cannot just remove a column.

$\endgroup$
  • $\begingroup$ I see. Thanks for the help, Michael! I have an explanatory variable in my data set where 99% of the values for the last purchase date for a customer is missing. The goal of the analysis is essentially to identify who the top customers are and how they vary from the others. Sure, the last purchase date intuitively seems like an important feature that could explain whether or not a customer is a "Top Customer," but with 99% of the data missing, what are my options? $\endgroup$ – Czar Yobero Aug 1 '16 at 17:53
  • $\begingroup$ I see. This comment indeed makes the question much more interesting! May I ask you to add it to your original post? I am sure you will get some high quality answer then. $\endgroup$ – Michael M Aug 1 '16 at 19:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.