2
$\begingroup$

Consider a sparse dataframe with lots of NA and a large number of explanatory variables and a response variable. The response variable doesn't contain any NA.

I want to clean this dataframe by discarding some of the explanatory variables having most of their entries NAs.

Is there any study that suggests the minimum percentage of non-NA values that must be present in a variable to qualify it as a predictor? For example, if the total percentage of non-NA values in column X is less than 5% of total records, then drop column X.

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Percent of missing values is important, but what's more important on the whole is whether missing values occur at random. Suppose just 1% of values were missing, but they happened to be the highest on some variable. That could mean serious bias. So, on the whole there can be no reliable criterion that so many percent missing is tolerable: it depends on the pattern of missingness. $\endgroup$
    – Nick Cox
    Oct 3, 2013 at 22:13

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.