0
$\begingroup$

I am asked to read up on how to deal with missing values. From what I read I can use imputation with a package like MICE (for R) to automate this process.

However I also read that when I am missing over 5% of data for a certain feature I should dismiss it and remove it from my ABT (Analytical Base Table), which will be used for Machine Learning.

However if the above is true then how should I deal with the following situation: I got 2 tables of which one will form my ABT. The other one is used to engineer new features.

The one used to create new features missed over 5% of the data for a column. The problem is that this column is pretty important. Without it the other columns have no use any longer. Should I ignore this table when engineering new features or should I just remove the rows missing a value for this feature?

$\endgroup$

1 Answer 1

1
$\begingroup$

How correlated is the missing data with the other variables? Might be possible to use the 95% of data you do have to build a predictive model for the other 5%. Really depends on how your variables are correlated. But just going with 95% of the data isn't a terrible approach, as long as the missingness is non-informative. Worth checking there isn't a difference between the 95% and the 5% in the other columns, there might be some inherent difference causing the missingness

$\endgroup$
5
  • 1
    $\begingroup$ It's not correlated at all with other data in that table. It describes if students have finished a course. The entire table describes a course a student is in and what the end grade for that course is. The data set missed approx. 20% of the resulting grades. I would like to use the set for engineering new features but is that acceptable to do after removing rows that are missing the final grade? $\endgroup$ Commented Apr 29, 2020 at 8:48
  • $\begingroup$ It's definitely worth reporting the correlation, even if it's not significant. e.g. is the rate of missingness the same among maths students and physics students? Is it the same among high grades and low grades? Do some courses naturally have higher ending grades than others? And if the censoring happens to be informative, e.g. nursing courses more likely than maths courses to be missing because of some admin error --> female students more likely to have missing data, then there might be some skew in the 95% which you'd need to report in a limitation (but isn't a game-breaker) $\endgroup$
    – E. Rei
    Commented Apr 29, 2020 at 8:58
  • $\begingroup$ And if you engineer new features after removing rows, consider how the features might / might not be different in the rows with missing data, if there is evidence in the dataset that missingness is not completely at random (it might not be correlated with variables you have, but with things you don't have) $\endgroup$
    – E. Rei
    Commented Apr 29, 2020 at 9:00
  • $\begingroup$ So I should figure out at first where the missing values are. See if the occure more often in certain courses etc.? $\endgroup$ Commented Apr 29, 2020 at 9:37
  • $\begingroup$ It's definitely relevant, even if you just include it as a supplementary table or something and not in the main report. I'd recommend taking a look at this: theanalysisfactor.com/missing-data-mechanism $\endgroup$
    – E. Rei
    Commented Apr 29, 2020 at 9:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.