There are a lot of threads on here about missing data, but I haven't found something that really gets at the best practices, and discussion of why to choose one approach over another. This is such a common topic in any introduction to machine learning that I've seen, but there doesn't appear to be clear consensus on what to do. Is there a robust approach works pretty well in practice? Can we automate how we deal with missing data? Is there a reason there isn't?
I'll list a few common answers to this question I see, and a couple more robust, less common approaches.
There are some common recommendations including:
- getting rid of rows with missing data;
- getting rid of columns (features) with missing data;
- filling in missing values with mean, median, or mode;
- use some nearest neighbors imputation;
These seem like ways to quickly get a model out the door, but don't seem realistic in practice. You will have a lot of messy data in the real world. Lots of missing data for lots of missing features. And a lot of times missing data can be a great predictor for things. Moreover, there's a strong emotional repulsion I have to disregarding data, or changing it. Every fiber of my being says this is a terrible idea. Here are some more common procedures: https://www.ncbi.nlm.nih.gov/pubmed/23853744
I went through a data science bootcamp and our instructors recommended a more thorough approach if data is really messy which I like quite a bit more:
- Create a new feature that is an indicator function for each feature being populated or not.
- Then impute the missing values to some statistic like the mean, so that our ML algorithm will not have issues with missing data
This makes a lot of sense to me, but there's the disadvantage of doubling your feature space. How bad is this in practice? Are there other disadvantages?
I'm also taking an ML classification class mooc and it recommends a different approach in the context of decision trees. At each split, have the algorithm lump missing values with another group. So if we're on a feature with categories A,B,C...we can split this feature a few different ways:
- A or missing,B, C
- A, B or missing, C
- A, B ,C or missing
This approach seems similar to the approach above, but has some practical drawbacks in rewriting every algorithm we use. Does this approach provide any benefits over the above? Are there other robust methods that compete with these approaches?