i know how to find and fill the missing values. But i am not sure when to fill the values with min., max. , mean, median or mode. Can someone help me to understand on what basis i can decide , i have to remove the missing rows or columns and if i have to keep the missing data on what basis i can fill the missing values.
how i can fill the missing data in categorical feature?
This is a complicated question. I recommend the book by little and Rubin statistical analysis with missing data. https://books.google.com/books/about/Statistical_Analysis_with_Missing_Data.html
Basically, there are three categories of missing data. We assume that each data record can be divided into an "observable component" and an "unobservable component". We also assume that the data records are independent and identically distributed.
- MCAR (Missing Completely At Random) where the pattern of missinginess is statistically independent of the data record. Example: you have a data set on a piece of paper and you spill coffee on the paper destroying part of the data.
- MAR (Missing At Random) where the probability distribution of the pattern of missingness is functionally dependent upon the observable component in the record. MCAR is a special case of MAR. Example: you have a question on a survey asking if the survey participant is a drug addict and another question which asks if the survey participant has less than one alcoholic drink per year. Assume the answer to the alcoholic drink question is always observable, then the probability that someone fails to answer the drug addict question is most likely functionally dependent upon their answer to the alcoholic drink question.
- MNAR (Missing Not at Random) which is defined as the case which is NOT MAR. In the MNAR case, you can have situations where both the drug addict and alcoholic drink questions are absent in the same record. Another example, is a case where the question is: "What is your gender?" Suppose that females are less likely to answer this question than males. This is another example of an MNAR question because the probability that the answer is observable is conditionally dependent upon the unobservable component of the data record.
Ok...now that we have some terminology, we can discuss some strategies and recommendations...
- 1) Never insert mean, mode,mean, max, min, median, or anything else for missing values. That is avoid deterministic imputation even though it is widely used and available in most software packages. It underestimates and distorts the statistical regularities (e.g., underestimates variance is one example) present in your data sample. In some special cases such as linear regression it might have some limited value but in general I wouldn't mess with it.
- 2) if the data records are MCAR Then you can delete records with missing data.
- 3) if the data records are MCAR, then sometimes you can stochastically impute the missing values rather than deterministically impute them. So this means that if you specify the marginal probability distribution of a missing value as Gaussian with some known mean and some known variance then you can sample from that distribution to impute values into the data set. You need to be careful here and do some additional research beyond this answer before moving forward with this.
- 4) If the data is MAR then an algorithm such as Expectation Maximization can be used to handle the missing observations.
- 5) If the data is MNAR you can include binary indicators in the data record which explicitly identify when a variable is not observable. The challenge with this approach is that a highly nonlinear model needs to be designed to properly integrate this information in an appropriate manner. This might work in a machine learning algorithm where the binary indicators "disconnect" the influence of predictors which are not observable. Consequently, the MNAR theory (i.e., the theory of the joint distribution of the complete data record and missing data pattern) is instantiated in the learning machine's probabilistic model of its statistical environment.