Can I delete missing data? I have a dataset (4,898 X 17,000) that follows 4898 mothers, fathers, and their children over a period of 15 years. The interviews have been conducted at baseline (when the child was born), year-1, year-3, year-5, year-9, and year-15. I want to predict GPA (at year-15) using random forest from a set of features (Gender, household income, household education, Parental involvement in studies, parental expectation, family structure, cognitive and non-cognitive variables. All of these variables except cognitive variables have been captured at year-15. Cognitive variables have been captured at year-9. Year-9 has about 1300 values missing whereas year-15 has about 1454 values missing due to non-response in year-9 and 15. I am quite new to imputation and I am not sure how to use multiple imputations here (especially when there are more than 1000 rows that have missing values for all the columns). Any help in this regard would be very helpful.
 A: If you decide to "delete" the missing data prior to analysis, that is called a "complete-case analysis" (i.e., you are only using data points that have complete information).  That is quite a simple and common method of analysis, but it has some risks.  In particular, if the variables under analysis are statistically related to the "missingness" then ignoring the missing data will induce bias in your inferences.
Imputation methods are created in order to try to approximately model statistical dependence between missing values and the "missingness" in the data.  In cases where entire classes of data points are missing, it may be the case that there is no information available to support imputation, in whih case you may have to fall back on complete-case analysis, with appropriate caveats and caution in your conclusions.  In any case missing data methods require quite a bit of learning to implement correctly, but imputation methods can perform better than complete-case analysis in a wide variety of problems where there is sufficient information to estimate relationships between missing data values and the "missingness" indicators.
If you would like to learn more about missing data methods, you can find a simple educational introduction in Pigott (2001) and a more detailed exposition in Little and Rubin (2002).
A: It is important to think about the mechanism leading to missing data. There are three kind of missing data that can happen:

*

*Missing completely at random (MCAR). It means that the probability that an entry is missing is the fixed, independent of its (unobserved) value and independent of other variables. In that case, deleting incomplete data is OK and will not bias your result. However, doing multiple imputation may be more efficient, since you don't need to delete any valuable data. It will also depend on how much of your dataset is missing (maybe you will lose too much data doing complete case analysis, or maybe there's so few missing data that it's not worth the effort of imputation).

*Missing at random (MAR). It means that the probability of an entry being missing depends on the other variables, but not on the unobserved value. In this case, ignoring missing data may bias your results, and multiple imputation is recommended.

*Missing not at random (MNAR). In this case, the probability of missingness does depend on the unobserved value. An extreme example of this would be censoring. In this situation, neither imputation nor complete case analysis will remove bias, and there is no general solution here.

If you are sure that you are in a MCAR scenario (unlikely) or that the fraction of missing data is tiny, you can do complete case analysis. Otherwise, you should try imputation. If you are in a MNAR situation, you may have to rethink if your dataset can answer the questions you ask in an unbiased way.
I think multiple imputation may still work for completely missing rows (at least Bayesian model-based imputation would work, I am not sure about other methods), but these rows are not informative at all, so I think it's safe to delete those anyway.
A: The structure of your dataset may lend itself to making this more difficult, if I read your question right.  Staying out of the higher statistics (covered in the other answers) and just in the basic survey/study design realm, you might have three kinds of respondents:

*

*Non Responders - people who did not respond at any point in the survey

*Partial responders - people who responded to some of the years, but not all

*Completes - people who responded to each year in the survey

Each group has some data, hopefully, available about them, even the non responders.  Sometimes, in surveys, you have data from the sample frame, data used to draw the sample, which often includes some demographic information such that you can draw a balanced sample.  That information is available whether they respond or not.  This is only true if the sample was drawn from a known population - not if it was drawn by, say, random digit dialing.  In that case, you may have no information about non responders, but you also would probably not have them in the data file.
The partial responders, people who responded to the initial round of the survey but then later on left (or missed a year but came back later), will have much more information available of course.
Either way, you need a dataset that is respondent level that has all of the starting demographic data you have about these respondents, whether they are nonresponders, partial responders, or completes.  It sounds like your data is not organized this way - so, reorganize it!  This doesn't have to be attached to the rows of data from each year - it can be a separate dataset.
Then, you use this baseline demographic information for whatever imputation you're doing, or for weighting.  The responses to the demographic questions from the first year would be used to impute the later years' variables, including their demographic variables.  You also could design a more complicated model that rolled from year to year - birth predicts 1, 1 predicts 3, 3 predicts 5, and so on.  That would likely be better, but I don't know your data nor your skill level with designing models like this; I'd often err on the side of simpler, as it's more likely I get it right!
I'm not an expert in imputation, so I won't speak to the specific choices - but hopefully this gets you an idea of where to start, and then you can use one of these other great answers to solve your imputation/deleting/etc. problem.
A: It's not a good a idea at all deleting missing data using tree-base models (if you're deleting the rows with at least one missing data) it'd be better you just ignore missing data when you're splitting a node on a decision tree because this model will just ignore the missing values unlike ignoring the whole instance (where there may be so many non-missing values since you got to much columns )
