I have a dataset (4,898 X 17,000) that follows 4898 mothers, fathers, and their children over a period of 15 years. The interviews have been conducted at baseline (when the child was born), year-1, year-3, year-5, year-9, and year-15. I want to predict GPA (at year-15) using random forest from a set of features (Gender, household income, household education, Parental involvement in studies, parental expectation, family structure, cognitive and non-cognitive variables. All of these variables except cognitive variables have been captured at year-15. Cognitive variables have been captured at year-9. Year-9 has about 1300 values missing whereas year-15 has about 1454 values missing due to non-response in year-9 and 15. I am quite new to imputation and I am not sure how to use multiple imputations here (especially when there are more than 1000 rows that have missing values for all the columns). Any help in this regard would be very helpful.

  • 9
    $\begingroup$ Just to get a grip on the situation, consider an example in which the survey asked about each person's health and the non-responses are due to people suddenly dying. Deleting the missing data would give a very misleading picture of the situation, wouldn't it? This is why one of the first things we would like to know is why are the data missing? What have you done to learn about the reasons for non-response? $\endgroup$
    – whuber
    Oct 21 at 19:53
  • $\begingroup$ To elaborate on the problem, I want to predict GPA from a set of predictors like "IQ score", "personality" "parental involvement", "parental expectation", "parental income" and "parental education" using random forest. I also have to create a composite for parental involvement for instance. In this case, what would be the best way to treat my missing values. The authors suggest using multiple imputations but how can I use the same if some of the rows have missing values for all the columns? I could maybe use it in the original dataset which is a 4898X17000 variables. $\endgroup$ Oct 21 at 20:12
  • 1
    $\begingroup$ Please say more about the overall structure of the data over the 15 years and the particular "wave" that is missing. In the question you talk of 10 variables for each of the 4898 rows (individuals?), but in a comment you say you have 17000 total "variables." What do those 17000 represent? That's a lot more than 10 variables recorded once a year for 15 years. Please provide this information (and that in your prior comment) by editing your original question, as comments are easy to overlook, are shown in small type, and can be deleted. $\endgroup$
    – EdM
    Oct 21 at 21:10
  • $\begingroup$ I made the change. I hope this is clearer than before. $\endgroup$ Oct 21 at 22:53

If you decide to "delete" the missing data prior to analysis, that is called a "complete-case analysis" (i.e., you are only using data points that have complete information). That is quite a simple and common method of analysis, but it has some risks. In particular, if the variables under analysis are statistically related to the "missingness" then ignoring the missing data will induce bias in your inferences.

Imputation methods are created in order to try to approximately model statistical dependence between missing values and the "missingness" in the data. In cases where entire classes of data points are missing, it may be the case that there is no information available to support imputation, in whih case you may have to fall back on complete-case analysis, with appropriate caveats and caution in your conclusions. In any case missing data methods require quite a bit of learning to implement correctly, but imputation methods can perform better than complete-case analysis in a wide variety of problems where there is sufficient information to estimate relationships between missing data values and the "missingness" indicators.

If you would like to learn more about missing data methods, you can find a simple educational introduction in Pigott (2001) and a more detailed exposition in Little and Rubin (2002).


It is important to think about the mechanism leading to missing data. There are three kind of missing data that can happen:

  1. Missing completely at random (MCAR). It means that the probability that an entry is missing is the fixed, independent of its (unobserved) value and independent of other variables. In that case, deleting incomplete data is OK and will not bias your result. However, doing multiple imputation may be more efficient, since you don't need to delete any valuable data. It will also depend on how much of your dataset is missing (maybe you will lose too much data doing complete case analysis, or maybe there's so few missing data that it's not worth the effort of imputation).
  2. Missing at random (MAR). It means that the probability of an entry being missing depends on the other variables, but not on the unobserved value. In this case, ignoring missing data may bias your results, and multiple imputation is recommended.
  3. Missing not at random (MNAR). In this case, the probability of missingness does depend on the unobserved value. An extreme example of this would be censoring. In this situation, neither imputation nor complete case analysis will remove bias, and there is no general solution here.

If you are sure that you are in a MCAR scenario (unlikely) or that the fraction of missing data is tiny, you can do complete case analysis. Otherwise, you should try imputation. If you are in a MNAR situation, you may have to rethink if your dataset can answer the questions you ask in an unbiased way.

I think multiple imputation may still work for completely missing rows (at least Bayesian model-based imputation would work, I am not sure about other methods), but these rows are not informative at all, so I think it's safe to delete those anyway.

  • $\begingroup$ Thanks. My question is that how do I check if the data is missing at random or not? For instance, I have the original dataset (4898X17000). I subset my dataset with the required 8 variables. Is there any test that I can do to know if my data falls into one of these categories? If yes, then what kind of test? Should I do that test before or after subsetting? (does not make sense to do it after subsetting since I have almost the same missing data for all the rows). It is recommended to multiply impute after shortlisting my variables (Src: fragilefamilieschallenge.org/missing-data). $\endgroup$ Oct 22 at 4:15

The structure of your dataset may lend itself to making this more difficult, if I read your question right. Staying out of the higher statistics (covered in the other answers) and just in the basic survey/study design realm, you might have three kinds of respondents:

  1. Non Responders - people who did not respond at any point in the survey
  2. Partial responders - people who responded to some of the years, but not all
  3. Completes - people who responded to each year in the survey

Each group has some data, hopefully, available about them, even the non responders. Sometimes, in surveys, you have data from the sample frame, data used to draw the sample, which often includes some demographic information such that you can draw a balanced sample. That information is available whether they respond or not. This is only true if the sample was drawn from a known population - not if it was drawn by, say, random digit dialing. In that case, you may have no information about non responders, but you also would probably not have them in the data file.

The partial responders, people who responded to the initial round of the survey but then later on left (or missed a year but came back later), will have much more information available of course.

Either way, you need a dataset that is respondent level that has all of the starting demographic data you have about these respondents, whether they are nonresponders, partial responders, or completes. It sounds like your data is not organized this way - so, reorganize it! This doesn't have to be attached to the rows of data from each year - it can be a separate dataset.

Then, you use this baseline demographic information for whatever imputation you're doing, or for weighting. The responses to the demographic questions from the first year would be used to impute the later years' variables, including their demographic variables. You also could design a more complicated model that rolled from year to year - birth predicts 1, 1 predicts 3, 3 predicts 5, and so on. That would likely be better, but I don't know your data nor your skill level with designing models like this; I'd often err on the side of simpler, as it's more likely I get it right!

I'm not an expert in imputation, so I won't speak to the specific choices - but hopefully this gets you an idea of where to start, and then you can use one of these other great answers to solve your imputation/deleting/etc. problem.


It's not a good a idea at all deleting missing data using tree-base models (if you're deleting the rows with at least one missing data) it'd be better you just ignore missing data when you're splitting a node on a decision tree because this model will just ignore the missing values unlike ignoring the whole instance (where there may be so many non-missing values since you got to much columns )


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.