As part of a school project I have to evaluate the impact of pre-school on the performance of children in academic tests (cognitive skills) as they enter kindergarten. I'm given a dataset with entries of 17'000 children, however, there are some problems with the data given: in 1696 cases, I do not have any data on their performance in these tests, in some more cases the gender, mother's education level, the father's education level etc. are not reported. All in all I do have almost 9000 children left with a complete set of entries. What is the best way to deal with this data? Just deleting these variables seems to violate the assumption of "random sampling" as there could be self-selection bias. Should I use multiple imputation?
The best thing to do with missing data is to use all of the information that is available to you.
Deleting the incomplete cases, also known as list-wise deletion, is known to cause bias for all types of missingness except missing completely at random data (MCAR)(Arbuckle, 1996; Brown, 1994; Wothke, 2000). MCAR data is rare in psychological and educational data.
The best thing you could do is use a full information maximum likelihood technique (FIML) to analyze your data. I am not sure what type of analysis you plan to do, but you should look into FIML variations of that analysis. For a few good papers on using various methods to deal with missing data see Enders, 2001, Howell, 2007, or Schafer & Graham, 2002.
If there is not a FIML variation of your analysis, multiple imputation might be the next best thing. What multiple imputation actually does is replace each missing value with a set of plausible values that represent the uncertainty about the right value to impute and then combines all of the possible imputation sets. Many researchers prefer FIML to multiple imputation because multiple imputation is actually making up numbers for the missing data, and they think that is much more likely to give incorrect estimates than just using the data you have as you do in FIML approaches.
Hope this helps!