This is a fairly broad question, so I expect there may be follow-up questions.

I have a fairly large opinion survey data set (about 40k rows, 40 columns) with many NA's (a couple variables have 40% NA). It is also historical data, so it is an aggregate of more than 50 years of surveys. There are few questions that have been asked on every survey, so I'm looking into imputation to allow for further analysis. (With the understanding that I will lose accuracy)

There may also be a few reasons underlying the missingness, although I suspect it's not far from random. Some missing data is from survey questions that were not asked in some years, although there are few "issue" type questions that may have changed over time, so I'm not too concerned here (plus I am using "year" as a dependent variable). Another reason for missingness is that the person refused to answer the question -- there may be some underlying characteristics here, but I'm not convinced it's something I need to control for during the imputation, although if others disagree, I'd absolutely be open to your thoughts.

I'm wondering if anyone can suggest best practices for moving forward, such as what type of algorithm to use. I've used R so far, but I'd be willing to move to Python for this task if it was quicker/better for any reason. Much Appreciated.

  • 1
    $\begingroup$ Thanks for edit, could you also tall us what do you mean by "better"? Additional details about nature of your data and the missingness (can you assume that the values are missing at random etc.) would also help. $\endgroup$ – Tim Nov 24 '17 at 8:59
  • $\begingroup$ Although I do not consider this question a duplicate - in the sense your exact situation might be different and thus require specific imputation strategies - you might find my answer here: stats.stackexchange.com/questions/257672/… helpful. In short, Stef van Buuren's R mice package is a very flexible multiple imputation tool. To answer why and how to apply this package, more detail on your data is required. E.g. do the same people answer multiple questionnaires over the years (which would imply some multilevel structure)? $\endgroup$ – IWS Nov 24 '17 at 9:21
  • $\begingroup$ Thanks -- I've been using the mice package so far, although it's struggling with the size and missingness of this dataset. (I'm told R can only use one core at a time) There are two surveys for each year, with the same respondents during that year, however questions are different on both surveys, so there is no missingness among the same respondents. Otherwise, the respondents change from year to year. $\endgroup$ – Andrew Nov 24 '17 at 9:26
  • $\begingroup$ AFAIK R indeed is not the fastest piece of statistical software. That said, it does often get the job done properly. As to your situation, it does seem imputation might get tricky if additional information on the survey participants is scarce. For example, if you have stuff like age, sex, socio-economic status, place of residence, happiness, health status, etc. (stuff which is reasonably related to the surveys) of the respondents at the moment of completing the questionnaire, that might greatly improve imputation by functioning as a 'bridge' between the different survey questions. $\endgroup$ – IWS Nov 24 '17 at 9:52
  • $\begingroup$ I've run R on smaller datasets with similar missingness, only to see it not finish after 12-24 hours. I'm thinking of trying amazon web services though for more computing power. Also, I have demographic data (with fewer NA's than other vars). Are you suggested I model the missing variables on the demographic data, or just be more content in the knowledge that it will help my results out. $\endgroup$ – Andrew Nov 24 '17 at 10:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.