I am looking for a reasonably scaling missing data imputation approach for big data (e.g. a well-scaling version of kNN - the standard versions we tried so far just ran out of memory) that fulfills the following criteria:

  • takes what we know about records into account (i.e. not just a naive median/majority class approach, e.g. if we impute exact age and we have a categorical variable like "school child", "working age" and "retired", then I'd want an imputation that tends to impute young ages for "school child" records with missing age)
  • scales to a dataset with 300 to 1000 predictors and about 5 million records
  • reasonable memory requirement (for example on the system I use, I can get $\leq 256$ GB)
  • bearable runtime (let's say < 1 hour on a reasonably decent recent Xeon CPU, or if parallelization up to 20 of them with Infiniband)
  • ideally automatically deals with a mixture of continuous, binary and categorical predictors
  • ideally implemented in R (or python)

I noticed this question, but the data in question there is much smaller and a lot of time has passed (giving me some hope that new options have become available).

  • 2
    $\begingroup$ If you have 5 million records and 1000 predictors, you're at 40GB for data storage if everything is double precision; you'll likely not be able to come anywhere close to your memory requirement. Also, in high dimensions, kNN (with Euclidean distance) isn't so useful, as everything becomes almost equally far apart from everything else. $\endgroup$ – jbowman Oct 1 '18 at 19:36
  • $\begingroup$ @jbowman, thanks. I will check what I can maximally get. The data did fit into memory, so I guess it is more (I was hunting for system documentation, but could not find it). $\endgroup$ – Björn Oct 2 '18 at 5:36
  • $\begingroup$ @jbowman On the particular system I can use, I can have up to 256 GB. I'll update the question. $\endgroup$ – Björn Oct 2 '18 at 8:03
  • $\begingroup$ Are you trying to do single imputation or multiple imputation with pooling? $\endgroup$ – Jeffrey Girard Oct 2 '18 at 22:43
  • $\begingroup$ No strong opinion either way. $\endgroup$ – Björn Oct 3 '18 at 4:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.