I am looking for a reasonably scaling missing data imputation approach for big data (e.g. a well-scaling version of kNN - the standard versions we tried so far just ran out of memory) that fulfills the following criteria:
- takes what we know about records into account (i.e. not just a naive median/majority class approach, e.g. if we impute exact age and we have a categorical variable like "school child", "working age" and "retired", then I'd want an imputation that tends to impute young ages for "school child" records with missing age)
- scales to a dataset with 300 to 1000 predictors and about 5 million records
- reasonable memory requirement (for example on the system I use, I can get $\leq 256$ GB)
- bearable runtime (let's say < 1 hour on a reasonably decent recent Xeon CPU, or if parallelization up to 20 of them with Infiniband)
- ideally automatically deals with a mixture of continuous, binary and categorical predictors
- ideally implemented in R (or python)
I noticed this question, but the data in question there is much smaller and a lot of time has passed (giving me some hope that new options have become available).