0
$\begingroup$

I am working on a data set which has a 84 variables recorded for a number of individuals. True to "real" data sets, there are many missing variables, NaN values and previous values carried forward (when looking at longitudinal data).

I am trying to find a way to maximise the number of variables present for the largest population, with no values carried forward. That is, how big can I make my population without loosing too many variables.

I have thought about using Expectation Maximisation, although I am not certain at this point how to apply it to this problem. For a lower dimensional problem it could have been possible to create a Venn diagram but in this higher dimensional space, its not really feasible.

I wondered how people have tackled this problem in the past and how you went about solving it. I will start (i think) my making a binary table indicating where values are present/not/carried forward.

Thanks in advance!

$\endgroup$
2
  • $\begingroup$ Would it not be better to impute them rather than selecting both variables and cases? $\endgroup$ – mdewey Apr 27 '17 at 9:32
  • $\begingroup$ possibly. Im not sure how comfortable I am with imputation when I will building machine learning models on these values afterwards - as these values are being build on the rest of the data set doesnt this "taint" the validity? I know its common practice i just dont feel its particularly natural $\endgroup$ – JB1 Apr 27 '17 at 9:34
0
$\begingroup$

For those interested, the problem was perhaps simpler than first thought must have been a slip of the mind.

A simple clustered heatmap solved the problem with variables on one axis and individuals on the other. :)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.