0
$\begingroup$

I'm trying to impute missing values for a binary variable (values 0 and 1) with some challenging data (of about 1 million observations).

The data can be divided into two groups:

  • in group 1, we know that all individuals take 1 as a value. This amounts to about 1% of the data.

  • in group 2, we don't know any of the values. We only know that the proportion of 1s is small (about as many as in group 1, i.e. 10,000 observations).

The fact that group 1 has only 1s and no 0s makes it impossible to train a model just on group 1, to then use it and get predictions for group 2. Still my thinking is that observations taking 1 as a value in group 2 should be closer to group 1 in their characteristics (i.e. other variables available in the data) than observations taking 0 as a value in group 2.

So how can I use the information given by group 1, to try and find the 1s in group 2 ? If someone has an idea, I'd be happy to hear their thoughts.

Thank you very much!

$\endgroup$

1 Answer 1

1
$\begingroup$

You could consider formulating a Bayesian model (e.g. a Bayesian logistic regression - if you somehow know subtree marginal distributions, you could use a suitable likelihood for that). You can then encapsulate you beliefs that you appear to have into priors for the model and fit the model to the data with observed values. This would allow you to (multiple) impute the missing values based on the posterior predictive distribution.

$\endgroup$
1
  • $\begingroup$ Thanks a lot for your answer! I know a bit about Bayesian models (prior and posterior distribution etc) but I haven't really used them before. Can I ask what the subtree marginal distributions correspond to in this case? $\endgroup$
    – user216423
    Aug 9, 2018 at 11:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.