Suppose I have a population of observations I want to model as being drawn from some distributional family, which I believe adequately represents the true distribution. My goal is to estimate the parameters of this distribution.

Suppose further that observations of my variable of interest are randomly missing, and the probability of missingness increases monotonically with the percentage by which the true value exceeds the value as estimated from the non-missing values, i.e. the error term in a semi-log or log-log regression of the variable of interest on the rest of the variables in the non-missing cases. In other words, for missing cases the value estimated by the regression is a lower bound of the true value. Missing observations appear as a visible blank – think of them as refusals to answer a survey questionnaire.

Finally, suppose that know that the probability that an observation will be missing is a known, increasing function of the (unobservable) error as described above, both as to form and parameters.

Can the visible observations from the population, together with the assumption of its distributional family, observations of missing values and the knowledge of the distribution of the probability of missingness, suffice to allow me to correctly estimate the parameters of the full distribution? If so, how? If not, why?

  • 1
    $\begingroup$ Pay attention to the difference between "censoring" and "missing" in statistics. In fact, you are talking about missing."the value of the draw" =? $\endgroup$ – user158565 Oct 22 '18 at 1:25
  • $\begingroup$ Thanks, @a_statistician, that's very helpful! I don't think that distinction was clear to me. I have at least six kinds of absent data in my data set: total non-response, item non-response, both top- and bottom-coding, values imputed by a hot-deck procedure I believe to be biased, categorical data that is collapsed for some years, and questions that were only asked to part of the samples. That's the problem with the real world: messy. Sigh. Anyway, this question is aimed at replacing hot-deck imputation with some other approach $\endgroup$ – andrewH Oct 25 '18 at 21:34
  • $\begingroup$ You can think of your missing data as latent variables and use a latent variables model to get the prediction of their possible values based in the data. $\endgroup$ – deasmhumnha Oct 26 '18 at 2:46
  • $\begingroup$ You introduced a new tag mnar. Can you please write a tag wiki? $\endgroup$ – kjetil b halvorsen Oct 26 '18 at 21:49

When you have missing data you need to (carefully) consider what is the missing data mechanism, i.e., how the probability of missingness depends on the observed an unobserved data. You wrote that the probability of missingingness depends on the true value of your variable of interest. This seems to be a missing not at random mechanism. To obtain unbiased parameter estimates in this case you will need to work with the joint distribution of the outcome and missingness processes. Based on the above description, the selection models seem to be appropriate, i.e., $$p(Y^o, M; \theta, \eta) = \int p(Y^o, Y^m; \theta) \, p(M \mid Y^o, Y^m; \eta) \, dY^m,$$ where $Y^o$ and $Y^m$ denote the observed and missing part of your variable of interest, $M$ is the missing data indicator, the first term in the integrand is your distribution of interest, parameterized by the vector $\theta$, and the second term specifies how the probability of missingness depends on $Y^o$ and $Y^m$ and is parameterized by $\eta$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.