Skip to main content
Cross Validated

Return to Question

Responding to advice from a_statistician, reframed problem as missing rather than censored data.
Source Link
andrewH
andrewH
  • 3.2k
  • 1
  • 20
  • 29

Can I correct for probabilistic censorship of especially high valuesrandomly missing data where missingness is has a known relationship to the error term?

Suppose I have a population of observations I want to model as being drawn from some distributional family, which I believe adequately represents the true distribution. My goal is to estimate the parameters of this distribution.

Suppose further that these observations of my variable of interest are randomly censoredmissing, and the probability of censorshipmissingness increases monotonically with the percentage by which the true value exceeds the value as estimated from the non-missing values, i.e. the error term in a semi-log or log-log regression of the drawvariable of interest on the rest of the variables in the non-missing cases. CensoredIn other words, for missing cases the value estimated by the regression is a lower bound of the true value. Missing observations appear as a visible blank – think of them as refusals to answer a survey questionnaire.

Finally, suppose that I have some ancillary data sourceknow that lets me estimate the parameters and distributional form of the probability of censorshipthat an observation will be missing is a known, but does not provide true values forincreasing function of the censored observations(unobservable) error as described above, both as to form and parameters.

Can the uncensoredvisible observations from the population, together with the assumption of its distributional family, observations of censoredmissing values and the knowledge of the distribution of censorshipthe probability of missingness, suffice to allow me to correctly estimate the parameters of the partially censoredfull distribution? If so, how?

Note: Because the true functional form of the distribution is If not known with certainty, I have a mild preference for GMM or GEE estimators over ML estimators. But I'll take whatever I can get.why?

Can I correct for probabilistic censorship of especially high values?

Suppose I have a population of observations I want to model as being drawn from some distributional family, which I believe adequately represents the true distribution. My goal is to estimate the parameters of this distribution.

Suppose further that these observations are randomly censored, and the probability of censorship increases monotonically with the value of the draw. Censored observations appear as a visible blank – think of them as refusals to answer a survey questionnaire.

Finally, suppose that I have some ancillary data source that lets me estimate the parameters and distributional form of the probability of censorship, but does not provide true values for the censored observations.

Can the uncensored observations from the population, together with the observations of censored values and the knowledge of the distribution of censorship, suffice to allow me to correctly estimate the parameters of the partially censored distribution? If so, how?

Note: Because the true functional form of the distribution is not known with certainty, I have a mild preference for GMM or GEE estimators over ML estimators. But I'll take whatever I can get.

Can I correct for randomly missing data where missingness is has a known relationship to the error term?

Suppose I have a population of observations I want to model as being drawn from some distributional family, which I believe adequately represents the true distribution. My goal is to estimate the parameters of this distribution.

Suppose further that observations of my variable of interest are randomly missing, and the probability of missingness increases monotonically with the percentage by which the true value exceeds the value as estimated from the non-missing values, i.e. the error term in a semi-log or log-log regression of the variable of interest on the rest of the variables in the non-missing cases. In other words, for missing cases the value estimated by the regression is a lower bound of the true value. Missing observations appear as a visible blank – think of them as refusals to answer a survey questionnaire.

Finally, suppose that know that the probability that an observation will be missing is a known, increasing function of the (unobservable) error as described above, both as to form and parameters.

Can the visible observations from the population, together with the assumption of its distributional family, observations of missing values and the knowledge of the distribution of the probability of missingness, suffice to allow me to correctly estimate the parameters of the full distribution? If so, how? If not, why?

Source Link
andrewH
andrewH
  • 3.2k
  • 1
  • 20
  • 29

Can I correct for probabilistic censorship of especially high values?

Suppose I have a population of observations I want to model as being drawn from some distributional family, which I believe adequately represents the true distribution. My goal is to estimate the parameters of this distribution.

Suppose further that these observations are randomly censored, and the probability of censorship increases monotonically with the value of the draw. Censored observations appear as a visible blank – think of them as refusals to answer a survey questionnaire.

Finally, suppose that I have some ancillary data source that lets me estimate the parameters and distributional form of the probability of censorship, but does not provide true values for the censored observations.

Can the uncensored observations from the population, together with the observations of censored values and the knowledge of the distribution of censorship, suffice to allow me to correctly estimate the parameters of the partially censored distribution? If so, how?

Note: Because the true functional form of the distribution is not known with certainty, I have a mild preference for GMM or GEE estimators over ML estimators. But I'll take whatever I can get.