2
$\begingroup$

I have a binary(1/0) classification task. I am trying to find $p(y = 1 | X)$ where $X$ is the vector of input variables and $y$ is the binary output label.

Suppose that for some records the output labels ($y$) are missing.

Scenario 1: Labels are Missing Completely at Random (MCAR)

While estimating the model, is there a difference between discarding records without labels vs. using Expectation Maximization to estimate missing labels?

Scenario 2: Labels are Missing at Random (MAR) conditional on $X$

While estimating the model, is there a difference between discarding records without labels vs. using Expectation Maximization to estimate missing labels?

Scenario 3: Labels are Not Missing at Random (NMAR)

In this case there is clearly a difference between discarding records without labels vs. using EM. Which option is better? Is there an all time winner or does it depend on missingness mechanism?

I am asking the difference in terms of model bias, and estimation efficiency.

| cite | improve this question | |
$\endgroup$
  • $\begingroup$ Under both situations, discarding records without label approach will produce correct statistical analysis results. The quality of other approaches cannot exceed the discarding records without label approach. You can use simulation to compare the two approaches. $\endgroup$ – user158565 Jul 8 '19 at 16:16
  • $\begingroup$ @user158565 Thank you for your comment. One thing I also don't understand is that if labels are MAR conditional on X, and p(y=1|X=x) is what I would like to model and s(a = 1|X=x) is the record selection probability, does p = s have to hold or s can be any arbitrary probability function s(x)? $\endgroup$ – Cagdas Ozgenc Jul 8 '19 at 19:14
  • $\begingroup$ What does a=1 mean? $\endgroup$ – user158565 Jul 8 '19 at 19:25
  • $\begingroup$ @user158565 a=1 record selected, a = 0 record not selected $\endgroup$ – Cagdas Ozgenc Jul 8 '19 at 20:27
0
$\begingroup$

Suppose $Y$ is a test result with pass (1) or fail (0) from the student. You have 100 students. If all of them gave you $Y$, you can estimate $P(Y=1)$. If 20 of them did not give you $Y$, and most of them due to shame of fail. Then you cannot use rest 80 $Y$ to estimate $P(Y=1)$. But If the reason has no relation of $Y$, for example, family emergency, sport activity,..., i.e., the probability of not reporting $Y$ is the same for $Y=0$ and $Y=1$, then you can use the 80 $Y$ to estimate the $P(Y=1)$.

Then you want to estimate $P(Y=1)$ separately for boys and girls. Among 100 students, 50 are girls and 50 are boys. But among 80 valid $Y$s, 45 are girls and 35 girls. Obviously it is not MCAR. But if same as before that the reason has no relation with $Y$, you can still use 45 $Y$s from girls to estimate $P(Y=1|girl)$ and use 35 $Y$s from girls to estimate $P(Y=1|boy)$. $s(a = 1|X=x)$ can be anything between 0-1 given it is not a function of $Y$.

Basically, missing of response variable has no relation with the value of response variable, you can do the analysis by drop the records with missed response variable. If missing is related to $Y$, then after dropping the records with missed response variable, the rest is not random sample anymore, so no statistical method is available.

For other approaches, at least you can use simulation to compare them with "discarding records without labels". I believe that "discarding records without labels" is the best.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I think there are some typos with "girl", "boy". Basically if data is selected by thresholding for example s(a=1|X=x) >= T include record otherwise exclude record, won't it mess up the distribution p(y=1|X=x) in the observable samples? $\endgroup$ – Cagdas Ozgenc Jul 9 '19 at 5:14
  • $\begingroup$ No problem, given s(a=1|X=x) has no direct relation with y. $\endgroup$ – user158565 Jul 9 '19 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.