5
$\begingroup$

The observations in my dataset can be split in two classes. The observations in class 1 are for sure correctly labeled. The observations that has been designated to class 2 have a huge percentage of mislabeled data. I also know that there are as much mislabeled data in class 2 as there are observations in class 1. As an example I did a small simulation of a potential dataset, to simplify things I only simulated 1 variable and I just copied class 1 observations as the mislabeled class 2 observations. (Note that the true dataset is much larger and contains multiple variables but the problem remains the same and this simulation captures the essence.)

Class1 = rnorm(1000,0)
Class2 = c(rnorm(1000,10),Class1)

Class1 = data.frame(score = Class1,Class1 = TRUE)  
Class2 = data.frame(score = Class2,Class1 = FALSE)  
dat = rbind(Class1,Class2)

My goal is to decide which observations in class 2 are truly from class 2 and which should have been in class 2. Since I have multiple variables I decided to fit a logistic regression model on my data and use the probabilities as a metric to decide if the observations of class 2 are mislabeled or not.

fit = glm(Class1~.,data= dat, family = binomial)
hist(fit$fitted.values,xlim = c(0,1),xlab = "P(Class 1)",
     main='Histogram of probabilities of being in Class 1' )

Imgur

If you plot the probabilities in the histogram, you see a lot of observations around 0. These are probably the true class 2 observations. You can also see a lot of observations around .5. This makes sense. You know that there are as many observations in class 1 as there are mislabeled observations in class 2. So if you see an observation with a parameter value that comes from the 'class 1' distribution, you can have 50 percent probability that is has a class 1 label and 50 percent probability that is has an class 2 label (so mislabeled).

But I find the above distribution of probabilities shown in the histogram somehow peculiar. I would have expected that all values would be between 0 and 0.5. Instead I also see some values for example around .6. So if I see an observations with these variable values I'm 60 percent sure it belongs to class 1. How can this be? In my simulation example I just copied the same values for class 1 and the mislabeled class 2, so above observation should in theory not be possible.

How should I interpret this? Are these probabilities meaningful?

EDIT: In response to @whuber I also added the following plot of the data.

plot(dat); curve(1/(1 + exp(-coef(fit) %*% rbind(1, x))), add=TRUE)

Imgur

$\endgroup$
  • 2
    $\begingroup$ (1) Your code won't work because it refers to nonexistent variables like C1. (2) Plotting your data is always useful. Try, for instance, plot(dat); curve(1/(1 + exp(-coef(fit) %*% rbind(1, x))), add=TRUE). $\endgroup$ – whuber Oct 28 '14 at 14:28
  • 2
    $\begingroup$ The plot shows the model you fit to your data; all the results are a consequence. Your two histograms correspond to the ranges of levels of the curve in the two regions to the left ("class 2") and right ("class 1") of $5$. The curve must pass very close to the point $(0,1/2)$ because you split a large set of data averaging $0$ into the two classes. The curve has to flow continuously in a decreasing manner from left to right. These qualitative characteristics are easily predictable before ever fitting the model--you don't need any software to draw the curve. $\endgroup$ – whuber Oct 28 '14 at 16:16
  • 2
    $\begingroup$ I believe you may be reasoning with two different, contradictory models. Your argument assumes that variation of score around the mean for each class (0 and 10) has no influence at all on the log odds of class inclusion, whereas the model explicitly supposes the log odds are a linear function of the score. The model used in your argument is the model used to generate the data but it is not the model used to fit the data. $\endgroup$ – whuber Oct 28 '14 at 17:13
  • 2
    $\begingroup$ @whuber : woow! thanks for the eye opener! You are right. I tried to fit a more flexible model (with splines) and now I can see that hard "cut - off" at 0.5 probability I was looking for. Too bad I can't flag a comment as an answer. :) $\endgroup$ – statastic Oct 28 '14 at 17:41
  • 1
    $\begingroup$ @statastic: You can use Bayes' Theorem to calculate $\Pr(\mathrm{Class1} |X=x)$ & compare that to fitted models: dnorm(score)*(1/3) / (dnorm(score)*(1/3) + (dnorm(score,10)/2 + dnorm(score)/2)*(2/3)) I think $\endgroup$ – Scortchi - Reinstate Monica Oct 28 '14 at 17:49
6
$\begingroup$

You may be reasoning with two different, contradictory models. The argument (in comments to the question) assumes that variation of score around the mean for each class ($0$ and $10$) has no influence at all on the log odds of class inclusion, whereas the model explicitly supposes the log odds are a linear function of the score. The model used in that argument is the model used to generate the data but it is not the model used to fit the data.

There's nothing the matter with most of this approach: it actually can be a good idea to generate data in a way that differs from what the model expects, because that can help you assess the model's robustness to violation of assumptions. Just be sure to keep the distinction between the actual data generation process and the assumptions of the model fitting process clearly in mind, and all should be OK.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.