7
$\begingroup$

I am using logistic regression to solve the classification problem.

g = glm(target ~ ., data=trainData, family = binomial("logit"))

There are two classes (target): 0 and 1

When I run the prediction function, it returns probabilities.

p = predict(g, testData, type = "response")

However, it is not clear to me how to understand which class has been assigned?

Real  p 

1   0.17568578
1   0.41698474
1   0.19151927
1   0.25587242
1   0.25604452
0   0.39976069
0   0.39910282
0   0.16879320

I appreciate if someone can explain me how this works based on the above example. Thanks

$\endgroup$

migrated from stackoverflow.com Jan 13 '15 at 23:08

This question came from our site for professional and enthusiast programmers.

  • 2
    $\begingroup$ These are the predicted probabilities of each observation to be 1 (i.e. success). $\endgroup$ – David Arenburg Jan 13 '15 at 22:26
  • 3
    $\begingroup$ You are the one who gets to choose the threshold for a decision based on the costs of mis-classification in either direction. $\endgroup$ – DWin Jan 13 '15 at 22:32
  • 1
    $\begingroup$ @ David Arenburg: Ok, but what is the threshold, 0.5? E.g. what does 0.41 mean if the real class is 1? Does it mean that the entry has been classified as the class 0? $\endgroup$ – Klausos Klausos Jan 13 '15 at 22:34
  • 1
    $\begingroup$ 0.41 should be the probability the observation goes to class 1, so, if you choose 0.5 threshold, you finally assign it to class 0. $\endgroup$ – Davide Passaretti Jan 13 '15 at 22:50
  • 2
    $\begingroup$ There is no standard threshold but as you have the target in your data, try threshold near to percentage of 1 in target and check where you get good classification. Use that threshold. Hope it will work. $\endgroup$ – Sangram Jan 14 '15 at 5:49
8
$\begingroup$

The predicted values only tell you how likely it is that an observation belongs to the class coded as 1 given its explanatory variables. For classification, you need to find a threshold $t$ which in some sense is optimal for your problem. This is e.g. affected by monetary costs or ethical boundaries.

If you don't have any of these costs or boundaries, i.e. is a cost function, one criterion could be to minimize the sum of the error frequencies. For this the following two terms are important:

Sensitivity denotes the fraction of positives that were correctly specified for a given $t$.

Specificity denotes the fraction of negatives that were correctly specified for a given $t$.

Denote $s_0$ as Senstitvity and $s_1$ as Specificty, minimizing the sum of the error frequencies is equivalent to finding maximum $s_0(t) + s_1(t)$ for all thresholds $t$.

Here, I recommend to use the pROC package in R. It provides a very useful function called roc. See the sample code below. Here, response is your vector of ones and zeros and predictor your predictions. Moreover, the code produces also the corresponding ROC curve and adds a vertical line where the optimal threshold was found.

Please note: You provided very little data so I simulated some myself to get more different Sensitivities and Specificities. You can find the code for the simulated data below the picture.

rm(list = ls()) # clear work space

#install and load package
install.packages("pROC")
library(pROC)

#apply roc function
analysis <- roc(response=p$Real, predictor=p$p)

#Find t that minimizes error
e <- cbind(analysis$thresholds,analysis$sensitivities+analysis$specificities)
opt_t <- subset(e,e[,2]==max(e[,2]))[,1]

#Plot ROC Curve
plot(1-analysis$specificities,analysis$sensitivities,type="l",
ylab="Sensitiviy",xlab="1-Specificity",col="black",lwd=2,
main = "ROC Curve for Simulated Data")
abline(a=0,b=1)
abline(v = opt_t) #add optimal t to ROC curve
opt_t #print t

                                           enter image description here

##Simulate Data
set.seed(123456)
n <- 10000
q <- 0.8

#Simulate predictions
Real <- c(sample(c(0,1), n/2, replace = TRUE, prob = c(1-q,q)),
        sample(c(0,1), n/2, replace = TRUE, prob = c(0.7,0.3)))

#Simulate Response
p <- c(rep(seq(0.4,0.9, length=100), 50),
    rep(seq(0.2,0.6, length=100), 50))
p <- data.frame(cbind(Real, p))
$\endgroup$
  • $\begingroup$ So in your example, opt_t (which seems to be identical to Youden's Index) = 0.5964646. Does that mean that Klausos should classify any observation with a predicted probability > 0.5964646 as "Class 1" and any observation with a predicted probability < 0.5964646 as "Class 0"? $\endgroup$ – coip Feb 14 '18 at 16:52
  • $\begingroup$ Related to that, the vertical abline at 0.5964646 on the ROC Curve offers a nice visual of the threshold, but I'm curious why it is on the x-axis instead of the y-axis, and how to interpret it in relation to the y-axis (as it seems to intersect the y-axis at the same value as the specificity value produced by coords(analysis, "best", best.method = "youden") in the pROC package, but in the plot produced from your code the y-axis is labeled as the "sensitivity". $\endgroup$ – coip Feb 14 '18 at 17:01
4
$\begingroup$

Put simply/blunty a logistic regression model is not a classifier. It is a model for the probability parameter of the binomial distribution. This is why predict() gives probabilities.

In order to make it a classifier you need to specify a function converts probabilities into classes. Choosing a cutoff is one way - although probably not ideal for a large number of predictions. Taking your example, and the "naive" cutoff of 0.5 leads to all observations being classified as $0$. This is despite the model expectation of the number of $1$ classes to be about $2.5$

A better approach could be taking a randomised decision - generate a bernoulli random variable with the probability, and set that as your classifier.

$\endgroup$
  • 1
    $\begingroup$ When/why would the randomization procedure described at the last paragraph be better than using a threshold? $\endgroup$ – Juho Kokkala Feb 23 '15 at 15:02
  • $\begingroup$ A randomised decision is more likely to give predictions consistent with "global" model expectations. In the example given, a randomised decision will give about $2$ or $3$ predictions of $1$. Perhaps a more general idea is that when doing a large number of predictions, applying the optimal prediction for each individual observation, does not necessarily lead to an optimal prediction for all observations taken together. A very basic example is a coin with $P(H)=0.51$. If you are prediction the next toss, then $H$ is best. When predicting the next $1,000$ might not want to predict all $H$ $\endgroup$ – probabilityislogic Feb 25 '15 at 6:07
0
$\begingroup$

For finding which level is 0 and which level is 1, use levels() function on the response column.

You can change the order of levels by using the factor() function:

response <- factor(response,levels=c(level2,level1))
$\endgroup$
  • $\begingroup$ That's not the question Klausos was asking. $\endgroup$ – coip Feb 14 '18 at 17:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.