I am working on a basic problem that requires developing a logistic regression model (the output is True/False, whether a person gets cancer). I have used glm() in R and got the model with some predictors. Now, I would like to test the predictions graphically but have no idea where to start. Unlike linear regression, we can just plot the predictions together with true values and see if true values follow the prediction function.
Could you give me some advice?
Thanks,
The most obvious plot to look at is a calibration plot. You form bins of predicted probabilities for "yes" (e.g. 0 to <0.05, 0.05 to <0.1 etc. or based on percentiles of the predicted probabilities) and show the proportion of "yes" for that bin. It should - up to randomness, which you can visualize with confidence intervals - be the mean if the predicted probabilities in the bin. This does not guarantee you have a great model, but at least that you fitted the data somewhat. If your bins do not cover the whole 0-1 range this also tells you something (e.g. all probabilities around 0 to 0.2 suggests that the model has trouble pinpointing extremely high risk patients - either due to a model that could be improved, or because there is not enough information on the data to do so).
As a fit check this can make sense on your training data, but if you truly want to evaluate calibration, you would do that on data not used for training. You may also wish to look at the ROC curve or the precision recall curve.
If you have only one predictor x, you can use it on the x axis, on the y axis you can put the binary outcome and additionaly, you can plot the function with the predicted values. Here is such a plot:
Further, keep in mind that "one very common way of assessing the usefulness of a binary classifier is the ROC curve" (here). So you can also make a ROC curve plot for a logistic regression. For the example above the ROC curve looks like this:
If you are not familiar with roc curves you can see the link above or here.
The R Code
library(ggplot2)
set.seed(1) # make it reproducible
x1 = rnorm(1000) # some continuous variables
z = 1 + 5*x1 # linear combination with a bias
pr = 1/(1+exp(z)) # pass through an inv-logit function
y = rbinom(1000,1,pr) # bernoulli response variable
df = data.frame(y=y,x1=x1) # make a dataframe
lmodel <- glm( y~x1,data=df,family="binomial") # now feed it to glm:
df$pred <- predict(lmodel, type = 'response') # save predicted values
# plot it
ggplot(df, aes(x1, y)) +
geom_point() +
geom_line(aes(x = x1, y = pred), color = 'red', size = 0.3)
# ROC
# vector of tresholds
treshold <- c(-Inf, seq(min(x1), max(x1), 0.1), Inf)
# calculate sensitivity and specifiticity per treshold
results <- sapply(treshold, function(treshold_i){
# what cases are over the treshold?
test_results <- factor(x1 >= treshold_i, levels=c(TRUE, FALSE))
# create a table
table_results <- table(test_results, y)
# estimate sensitivity
sens <- table_results[1, 1]/ sum(table_results[ , 1])
# estimate specifiticity
spec <- table_results[2, 2]/ sum(table_results[ , 2])
# save both and the used treshold in a matrix
m <- matrix(c(treshold_i, sens, spec), ncol= 3)
# return matrix
return(m)
})
# organize the data
# flip matrix
results <- t(results)
# name columns
colnames(results) <- c("treshold", "sens", "spec")
# ROC curve (2nd plot)
plot(1 - results[ , "spec"], results[ , "sens"], type= "l", col= "red",
xlab= "1 - Specificity", ylab= "Sensitivity")
# diagonal line
abline(0, 1)
You can plot the predictions (probabilities, see predict(, type = "response") in R) versus the true values (0 or 1) in this case as well, although this plot may not be the most informative.
