Measuring the Performance of Logistic Regression: Regression vs Classification

Question

I have noticed that Logistic Regression (https://en.wikipedia.org/wiki/Logistic_regression) is a model that used significantly for both Regression problems and Classification problems.

When used for Regression, the main purpose of Logistic Regression appears to be to estimate the effect of a predictor variable on the response variable. For example, here are some examples in which Logistic Regression is used for Regression problems:

• Modelling of binary logistic regression for obesity among secondary students in a rural area of Kedah : https://aip.scitation.org/doi/pdf/10.1063/1.4887702
• A logit model for the estimation of the educational level influence on unemployment in Romania : https://mpra.ub.uni-muenchen.de/81719/1/MPRA_paper_81719.pdf
• A logistic regression investigation of the relationship between the Learning Assistant model and failure rates in introductory STEM courses : https://stemeducationjournal.springeropen.com/articles/10.1186/s40594-018-0152-1

When used for Classification, the main purpose of Logistic Regression appears to be to estimate the probability of the response variable assuming a certain value given an observed set of predictor variables. For example, here are some examples in which Logistic Regression is used for Classification problems:

• Using logistic regression to develop a diagnostic model for COVID‑19: A single‑center study : https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9277749/pdf/JEHP-11-153.pdf
• Logistic regression technique for prediction of cardiovascular disease : https://www.sciencedirect.com/science/article/pii/S2666285X22000449
• A Study of Logistic Regression for Fatigue Classification Based on Data of Tongue and Pulse : https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8917949/pdf/ECAM2022-2454678.pdf

Based on surveying such articles, I noticed the following patterns:

• When Logistic Regression is being used for Regression problems, the performance of the Regression Model seems to be primarily measured using metrics that correspond to the overall "Goodness of Fit" and "Likelihood" of the model (e.g. in the Regression Articles, the Confusion Matrix is rarely reported in such cases)
• When Logistic Regression is being used for Classification problems, the performance of the Regression Model seems to be primarily using metrics that correspond to the ability of the model to accurately classify individual subjects such as "AUC/ROC", "Confusion Matrix" and "F-Score".

The interesting thing being that regardless of whether you working on a Regression problem or a Classification problem - if you do decide to use Logistic Regression, in both cases you can calculate Classification metrics such as the Confusion Matrix. Based on these observations, I have the following question:

My Question: Suppose if I am using Logistic Regression in a regression problem (e.g. estimating the effect of predictors such as age on employment vs unemployment) and the model seems to be performing well (e.g. statistically significant model coefficients, statistically significant overall model fit, etc.). Even though I technically still able to calculate Classification metrics such as the Confusion Matrix, F-Score and AUC/ROC - am I still obliged to measure the ability of this Regression model to successfully classify individual observations based on metrics such as ROC/AUC? Or am I not obliged to this since I not working on a Classification problem?

I feel that it might be possible to encounter a situation/dataset in which the goal was to build a Logistic Regression model for a Regression problem - and the resulting model might have good performance metrics used in regression problems, but might have poor ROC/AUC values. In such a case, is this a good Logistic Regression model as it performs well for the regression problem as intended - or is it a questionable model as it is unable to perform classification at a satisfactory level?

Answer 1

Some of the trouble with this question is that many of the "metrics typically associated to measure the performance of predictive modelling" are applied inappropriately, such as assessing the classification accuracy with no regard for how those classifications are created and if a software-default threshold (usually $1/2$) is appropriate.

That said, I have seen plenty of papers (published in the elite journals of their respective fields) where the ultimate goal is causal inference on the coefficients, yet the logistic regression models have standard metrics calculated like McFadden's $R^2$ or classification accuracy. For instance, Sundaram & Yermack (2007) report classification accuracy in their table 6 despite the main purpose of running those logistic regressions being the coefficient inference. (My take is that they made a mistake in doing so, because one of their models reports classification accuracy worse than would be achieved by predicting the majority class every time.) On the other hand, I recently saw another paper that had $R^2_{adj}<0$ all over the place. That their regression models were, arguably, doing worse than doing no modeling at all, helped me form a rationale for being skeptical of their results (there were other issues with the statistics). Thus, to explictly address your question, it can be helpful to give some sense of model performance, no matter how modest, even if prediction is not the main goal of the work. While a reviewer/professor/customer might not require it and oblige you to report it, I still believe it to be valuable information.

In such a case, it is typically acceptable to editors and referees to have rather modest performance in terms of the performance metrics. For instance, I have seen papers in top journals with adjusted $R^2$ around $0.05$, maybe even lower. Even that Sundaram & Yermack paper has rather pedestrian performance when accuracy scores are compared to naïvely predicting the majority category every time. (Most of their $R^2$-style scores, defined as I do in that link, are greater than zero (one is less than zero), but they do not scream out, "This model gets an $\text{A}$," the way that a classification accuracy of $97\%$ might.)

REFERENCE

Sundaram, Rangarajan K., and David L. Yermack. "Pay me later: Inside debt and its role in managerial compensation." The Journal of Finance 62.4 (2007): 1551-1588.