Area Under Curve ROC penalizes somehow models with too many explanatory variables?

Question

I'm using Area Under Curve ROC as a performance measure of my classification algorithms (logistic regressions). Since I'm going to choose the model that maximize the Area Under Curve ROC, I would like to know if AUC penalizes somehow models with too many regressors (for example, like BIC information criterion).

Answer 1

You mention in the comments that you are computing the AUC using a 75-25 train-test split, and you are puzzled why AUC is maximized when training your model on only 8 of your 30 regressors. From this you have gotten the impression that AUC is somehow penalizing complexity in your model.

In reality there is something penalizing complexity in your model, but it is not the AUC metric. It is the train-test split. Train-test splitting is what makes it possible to use pretty much any metric, even AUC, for model selection, even if they have no inherent penalty on model complexity.

As you probably know, we do not measure performance on the same data that we train our models on, because the training data error rate is generally an overly optimistic measure of performance in practice (see Section 7.4 of the ESL book). But this is not the most important reason to use train-test splits. The most important reason is to avoid overfitting with excessively complex models.

Given two models A and B such that B "contains A" (the parameter set of B contains that of A) the training error is mathematically guaranteed to favor model B, if you are fitting by optimizing some fit criterion and measuring error by that same criterion. That's because B can fit the data in all the ways that A can, plus additional ways that may produce lower error than A's best fit. This is why you were expecting to see lower error as you added more predictors to your model.

However, by splitting your data into two reasonably independent sets for training and testing, you guard yourself against this pitfall. When you fit the training data aggressively, with many predictors and parameters, it doesn't necessarily improve the test data fit. In fact, no matter what the model or fit criterion, we can generally expect that a model which has overfit the training data will not do well on an independent set of test data which it has never seen. As model complexity increases into overfitting territory, test set performance will generally worsen as the model picks up on increasingly spurious training data patterns, taking its predictions farther and farther away from the actual trends in the system it is trying to predict. See for example slide 4 of this presentation, and sections 7.10 and 7.12 of ESL.

If you still need convincing, a simple thought experiment may help. Imagine you have a dataset of 100 points with a simple linear trend plus gaussian noise, and you want to fit a polynomial model to this data. Now let's say you split the data into training and test sets of size 50 each and you fit a polynomial of degree 50 to the training data. This polynomial will interpolate the data and give zero training set error, but it will exhibit wild oscillatory behavior carrying it far, far away from the simple linear trendline. This will cause extremely large errors on the test set, much larger than you would get using a simple linear model. So the linear model will be favored by CV error. This will also happen if you compare the linear model against a more stable model like smoothing splines, although the effect will be less dramatic.

In conclusion, by using train-test splitting techniques such as CV, and measuring performance on the test data, we get an implicit penalization of model complexity, no matter what metric we use, just because the model has to predict on data it hasn't seen. This is why train-test splitting is universally used in the modern approach to evaluating performance in regression and classification.

Answer 2

There is a good reason why the regression coefficients in logistic regression are estimated by maximizing the likelihood or penalized likelihood. This leads to certain optimality properties. The concordance probability ($c$-index; AUROC) is a useful supplemental measure for describing the final model's predictive discrimination, but it is not sensitive enough for the use you envisioned nor would it lead to an optimal model. This is quite aside from the overfitting issue, which affects both the $c$-index and the (unpenalized) likelihood.

The $c$-index can reach its maximum with a misleadingly small number of predictors, even though it does not penalize for model complexity, because the concordance probability does not reward extreme predictions that are "correct". $c$ uses only the rank order of predictions and not the absolute predicted values. $c$ is not sensitive enough to be used to compare two models.

Seeking a model that does not use the entire list of predictors is often not well motivated. Model selection brings instability and extreme difficulty with co-linearities. If you want optimum prediction, using all candidate features and incorporating penalization will work best in most situations you are likely to encounter. The data seldom have sufficient information to allow one to make correct choices about which variables are "important" and which are worthless.

Answer 3

This should help clarify a few things, in as few words as possible:

• AUC = measure of model's actual predictive performance
• BIC = estimate of model's predictive performance

Performance Measures, like AUC, are something you would use to evaluate a model's predictions on data it has never seen before.

Information Criteria, like BIC, on the other hand, attempt to guess at how well a model would make predictions by using how well the model fit the training data AND the number of parameters used to make that fit as a penalty (using the number of parameters makes for better guesses).

Simply put, BIC (and other information criteria), approximate what performance measures, like AUC, give you directly. To be more precise:

• Information criteria attempt to approximate out-of-sample deviance using only training data, and make better approximations when accounting for the number of parameters used.
• Direct performance measures, like deviance or AUC, are used to asses how well a model makes predictions on validation/test data. The number of parameters is irrelevant to them because they're illustrating performance in the most straightforward way possible.

I thought the link between information criteria and performance measures was hard to understand at first, but it's actually quite simple. If you were to use deviance instead of AUC as a performance measure then BIC would basically tell you what deviance you could expect if you actually made predictions with your model, and then measured their deviance.

This begs the question, why use information criteria at all? Well you shouldn't if you're just trying to build the most accurate model possible. Stick to AUC because models that have unnecessary predictors are likely to make worse predictions (so AUC doesn't penalize them per se, they just happen to have less predictive power).

Answer 4

In logistic regression (I do it univariate for easier typing) you try to explain a binary outome $y_i \in \{0,1\}$ by assuming that it is the outcome of a Bernouilli random variable with a success probability $p_i$ that depends on your explanatory variable $x_i$, i.e. $p_i=P(y_i=1|_{x_i})=f(x_i)$, where $f$ is the logistic function: $f(x)=\frac{1}{1+e^{-(\beta_0+\beta_1 x)}}$. The parameters $\beta_i$ are estimated by maximum likelihood. This works as follows: for the $i$-th observation you observe the outcome $y_i$ and the success probability is $p_i=f(x_i)$, the probability to observe $y_i$ for a Bernouilli with success probability $p_i$ is $p_i^{y_i}(1-p_i)^{(1-y_i)}$. So, for all the observations in the sample, assuming independence between observations, the probability of observing $y_i, i=1,2, \dots n$ is $\prod_{i=1}^np_i^{y_i}(1-p_i)^{(1-y_i)}$. Using the above definition of $p_i=f(x_i)$ this becomes $\prod_{i=1}^nf(x_i)^{y_i}(1-f(x_i))^{(1-y_i)}=$. As the $y_i$ and $x_i$ are observed values, we can see this as a function of the unknown parameters $\beta_i$, i.e. $\mathcal{L}(\beta_0, \beta_1)=\prod_{i=1}^n\left(\frac{1}{1+e^{-(\beta_0+\beta_1 x_i)}}\right)^{y_i}\left(1-\frac{1}{1+e^{-(\beta_0+\beta_1 x_i)}}\right)^{(1-y_i)}$. Maximimum likelihood finds the values for $\beta_i$ that maximise $\mathcal{L}(\beta_0, \beta_1)$. Let us denote this maximum $(\hat{\beta}_0, \hat{\beta}_1)$, then the value of the likelihood in this maximum is $\mathcal{L}(\hat{\beta}_0, \hat{\beta}_1)$.

In a similar way, if you would have used two explanatory variables $x_1$ and $x_2$, then the likelihood function would have had three parameters $\mathcal{L}'(\beta_0, \beta_1, \beta_2)$ and the maximum would be $(\hat{\beta}'_0, \hat{\beta}'_1, \hat{\beta}'_2)$ and the value of the likelihood would be $\mathcal{L}'(\hat{\beta}'_0, \hat{\beta}'_1, \hat{\beta}'_2)$. Obviously it would hold that $\mathcal{L}'(\hat{\beta}'_0, \hat{\beta}'_1, \hat{\beta}'_2) > \mathcal{L}(\hat{\beta}_0, \hat{\beta}_1)$, whether the incerase in likelihood is significant has to be 'tested' with e.g. a likelihood ratio test. So likelihood ratio tests allow you te 'penalize' models with too many regressors.

This is not so for AUC ! In fact AUC does not even tell you whether your 'success probabilities' are well predicted ! If you take all possible couples $(i,j)$ where $y_i=1$ and $y_j=0$ then AUC will be equal to the fraction of all these couples that have $p_i < p_j$. So AUC has to do with (1) how good your model is in distinguishing between '0' and '1' (it tells you about couples with one 'zero' and one 'one'), it does not say anything about how good your model is in predicting the probabilities ! and (2) it is only based on the 'ranking' ($p_i < p_j$) of the probabilities. If adding 1 explanatory variable does not change anything to the ranking of the probabilities of the subjects, then AUC will not change by adding an explanatory variable.

So the first question you have to ask is what you want to predict: do you want to distinguish between zeroes and ones or do you want to have 'well predicted probabilities' ? Only after you have answered this question you can look for the most parsimonious technique.

If you want to distinguish between zeroes and ones then ROC/AUC may be an option, if you want well predicted probabilities you should take a look at Goodness-of-fit test in Logistic regression; which 'fit' do we want to test?.

Answer 5

As Marc said, AUC is only a measure of performance, just like missclassification rate. It does not require any information about the model.

Conversely, BIC, AIC, need to know the number of parameters of your model to be evaluated.

There is no good reason, if all of your predictors are relevant, that the missclassification rate or the AUC decreases when removing variables.

However, it is quite common that combining a learning algorithm, an importance measure of the variables and variable selection (based on the importance the algorithm grants them) will perform better than fitting the model on the whole data set.

You have an implementation of this method for Random Forests in the R RFauc package.