Why generalized linear models performs best for data with high correlation?

Question

I was reading an article on "Credit Card Approval" using Logistic Regression, which basically tells us how Logistic Regression can be used to predict whether or not a credit card will be approved to a person or not. The features of this data set are found to be highly co-related with each other and while talking about why we should use logistic regression the author states that-->

Which model should we pick? A question to ask is: are the features that affect the credit card approval decision process correlated with each other? Although we can measure correlation, that is outside the scope of this notebook, so we'll rely on our intuition that they indeed are correlated for now. Because of this correlation, we'll take advantage of the fact that generalized linear models perform well in these cases. Let's start our machine learning modeling with a Logistic Regression model (a generalized linear model).

I am not able to understand why generalized linear models performs best for data with high correlation? Any help will be highly appreciated. Thank you. :)

Answer 1

In your comments you ask why GLMs fit correlated data well. I think we can answer your question by examining how GLMs are fit and how that fitting procedure changes under correlated features.

A GLM is fit by maximizing the log likelihood function. For Binomial Data, this looks like

$$ \ell(\beta; y,x) = \sum_i y_i \log(\theta(x_i,\beta)) + (1-y_i)\log(1-\theta(x_i,\beta))$$

Here, $y$ is the outcome and $\theta$ is the mean of the binomial conditioned on a vector of covariates $x$ so that $\theta(x,\beta) = (1 + \exp(-x^T\beta)^{-1}$. You can argue that this function is convex and has a unique maximum under some easy to satisfy conditions (we'll assume these are met and ignore the details for now).

The model is fit by finding the vector $\beta$ which maximizes $\ell$

$$ \hat{\beta} = \underset{\beta \in \mathbb{R}^p}{\mbox{armgax}} \Big\{ \ell(\beta;y, x) \Big\} $$

Again, there are some guarantees that this maximum exists under some assumptions about $\ell$ which are not important to the present conversation.

Great, so now we know GLMs are fit by maximizing a function. How is the maximization (and hence the fit) affected by correlated features $x$? I'm going to show you 3 versions of $\ell$: 1 where the features are independent (top left), 1 where the features have a correlation of 0.5 (top right), and 1 where the features have a correlation of 0.99 (bottom left)

When the features are independent, $\ell$ is bowl shaped and we can easy spot the maximum. As the correlation increases, $\ell$ is stretched out along the diagonal, making it flatter. When the features are perfectly correlated, the function becomes completely flat along the diagonal and a unique maximum fails to exist.

So why does a GLM do so well at fitting correlated data. Well, you have to define what "well" means. A GLM can fit correlated data because of guarantees on the maximum existing. The quality of the fit is a different question all together, but those are statistical properties perhaps best left for another time.