What is the idea behind generalized linear models?

Question

I am watching andrew ng's video lectures on machine learning. I am trying to understand what is even the point of generalized linear models. I understand what goes on step by step in deriving things related to generalized models but as a whole it just looks like a lot of much-ado-about-nothing. Here is my understanding (In reference to modeling a bernoull distribution):

1. We make the assumption that the output is distributed according to a bernoulli distribution with parameter $\phi$
2. $p(y;\phi) = \phi^{y}(1-\phi)^{1-y}$ $\tag 1$
3. We use craft to represent the same equation such that it is in the structure of $ExponentialFamily(\eta): p(y;\eta) = b(y)(\eta^{T}T(y)-a(y)$ where $T(y)$ is a sufficient statistic
4. For modeling, we assume the hypothesis to be $h(x;\theta) = E[T(y)|x]$
5. We extract out the values of $a,b,T,\eta$ in terms of $\phi$. $\phi = \frac1{1+e^{-\eta}}$ and for bernoulli $T(y) = y$. We write the likelihood of data as a function of $\eta$ instead of $\phi$. We assume a linear relationship between $\eta$ and ($x,\theta$) in that $\eta = \theta^T.x$
6. we train this model with our fav algorithm to maximize the likelihood

Questions:

1. This sounds like a lot of 'why did we go through this all?'. We went from one parameter to another and now we are maximizing likelihood over that parameter. Why?

2. What did the assumptions of $\eta = \theta^T.x$ and $h(x;\theta) = E[T(y)|x]$ achieve?

Answer 1

1. The nice thing about GLMs is that they bring together different models under a common denominator. In your question you have discussed logistic regression as a GLM, but you can also show that Poisson regression is a GLM, etc. Properties derived for GLMs apply to all of them.

2. Those are just the linear predictor and the link function. The background is some sort of regression model, so you have a part that is $\beta_0 + \beta_1 x_1 + \beta_2 x_2 ...$. That is the $\eta$. That $\eta$ is related to the dependent variable, through some transformation, the $h(\cdot)$. These are parametric models, so these assumptions can be wrong. If you apply such a model to a particular dataset, it is up to you to make sure that these assumptions are a reasonable simplification of the reality you are studying.

Answer 2

In the case of a Bernoulli response you are looking to model the change in the probability $p = f(x;\eta)$ as a function of your predictors. You can just make a special model for doing this, and that is fine, but suppose you want to use some general technique, such as say a linear model? The immediate problem you hit is that they are going to give you predictions for some inputs that lie outside (0,1) that is allowed for p.

You can fix this! All you need to do is put a transform on your model, we need some function that will map $(-\infty, \infty)$ on to $(0,1)$. Let's use the Normal CDF $\Theta(z)$:

$p = \Theta(f(x;\eta))$

That's it! We have discovered GLM, we have solved our problem. You can now go out and get your data and fit your model using maximum likelihood. For any trial parameter value $\eta$ you can calculate the predictor $p$ for each data point, and calculate the total likelihood for your sample and off you go.

But... that isn't a "canonical" GLM. In fact this example is Probit regression. The canonical form for handling binomial data would be to use Logistic regression, and it is about finding canonical form that the exponential family stuff is all about. The reasons to prefer canonical form are subtle, and not always compelling. However if you use the canonical link function, then some of the mathematics of fitting is simplified as are another of derived results. An excellent answer on the difference between Probit and Logistic this can be found here, and the idea carries forward to any canonical vs non-canonical link: https://stats.stackexchange.com/q/30909