Link Functions where MLE=OLS

Question

This is a follow-up of a question that I posted previously. I'm trying to get parameter estimates from two different SAS functions (Proc REG and Proc GENMOD) to equal each other when run on the same data set. Proc REG uses OLS/WLS to estimate parameters; Proc GENMOD uses MLE.

My understanding is that OLS is equal to MLE when the assumed distribution is normal.

In my program, I can get the parameter estimates to be equal if I select a normal distribution for Proc GENMOD and if my link function is set to identity. This works whether I choose to use weights or not.

However, if I change my link function in GENMOD to Log, and change the response variable in Proc REG from Y to LOG(Y), the parameter estimates no longer match.

Is there something about the use of the log link function that no longer makes OLS=MLE? I've only ever read that equivalence is simply based on the assumption of a normal distribution.

Answer 1

My understanding is that OLS is equal to MLE when the assumed distribution is normal.

Right. Given the model $y=X\beta+\varepsilon$, OLS minimizes the quantity $$S(\beta)=\sum_{i=1}(y_i-X_i'\beta)^2=(y-X\beta)'(y-X\beta)=y'y-2y'X\beta+\beta'X'X\beta$$ Since $$\frac{\partial S(\beta)}{\partial\beta}=-2X'y+2X'X\beta$$ you get $$\hat\beta_{OLS}=(X'X)^{-1}X'y$$ whatever the distribution of $y$. Normality is only needed for $t$- and $F$-tests: if $\varepsilon\sim\mathcal{N}(0,\sigma^2I)$, then $y\sim\mathcal{N}(X\beta,\sigma^2I)$ and $\hat\beta\sim\mathcal{N}\left(\beta,\sigma^2(X'X)^{-1}\right)$.

However, if $y\sim\mathcal{N}(X\beta,\sigma^2I)$ then the log-likelihood for $\beta$ is: $$\ell(\beta)=-\frac{n}{2}\ln(2\pi)-\frac{n}{2}\ln\sigma^2 -\frac{(y-X\beta)'(y-X\beta)}{2\sigma^2}\tag{1}$$ which is maximized when $(y-X\beta)'(y-X\beta)$ is minimized. Thus $\hat\beta_{ML}=\hat\beta_{OLS}$.

In linear models $E[y\mid X]=X\beta$. In generalized linear models $X\beta=g(E[y\mid X])$ and $E[y\mid X]=g^{-1}(X\beta)$, where $g$ is a link function.

If $g$ is the identity function, then you still have $E[y\mid X]=X\beta$ as in linear models. But if $g(\cdot)=\ln(\cdot)$, then $E[y\mid X]=\exp(X\beta)$. The log link is tipically used in Poisson regression, where one assumes $$y\sim\text{Poisson}(\theta),\quad f(y\mid X)=\frac{e^{-\theta}\theta^y}{y!},\quad E[y\mid X]=\theta=\exp(X\beta)$$ and the log-likelihood for $\beta$ is $$\ell(\beta)=\sum_{i=1}^n[y_iX_i'\beta-\exp(X_i'\beta)-\ln y!] \propto \sum_{i=1}^n[y_iX_i'\beta-\exp(X_i'\beta)]$$ which is very different from $(1)$, and $\hat\beta_{ML}$ is the value such that $$\frac{\partial\ell}{\partial\beta}=\sum_{i=1}^n[y_i-\exp(X_i'\beta)]X_i=0$$ This is why if the link function is not the identity function then $\hat\beta_{ML}\ne\hat\beta_{GLS}$.

Answer 2

Caution: I am not a SAS user.

My understanding is that OLS is equal to MLE when the assumed distribution is normal.

You are falsely opposing OLS and MLE. OLS estimates are MLE parameter estimates. OLS models are appropriate for the class of models where you are modeling $\mathbb{E}(y|x) \sim \mathcal{N}( \beta_0+\beta_1x, \epsilon^2)$. But there are many alternative cases. These alternative cases are what GENMOD was designed for.

When there are not weights specified, OLS parameters are maximum likelihood estimates.

Is there something about the use of the log link function that no longer makes OLS=MLE? I've only ever read that equivalence is simply based on the assumption of a normal distribution.

Yes. That is the manifest purpose of GENMOD with non-identity links: to specify non-OLS models. Function GENMOD constructs generalized linear models (GLMs). This is a class of models where the expectation of the response may be of an alternative distribution than would be assumed under an OLS model. You can still use the GLM command to build an OLS model; simply specify the identity link as you have already done!

Selecting a log link in the GLM is telling SAS that you want to build a specific kind of model: one in which $\mathbb{E}(y|x)\sim\exp(\beta_0+\beta_1x)$. For a Poisson regression model, the log link is the canonical link, and transforms your linear predictors to be on the appropriate scale of the assumed response.

There are, of course, other link functions. The most commonly used in my experience is binary regression, in which the response variable is a "yes" or "no" or a 1 or 0 or any dichotomous outcome. In this case, the link function is $\text{logit}$ and you are performing inference on the factors which cause our outcome to appear as 1 or 0.

Parameters of GLMs are estimated using MLE procedures. So the coefficients will have the desirable qualities of MLEs, provided that the usual assumptions are met.