6
$\begingroup$

Can anyone please shed some light on the relationship between OLS and generalised linear model?

Has it to do with the distribution of the error terms, general linear model requires normality in the distribution of the errors (foundation of least squares) while generalised linear models dont have the same assumptions?

Have I missed something?

$\endgroup$
8
$\begingroup$

In the context of generalized linear models (GLMs), OLS is viewed as a special case of GLM. Under this framework, the distribution of the OLS error terms is normal (gaussian) and the link function is the identity function.

Generalized linear models allow for different error distributions and also allow the dependent (or response) variable to have a different relationship with the independent variables. This allows for modelling counts or binary or multinomial outcomes. This relationship is encoded in the link function.

Below is an example using R to show that OLS is a special case of GLM:

# create data
x <- 1:20
y <- 2*x + 3 + rnorm(20)

# OLS
lm(y~x)

Call: lm(formula = y ~ x)

Coefficients: (Intercept) x
2.706 2.011

# GLM
glm(y~x, family=gaussian(identity))

Call: glm(formula = y ~ x, family = gaussian(identity))

Coefficients: (Intercept) x
2.706 2.011

Degrees of Freedom: 19 Total (i.e. Null); 18 Residual Null Deviance: 2717 Residual Deviance: 28.98 AIC: 70.18

It is important to note that OLS can also be viewed mathematically as the linear projection of the dependent variable onto the independent variables in a manner that minimizes the squared distance from the projection to the observations. From this bare-bones viewpoint, the assumption of normality in the conditional error term is irrelevant.

However, to make statistical inferences, assumptions must be added. Either normality of the error term, or more typically, the central limit theorem are invoked for the purpose of inference following estimation of an OLS model.

$\endgroup$
  • $\begingroup$ The way I remember it, OLS can be very Ordinary: without any distributional assumptions. Apparently it is often associated with the BLUE property, implied by the distributional assumption, but unless my brain has turned to mush (I haven't done OLS in years), you can do OLS without BLUE and without distributional assumptions. $\endgroup$ – PatrickT May 9 '16 at 14:22
  • $\begingroup$ The operation of estimating coefficients ("betas") as you say, is simply a mathematical formula, but if any inference is desired, then we have to make some assumptions. In my experience, this is either in the distribution of the error term or in some reliance on asymptotics and CLM. I'll edit my answer to reflect this. $\endgroup$ – lmo May 9 '16 at 14:36
5
$\begingroup$

Generalized linear models are an extension of OLS. In both there is a linear relationship between the "dependent" variable and the explanatory variables of the form: $y=β_0+β_1x_1+β_2x_2+...β_nx_n+ε$ or $\mathbf{y}=\mathbf{X} \mathbf{\beta}$. In generalized linear models, though, $\mathbf{\rho}=\mathbf{X} \mathbf{\beta}$, so that the relationship to $E(Y) = μ = g^{−1}(\rho)$.

In OLS the assumption is that the residuals follow a normal distribution with mean zero, and constant variance. This is not the case in glm, where the variance in the predicted values to be a function of $\ E(y)$.

$\endgroup$
  • 3
    $\begingroup$ OLS can be, and almost always is, derived without any regards to distributions $\endgroup$ – Repmat May 9 '16 at 12:32
  • 1
    $\begingroup$ @Repmat yes since normality is needed for confidence intervals and testing only; and no since to define regression as a statistical model notion of distribution is needed. $\endgroup$ – Tim May 9 '16 at 12:36
  • $\begingroup$ OLS can be used for inference, hence also for CI, without any distributional assumptions. Yes you need an distribution to the define model, OLS puts no restriction on that distribution (you can ofc do that, with large gains in power and efficiency. But it is completely arbitrary to to do so). $\endgroup$ – Repmat May 9 '16 at 12:39
  • $\begingroup$ @Repmat agree, but still making such distributional assumption makes it easier to compare it with GLM. $\endgroup$ – Tim May 9 '16 at 12:41
  • 1
    $\begingroup$ OLS is efficient in the class of linear estimator, and all estimators which uses the function g(x). But yes you need normality to get efficiency in the Cramer Rao lower bound sense. All I was trying say is that normality is not needed for OLS to be extremely usefull. $\endgroup$ – Repmat May 9 '16 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.