Using generalized linear models over OLS and why do estimates differ?

Question

I am trying to fit a regression model in which the dependent variable is a continuous positive variable (costs, no zero costs). In literature I often read that considering the lack of negative values, a ordinary least square regression is not appropriate. Generalized linear models with for example a Gamma distribution is often recommended over OLS. This leads to the following questions

1. To get unbiased regression estimates the assumptions of the OLS do not specify anything regarding the specification of the error distribution. Hence, even though the errors are clearly not normally distributed, an ordinary least square regression would still provide you with unbiased estimates. In case the sample is sufficiently large and the errors are homoscedastic (or use robust standard errors when they are not), why would one prefer a glm?

2. I fitted a GLM with a gaussian family as well as with a Gamma distribution, with both the identity link function as well as the log link function. However, the regression estimates for the different families (with the same link function) are substantially different (even when considering robust standard errors). Given that both should give unbiased estimates why would this happen?
3. Some papers say that fitting non negative data with an identity link function is inappropriate as it might lead to negative predictions. However, when given the regression coefficients negative predictions are impossible would this still be argument? I am asking as I am interested in the additive effect of my variables. Moreover, the AIC of the Gamma regression with the identity link is the lowest. The error plots for the gamma regression with the identity link also look superior compared to the others.

Answer 1

This is a very broad question, and similar questions have been posted before. So this is mostly and enlarged comments giving links to earlier posts which can help you. But first, shortly, glm's are a modeling framework for many different regression models, and the choice mostly depends on your goals and what kind of parameters you find interpretable in your context. AS to the question in title, estimated differ because this are different models!

So:

• How does OLS regression relate to generalised linear modelling

• What are the differences between ANOVAs and GLMs?

• General Linear Model vs. Generalized Linear Model (with an identity link function?)

• Goodness of fit and which model to choose linear regression or Poisson

(many other relevant posts, search our site)

Answer 2

About your minor sub questions

• 1) In case ... the errors are homoscedastic ... why would one prefer a glm?

  GLM does two things:

  • model different link functions
  • model different conditional error distributions

  If the errors are homoskedastic then the latter is not important, but the first might still be useful.

  Related: Conditions for Poisson regression to outperfom OLS in linear inverse problems

• 3) Why would logistic regression still be important if the values won't be zero?

  It is inappropriate because it is obviously a wrong model. However, occasionally it works fine. You can use linear link function if

  • You only model with a predictor that has two values.(Should I use a Simple Linear or Logistic Regression)
  • When you model only a small range of outcomes. (Interpreting logistic modelling and linear modelling results for the same formula)

2) Using generalized linear models over OLS and why do estimates differ?

While both are unbiased (if the model of the mean is correct), they do not have the same error! So the coefficients do not need to be the same.

The coëfficiënts can differ because the weights of the data points will differ. Fitting with a Gaussian family will give all points equal weight. Fitting with Gamma family will give more weight to lower values (which are considered to be measured with less error).

See the example below. In this case the data was generated with a misspecification ($E[y] = x^2$ but fitted with linear function) to make the differences more clear. The Gamma fit alligns more with the smaller values which have a smaller variance. The Gaussian fit will not know that these have a smaller variance and fits them as if the y had a larger variance.

Fitting with the correct weights will reduce the error/variance of the estimated coëfficiënts. Related: Can robust standard errors be less than those from normal OLS?

Also related is fitting an exponential curve either directly or by using linearized data Differences between approaches to exponential regression and Fitting exponential decay with negative y values