Skip to main content
67 votes

How to decide which glm family to use?

Generalized linear model is defined in terms of linear predictor $$ \eta = \boldsymbol{X} \beta $$ that is passed through the link function $g$: $$ g(E(Y\,|\,\boldsymbol{X})) = \eta $$ It models ...
Tim's user avatar
  • 139k
50 votes
Accepted

Why using Newton's method for logistic regression optimization is called iterative re-weighted least squares?

Summary: GLMs are fit via Fisher scoring which, as Dimitriy V. Masterov notes, is Newton-Raphson with the expected Hessian instead (i.e. we use an estimate of the Fisher information instead of the ...
jld's user avatar
  • 20.4k
46 votes
Accepted

Is there any intuitive explanation of why logistic regression will not work for perfect separation case? And why adding regularization will fix it?

A 2D demo with toy data will be used to explain what was happening for perfect separation on logistic regression with and without regularization. The experiments started with an overlapping data set ...
Haitao Du's user avatar
  • 37k
39 votes
Accepted

What important ideas came since Nelder and McCullagh's book Generalized Linear Models (a 40 year old book)?

Your premise that the elapsing of 40 years means that "surely things have changed" is quite dubious in a field relating to applied mathematics. In mathematical work it is often the case ...
Ben's user avatar
  • 127k
38 votes
Accepted

How do you deal with "nested" variables in a regression model?

Meaningless values of nested variables must not affect your model: The crucial desideratum with this type of data analysis is that the nested variable must not ...
Ben's user avatar
  • 127k
38 votes
Accepted

Why do we model noise in linear regression but not logistic regression?

Short answer: we do, just implicitly. A possibly more enlightening way of looking at things is the following. In Ordinary Least Squares, we can consider that we do not model the errors or noise as $N(...
Stephan Kolassa's user avatar
35 votes
Accepted

When to use a GAM vs GLM

The main difference imho is that while "classical" forms of linear, or generalized linear, models assume a fixed linear or some other parametric form of the relationship between the dependent variable ...
matteo's user avatar
  • 3,253
32 votes
Accepted

How to choose "family" in Generalized Additive Model (GAM)

Here is an example of what I mean by "outcome conditioned on the covariate". I want to do a linear regression. I have a continuous outcome and I am regressing it on a binary variable. This ...
Demetri Pananos's user avatar
30 votes
Accepted

Interpretation of Fixed Effects from Mixed Effect Logistic Regression

Indeed, in a mixed effects logistic regression and because of the nonlinear link function that is used to connect the mean of the outcome with the linear predictor, the fixed effects coefficients have ...
Dimitris Rizopoulos's user avatar
29 votes

Family of GLM represents the distribution of the response variable or residuals?

The family argument for glm models determines the distribution family for the conditional distribution of the response, not of the residuals (except for the quasi-models). Look at this way: For the ...
kjetil b halvorsen's user avatar
29 votes

When to use a GAM vs GLM

I'd emphasize that GAMs are much more flexible than GLMs, and hence need more care in their use. With greater power comes greater responsibility. You mention their use in ecology, which I have also ...
Wayne's user avatar
  • 21.3k
28 votes

What is the difference between a "link function" and a "canonical link function" for GLM

Here is a little diagram inspired from MIT's 18.650 class which I find quite useful as it helps visualizing the relationships between these functions. I have used the same notation as in @momo's post: ...
Xavier Bourret Sicotte's user avatar
28 votes

How to calculate goodness of fit in glm (R)

Use the Null Deviance and the Residual Deviance, specifically: 1 - (Residual Deviance/Null Deviance) If you think about it, you're trying to measure the ratio of ...
noLongerRandom's user avatar
28 votes
Accepted

Can you give a simple intuitive explanation of IRLS method to find the MLE of a GLM?

Some years ago I wrote a paper about this for my students (in spanish), so I can try to rewrite those explanations here. I will look at IRLS (iteratively reweighted least squares) through a series of ...
kjetil b halvorsen's user avatar
26 votes

What do the residuals in a logistic regression mean?

Response: $$y_i - \hat\mu_i$$ response residuals are inadequate for assessing a fitted glm, because GLMs are based on distributions where (in general) the variance depends on the mean. Pearson: The ...
Maverick Meerkat's user avatar
25 votes
Accepted

Why Beta/Dirichlet Regression are not considered Generalized Linear Models?

Check the original reference: Ferrari, S., & Cribari-Neto, F. (2004). Beta regression for modelling rates and proportions. Journal of Applied Statistics, 31(7), 799-815. as the authors note, ...
Tim's user avatar
  • 139k
24 votes

Does including gender as a predictor variable mean I should use a glm function, not an lm function, in R?

A few points: The short (tl;dr) answer to your question is that the choice of linear (lm) vs generalized linear (glm) models ...
Ben Bolker's user avatar
23 votes
Accepted

Splines in GLM and GAM

You are mistaken. Splines have a linear representation using derived covariates. As an example, a quadratic trend is non-linear, but can be modeled in a linear model by taking: $E[Y|X] = \beta_0 + \...
AdamO's user avatar
  • 63.1k
23 votes

What important ideas came since Nelder and McCullagh's book Generalized Linear Models (a 40 year old book)?

In addition to Ben's great answer (+1): Penalised regression models ($L_1$, $L_2$, elastic net, SCAD (Smoothly clipped absolute deviation), LARS (least-angle regression), MCP (Multiple Change Points),...
usεr11852's user avatar
  • 44.5k
22 votes
Accepted

What is the difference between logistic regression and Fractional response regression?

If your question is: what is the difference between these two codes? A look at ?glm says ...
coffeinjunky's user avatar
  • 2,016
22 votes
Accepted

How is a Poisson rate regression equal to a Poisson regression with corresponding offset term?

This also confused me. I thought, "what is the point of explicitly including an offset instead of just pretending that the response divided by the offset / exposure is the $y$ value?". You actually ...
Cokes's user avatar
  • 342
22 votes

Family of GLM represents the distribution of the response variable or residuals?

Further to Kjetil's excellent answer, I wanted to add some specific examples to help clarify the meaning of a conditional distribution, which can be a bit of an elusive concept. Let's say you took a ...
Isabella Ghement's user avatar
21 votes

Which optimization algorithm is used in glm function in R?

The method used is mentioned in the output itself: it is Fisher Scoring. This is equivalent to Newton-Raphson in most cases. The exception being situations where you are using non-natural ...
AdamO's user avatar
  • 63.1k
21 votes

R - lmer vs glmer

lmer is used to fit linear mixed-effect models, so it assumes that the residual error has a Gaussian distribution. If your dependent variable A is a binary outcome (...
matteo's user avatar
  • 3,253
21 votes

Interpretation of .L & .Q output from a negative binomial GLM with categorical data

Your variables aren't just coded as factors (to make them categorical), they are coded as ordered factors. Then, by default, R fits a series of polynomial functions to the levels of the variable. ...
gung - Reinstate Monica's user avatar
21 votes
Accepted

Why exactly can't beta regression deal with 0s and 1s in the response variable?

Because the loglikelihood contains both $\log(x)$ and $\log(1-x)$, which are unbounded when $x=0$ or $x=1$. See equation (4) of Smithson & Verkuilen, "A Better Lemon Squeezer? Maximum-Likelihood ...
Kevin Wright's user avatar
21 votes
Accepted

What to do with GLM (Gamma) when residuals are not normally distributed?

Residuals in glm's such as with the gamma family is not normally distributed, so simply a QQ plot against the normal distribution isn't very helpful. To understand this, note that the usual linear ...
kjetil b halvorsen's user avatar
21 votes
Accepted

Do test scores really follow a normal distribution?

Height, for instance, is often modelled as being normal. Maybe the height of men is something like 5 foot 10 with a standard deviation of 2 inches. We know negative height is unphysical, but under ...
Demetri Pananos's user avatar
20 votes

Diagnostic plots for count regression

This is an old question, but I thought it would be useful to add that my DHARMa R package (available from CRAN, see here) now provides standardized residuals for GLMs and GLMMs, based on a simulation ...
Florian Hartig's user avatar
20 votes

Diagnostics for generalized linear (mixed) models (specifically residuals)

This is an old question, but I thought it would be useful to add that option 4 suggested by the OP is now available in the DHARMa R package (available from CRAN, see here). The package makes the ...
Florian Hartig's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible