In helping us understand how to fit a logistic regression in R, we are told to first replace 0 and 1 in the response variable by 0.05 and 0.95, respectively and second to take the logit transform of the resulting response variable. Last we fit these data using iterative re-weighted least squares method.

Then we are asked to use 0.005 and 0.995 instead of 0.05 and 0.95. Then the resulting coefficients are quite different.

My question is in glm function, how are 0 and 1 dealt with? Are they replaced by some numbers as above? What numbers are used by default and why are they used? How sensitive is the choice of these numbers?

  • $\begingroup$ I am wondering if you replace 0 and 1 with 0.05 and 0.95 what link function will you use then? $\endgroup$ – Deep North Sep 22 '15 at 0:51
  • $\begingroup$ @DeepNorth For logistic regression, obviously the link function is logit. In particular, log(p/1-p). $\endgroup$ – LaTeXFan Sep 22 '15 at 0:55
  • $\begingroup$ while logit link is a transformation for binomial distribution and for binomial distribution there are only 1s and 0s. $\endgroup$ – Deep North Sep 22 '15 at 1:01

That is very strange advice, I am forced to wonder who in the world advanced it.

The correct way to fit a logistic regression leaves the zeros and ones alone, and determines the parameters that minimize the log likelihood function:

$$ f(\beta) = \sum_i y_i \log(p_i) + (1 - y_i) \log(1 - p_i) $$

Where $p_i$ is shorthand for

$$ p_i = \frac{e^{\beta \cdot x_i}}{1 + e^{\beta \cdot x_i} }$$

The exponents are vector dot products and $p_i$ is a function of the parameter vector $\beta$. The $y_i$s in this expression are either $0$ or $1$, and it's pleasant to notice that this causes each term to be equal to either

$$ \log(p_i) $$


$$ \log(1 - p_i) $$

Generally, yes, this expression is minimized using a method called iteratively re-weighted least squares, which is itself derived from Newton's classical method for minimizing non-linear functions.

R's glm function does exactly this. No response replacement in sight.

  • $\begingroup$ Could you give a reference on how to solve this minimisation problem using the iteratively re-weighted least squares, please? $\endgroup$ – LaTeXFan Sep 22 '15 at 1:27
  • $\begingroup$ It's the kind of thing that's worth working out for yourself, but here's a reference that I like: win-vector.com/blog/2011/09/… $\endgroup$ – Matthew Drury Sep 22 '15 at 1:43
  • $\begingroup$ Thanks. BTW, I do not think the idea in the question strange. The idea of link function is to transform the original response variable (0, 1) so that after transformation we obtain linear structure. Hence, the name GLM. $\endgroup$ – LaTeXFan Sep 22 '15 at 2:05
  • 1
    $\begingroup$ You're welcome. I don't think your explanation of a glm is quite right. I do not believe in any cases does the algorithm for fitting a glm transform the original response variable (or, none that I have in my working memory). The structure equation for a glm is $g(E(y \mid x)) = \beta \cdot x$, it is the conditional expectation that is transformed, not $y$ itself. $\endgroup$ – Matthew Drury Sep 22 '15 at 2:08
  • 2
    $\begingroup$ In GLM you transform the conditional mean, not the dependent variable. This is why a log link function can deal with 0s in the dependent variable and a logit link function can deal with 0s and 1s in the dependent variable. For the former see: dx.doi.org/10.1162/rest.88.4.641, for the latter see dx.doi.org/10.1002/… $\endgroup$ – Maarten Buis Sep 22 '15 at 8:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.