Why GLM fit is different from direct fit using logistic function [closed]

Question

I found in my data that a direct fit using the logistic function gives a different and better (R^2) fit than GLM fit using binomial distribution with logit link function. I was naively expecting the same results but now can't explain why this is not the case.

As pointed out below in comments it could be due to different minimisation procedures.

Assuming minimisations used by fit and glm are the same does one expect identical fitting results?

Below examples assume the same minimisation procedure.

Here is some matlab code to illustrate the issue

Direct fit

x = [-50.4 -39.6 -29.7 -21.6 -18.0 -14.4 -9.9 -8.1 -6.3 -3.6 -1.8 0.0 
      1.8 3.6 6.3 8.1 9.9 14.4 18.0 21.6 29.7 39.6 50.4]';
y = [0.0 0.0 0.0 0.0 0.1 0.1 0.2 0.2 0.2 0.3 0.3 0.4 0.5 0.5 0.6 0.6 
     0.7 0.8 0.9 0.9 1.0 1.0 1.0]';

ft=fittype('exp(b0+b1*x)/(1 + exp(b0+b1*x))', 'indep', 'x');
stp=[-0.5 0.1];
[mo, gf] = fit(x, y, ft, 'StartPoint', stp);

mo = 
     General model:
     mo(x) = exp(b0+b1*x)/(1 + exp(b0+b1*x))
     Coefficients (with 95% confidence bounds):
       b0 =     -0.4201  (-0.4983, -0.3419)
       b1 =      0.1268  (0.1167, 0.1368)

GLM part

data=table(y,x);

modelspec{1}='y ~ x';
mdl1 = fitglm(data, modelspec{1},...
    'Distribution', 'binomial', 'Link','logit');

Generalized linear regression model:
    logit(y) ~ 1 + x
    Distribution = Binomial

Estimated Coefficients:
                   Estimate       SE        tStat       pValue 
                   ________    ________    ________    ________

    (Intercept)    -0.42089     0.60619    -0.69432     0.48748
    x               0.13401    0.060266      2.2236    0.026175

The difference is not huge but noticeable especially if one expects identical results.

Here is another data example (this one is asymmetric and I know that fit will be by definition symmetric).

y=[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.1, 0.5, 1.0, 
   1.0, 1.0, 1.0, 1.0, 0.7, 0.9, 0.7, 0.8, 0.8, 1.0];

Results

mo = 
     General model:
     mo(x) = exp(b0+b1*x)/(1 + exp(b0+b1*x))
     Coefficients (with 95% confidence bounds):
       b0 =     0.07416  (-0.793, 0.9413)
       b1 =       1.606  (-0.1469, 3.359)

mdl1 = 
Generalized linear regression model:
    logit(y) ~ 1 + x
    Distribution = Binomial

Estimated Coefficients:
                   Estimate       SE        tStat       pValue 
                   ________    ________    ________    ________

    (Intercept)    -0.35437     0.61901    -0.57247       0.567
    x               0.14666    0.066227      2.2145    0.026791

Here the difference is huge.

Is there any simple explanation as to why the outputs are different? (I check the same data in R using nlsLM and glm and got identical results)

Answer 1

One of these is a logistic regression model (a generalised linear model) and the other is a nonlinear model fitted by least squares.

I verified this by fitting those two models in R and seeing that they give exactly these results. It's not an optimisation issue; it's a model issue.

For the first example: non-linear least squares

> nls(y~exp(a+b*x)/(1+exp(a+b*x)),data=data.frame(x=x,y=y))
Nonlinear regression model
  model: y ~ exp(a + b * x)/(1 + exp(a + b * x))
   data: data.frame(x = x, y = y)
      a       b 
-0.4201  0.1268 
 residual sum-of-squares: 0.01512

and logistic regression

> glm(y~x, family=binomial())

Call:  glm(formula = y ~ x, family = binomial())

Coefficients:
(Intercept)            x  
    -0.4209       0.1340  

The nonlinear least squares fit finds $a$ and $b$ to minimise the residual sum of squares $${\text RSS} = \sum_{i=1}^n \left(y_i - \mathrm{expit}(a+bx_i) \right)^2$$ where $\mathrm{expit}$ is the function $x\mapsto \frac{\exp(x)}{1+\exp(x)}$

The glm finds $a$ and $b$ to minimise the deviance $$-2\log L= -2\sum_{i=1}^n y_i\log \mathrm{expit} (a+bx_i)+ (1-y_i) \log(1-\mathrm{expit} (a+bx_i))$$

A key difference between these is that nonlinear least squares gives the same weight to all the squared errors but logistic regression weights them according to how close they are to 0 or 1

Answer 2

I found in my data that a direct fit using the logistic function gives a different and better (R^2) fit than GLM fit using binomial distribution with logit link function

There are two seperate issues here. Why is it different? Why is least squares better (in terms of R^2)?

• The results are different because GLM is different from least squares regression. A different cost function is used. Here is an example for the difference in cost functions with GLM using either Gaussian or Poisson family: What is the objective function to optimize in glm with gaussian and poisson family?. In the case of a binomial family you get some similar sort of difference.

• The $R^2$ may be better, but that doesn't necessarily make the fit better. Obviously, the $R^2$ will be better because least squares regression minimizes exactly that value.

  But this measure of performance is biased, because you compare with the same sample that you used to train/fit the model.

  It is often better to use the likelihood function, based on the assumed distribution of the data, to fit the model. This will result in a smaller variance/error of the estimate. See for example: Could a mismatch between loss functions used for fitting vs. tuning parameter selection be justified? Other cases (where OLS doesn't perform better): Is OLS asymptotically the best estimator even without gaussian error? or Should we always minimize squared deviations if we want to find the dependency of mean on features?