1
$\begingroup$

I am working with regression on a data set and I am looking for a way to compare the results.

From the data ($x$) and observed values ($y$) where $y\in[0, 1]$, I have three models:

1 (baseline): Predict all values as the mean observed value $\hat{y} = \bar{x}$.

2 (linear): Fit a model of the form $\hat{y}=ax+b$.

3 (generalised linear model): Fit a GLM with quadratic model and logistic link function, i.e. $\hat{y} = \left(1 + e^{-\left(\beta_0 + \beta_1x + \beta_2x^2\right)}\right) ^{-1}$.

I want to test whether models 2 and 3 have a statistically significant improvement over model 1.

I understand that I can use an F-test to compare models 1 and 2. Matlab does this with the GeneralizedLinearModel.devianceTest function and I understand how it works there.

Can I use an F-test to compare models 1 and 3? If not, why not?

For model 3, using GeneralizedLinearModel.devianceTest in Matlab to compare with the constant model uses a chi-squared test rather then the F-test. Why is this? It gives very different values to when I attempt an F-test manually.


Code

My matlab code is as follows:

model1 = mean(output);
model2 = fitglm(data, output, 'linear');
model3 = fitglm(data, output, 'purequadratic', 'Distribution', 'binomial', 'Link', 'logit');

model1_rss = sum((output - model1) .^ 2);
model2_rss = sum((output - predict(model2, data)) .^ 2);
model3_rss = sum((output - predict(model3, data)) .^ 2);

n = numel(output);

model1_params = 1;
model2_params = 2;
model3_params = 3;

model2_F = ((model1_rss - model2_rss) / (model2_params - model1_params)) / (model2_rss / (n - model2_params))
model3_F = ((model1_rss - model3_rss) / (model3_params - model1_params)) / (model3_rss / (n - model3_params))

model2_p = 1 - fcdf(model2_F, model2_params - model1_params, n - model2_params)
model3_p = 1 - fcdf(model3_F, model3_params - model1_params, n - model3_params)

Running GeneralizedLinearModel.devianceTest gives the following results:

>> model2.devianceTest

ans = 

                  Deviance    DFE    FStat    pValue 
                  ________    ___    _____    _______

    y ~ 1         9.5612      293                    
    y ~ 1 + x1    9.3669      292    6.057    0.01443

>> model3.devianceTest

ans = 

                                Deviance    DFE    chi2Stat    pValue 
                                ________    ___    ________    _______

    logit(y) ~ 1                53.189      293                       
    logit(y) ~ 1 + x1 + x1^2    51.696      291    1.4923      0.47418

My values for model2_F and model2_p match, but not model3_F and model3_p because Matlab uses a chi-squared test rather than an F-test:

          modelx_F  modelx_p
model2    6.0570    0.0144
model3    4.3372    0.0139

My fundamental question is: is it reasonable to test model 1 vs model 3 using an F-test in this way?

$\endgroup$
3
  • $\begingroup$ I see no justification for using an F-test when doing testing in a binomial model, whereas there is an argument for an asymptotic chi-square test. On the other hand some packages try to do t-tests and these equally lack an argument in the same situation. The best one can say for them is they asymptotically approximate the asymptotic approximation, which isn't an argument for any small sample benefit. Does Matlab offer any justification you know of? $\endgroup$
    – Glen_b
    Feb 7, 2016 at 6:45
  • $\begingroup$ @Glen_b, thank you for your comment. Please can I turn your first sentence around and ask why not to use an F-test for the binomial model? I'm still quite new to these techniques so your input is very appreciated. Perhaps the use of a binomial model is misleading - The predicted value isn't really binomial, I'm simply using it because the output is in [0, 1]. Matlab's justification from the documentation is that it uses the "F statistic or Chi-squared statistic, depending on whether the dispersion is estimated (F statistic) or not (Chi-squared statistic)", but I don't really understand this. $\endgroup$
    – zelanix
    Feb 7, 2016 at 11:24
  • $\begingroup$ Or perhaps a better (and more general) question is what is the best way to compare two (unrelated) regression models? (although I would argue that the constant value GLM is equivalent to the constant value linear model, just with a scaling factor applied to the constant value). $\endgroup$
    – zelanix
    Feb 7, 2016 at 11:25

1 Answer 1

1
$\begingroup$

No it's not reasonable to compare Models 3 and 1 via an F-test.

F-tests are used as a approximate way to accommodate overdispersion for binomial, Poisson and negative binomial generalized linear models. For binary regression (with $y=0$ or $y=1$), overdispersion relative to a binomial model is not possible, so you should stick to chisquare likelihood ratio tests.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.