This is perhaps basic but I couldn't find a suitable reference.

I have a regression model with a rather complicated link function. So $\vec{x}$ is a vector of continuous predictors, and $z$ is a binary variable such that according to the model: $Pr(z=1) = f(\vec{x})$ for some (known) function $f$.

I observe data of the form $(\vec{x}^{(1)}, z^{(1)}), (\vec{x}^{(2)}, z^{(2)}), (\vec{x}^{(n)}, z^{(n)})$ and want to test the null hypothesis that the above model is the one generating the data - that is compute a statistic and reject the model if the statistic is too extreme. What would be a good goodness-of-fit test for this case? is there a 'standard' way to test for this?

One possibility is binning the data points by the value of $f(\vec{x})$, (say to $10$ bins: $([0,0.1], ..[0.9,1])$ and performing a chi-square test for expected vs. observed proportion of $z$'s in each bin. Another is to bin the multidimensional space of the $\vec{x}$'s (say if $\vec{x}$ is two-dimensional, we can divide $R^2$ to $100$ squares and compute a chi-square for observed vs. expected for each square). Yet another one is not binning at all but just computing $\sum_i (z^{(i)} - f(\vec{x}^{(i)}))^2/f(\vec{x}^{(i)})$ but this seems to cause numerical issues since sometimes $f(\vec{x}^{(i)})$ is very small. Are there other known approaches? which test would be the most appropriate?

  • $\begingroup$ what your s regression model is designed for? Do you have to interpret parameters, or you indeed are interested in the best fit (that not necessarily result best forecasting performance)? Perhaps what you indeed are looking for is some sort of cross-validation, to be sure that parameter estimates are stable and not sample dependent. There are many nice suggestions how to do cross-validation in other posts. $\endgroup$ – Dmitrij Celov May 21 '11 at 5:45
  • $\begingroup$ Thanks. My goal is actually a bit different. I want to compute power to reject the null hypothesis (the regression model). I'm generating data using different (other) models and compare them in terms of how easily can you detect that they're different from the null model - but I assume that the statistician performing the test doesn't know the alternative model, so he can't apply a log-likelihood ratio test between the two models, he just needs a test for rejecting the null model and the goal is to see how powerful is this test under different alternative models $\endgroup$ – Or Zuk May 21 '11 at 14:35

The $R^2$ in regression is given that name because it is the correlation between $y_i$ and $\hat{y}_i$ squared. You could calculate the correlation between $z_i$ and $\hat{z}_i$ in your case, square it, and use that as a measure of goodness-of-fit. I can't say what the statistical properties of this measure will be for your particular case, however.

| cite | improve this answer | |
  • 1
    $\begingroup$ It won't be too good since R^2 doesn't apply to logistic regression models. There are "pseudo R^2's" of various sorts but they are all hokey & pander to the view that OLS is some sort of gold standard for multivariate testing. Best to use some form of likelihood ratio test that is based on agreement of predicted & observed values. Good discussion of options at faculty.chass.ncsu.edu/garson/PA765/logistic.htm $\endgroup$ – dmk38 May 26 '11 at 21:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.