1
$\begingroup$

This is perhaps basic but I couldn't find a suitable reference.

I have a regression model with a rather complicated link function. So $\vec{x}$ is a vector of continuous predictors, and $z$ is a binary variable such that according to the model: $Pr(z=1) = f(\vec{x})$ for some (known) function $f$.

I observe data of the form $(\vec{x}^{(1)}, z^{(1)}), (\vec{x}^{(2)}, z^{(2)}), (\vec{x}^{(n)}, z^{(n)})$ and want to test the null hypothesis that the above model is the one generating the data - that is compute a statistic and reject the model if the statistic is too extreme. What would be a good goodness-of-fit test for this case? is there a 'standard' way to test for this?

One possibility is binning the data points by the value of $f(\vec{x})$, (say to $10$ bins: $([0,0.1], ..[0.9,1])$ and performing a chi-square test for expected vs. observed proportion of $z$'s in each bin. Another is to bin the multidimensional space of the $\vec{x}$'s (say if $\vec{x}$ is two-dimensional, we can divide $R^2$ to $100$ squares and compute a chi-square for observed vs. expected for each square). Yet another one is not binning at all but just computing $\sum_i (z^{(i)} - f(\vec{x}^{(i)}))^2/f(\vec{x}^{(i)})$ but this seems to cause numerical issues since sometimes $f(\vec{x}^{(i)})$ is very small. Are there other known approaches? which test would be the most appropriate?

Improve this question
$\endgroup$
2
  • $\begingroup$ what your s regression model is designed for? Do you have to interpret parameters, or you indeed are interested in the best fit (that not necessarily result best forecasting performance)? Perhaps what you indeed are looking for is some sort of cross-validation, to be sure that parameter estimates are stable and not sample dependent. There are many nice suggestions how to do cross-validation in other posts. $\endgroup$
    – Dmitrij Celov
    Commented May 21, 2011 at 5:45
  • $\begingroup$ Thanks. My goal is actually a bit different. I want to compute power to reject the null hypothesis (the regression model). I'm generating data using different (other) models and compare them in terms of how easily can you detect that they're different from the null model - but I assume that the statistician performing the test doesn't know the alternative model, so he can't apply a log-likelihood ratio test between the two models, he just needs a test for rejecting the null model and the goal is to see how powerful is this test under different alternative models $\endgroup$
    – Or Zuk
    Commented May 21, 2011 at 14:35

1 Answer 1

Reset to default
2
$\begingroup$

The $R^2$ in regression is given that name because it is the correlation between $y_i$ and $\hat{y}_i$ squared. You could calculate the correlation between $z_i$ and $\hat{z}_i$ in your case, square it, and use that as a measure of goodness-of-fit. I can't say what the statistical properties of this measure will be for your particular case, however.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ It won't be too good since R^2 doesn't apply to logistic regression models. There are "pseudo R^2's" of various sorts but they are all hokey & pander to the view that OLS is some sort of gold standard for multivariate testing. Best to use some form of likelihood ratio test that is based on agreement of predicted & observed values. Good discussion of options at faculty.chass.ncsu.edu/garson/PA765/logistic.htm $\endgroup$
    – dmk38
    Commented May 26, 2011 at 21:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.