Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a logistic regression model for 0/1 binary response data that is built from samples $(x_1,Y_1),\ldots,(x_m,Y_m)$, where $x_1,\ldots,x_m$ are, fixed, nonrandom, real values and $Y_1,\ldots,Y_m$ are the corresponding binary response values. So, my model is $$\log(\frac{P(Y=1|x)}{1-P(Y=1|x)}) = \alpha + \beta x.$$

Using our data, we estimate $\alpha$ and $\beta$ to be $\hat{\alpha}$ and $\hat{\beta}$.

Now, someone wants to test my model. They bring in $n$ new values of $Y$, say, $Y^'_1,\ldots,Y^'_n$, all corresponding to a particular value of $x$, say $x = x_0$. They claim that if my model is correct, the proportion of $Y^'_i$'s that are equal to 1 must be close to $$\frac{1}{1+\exp(\hat{\alpha} + \hat{\beta} x_0)},$$ (say within a 95% confidence interval for this quantity, which we can obtain using the asymptotic normality of $\hat{\alpha}$ and $\hat{\beta}$ and their asymptotic covarriance matrix). But we know that even if the model is perfectly correct and $$P(Y =1 | x_0) = \frac{1}{\exp(\hat{\alpha} + \hat{\beta} x_0)},$$ the rate of convergence of $(Y^'_1+\cdots+Y^'_n)/n$ to $P(Y = 1|x_0)$ is of order $1/\sqrt{n}$ (please correct me if I'm wrong on this). Therefore, even a sample of size 100 (which would be considered quite large) isn't really large enough for the sample proportion to be close to its presumed expected value that the logistic regression model claims.

So here is my question: How do we really test the claim that the model is correct using this newly obtained data? I've looked around quite a bit, and the Bayesian approach seems to be to use simulation to create a prediction interval for the sample proportion of a sample of size $n$ (see for instance chapter 7 in Gelman and Hill's regression book), but I have't been able to find anything in the frequentist world to address this question. I'd very much appreciate any comments, suggestions, or pointers to the literature.

share|improve this question
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.