Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I have a logistic regression model for 0/1 binary response data that is built from samples $(x_1,Y_1),\dots,(x_m,Y_m)$, where $x_1,\dots,x_m$ are, fixed, nonrandom, real values and $Y_1,\dots,Y_m$ are the corresponding binary response values. So, my model is: $$\log\bigg(\frac{\text{P}(Y=1|x)}{1-\text{P}(Y=1|x)}\bigg) = \alpha + \beta x.$$

Using our data, we estimate $\alpha$ and $\beta$ to be $\hat{\alpha}$ and $\hat{\beta}$.

Now, someone wants to test my model. They bring in $n$ new values of $Y$, say, $\hat Y'_1,...,\hat Y'_n$, all corresponding to a particular value of $x$, say $x = x_0$. They claim that if my model is correct, the proportion of $\hat Y'_i$'s that are equal to 1 must be close to $$\frac{1}{1+\exp(\hat{\alpha} + \hat{\beta} x_0)},$$ (say within a 95% confidence interval for this quantity, which we can obtain using the asymptotic normality of $\hat{\alpha}$ and $\hat{\beta}$ and their asymptotic covarriance matrix). But we know that even if the model is perfectly correct and $$\text{P}(Y = 1 | x_0) = \frac{1}{\exp(\hat{\alpha} + \hat{\beta} x_0)},$$ the rate of convergence of $(\hat Y'_1+\ldots+\hat Y'_n)/n$ to $\text{P}(Y = 1|x_0)$ is of order $1/\sqrt{n}$ (please correct me if I'm wrong on this). Therefore, even a sample of size 100 (which would be considered quite large) isn't really large enough for the sample proportion to be close to its presumed expected value that the logistic regression model claims.

How do we really test the claim that the model is correct using this newly obtained data? I've looked around quite a bit, and the Bayesian approach seems to be to use simulation to create a prediction interval for the sample proportion of a sample of size $n$ (see for instance chapter 7 in Gelman and Hill's regression book), but I haven't been able to find anything in the frequentist world to address this question. I'd very much appreciate any comments, suggestions, or pointers to the literature.

share|improve this question

1 Answer 1

By its construction the logistic regression models predicts probabilities,

$$\hat P(Y =1 \mid X=x_0) = \frac{1}{1+\exp{\{\hat{\alpha} + \hat{\beta} x_0\}}}$$

The proportion of $Y=1$ in the test sample of size $n$, with all $x$'s equal, is a different estimator of the same conditional probability, denote it $\hat p_{1|x_0}$.

Why the performance of the logistic regression model should be judged against $\hat p_{1|x_0}$? What $\hat p_{1|x_0}$ has in favor of it, so as to function as a test for the validity/adequacy of the logistic regression approach?

Well, one could argue that it is non-parametric and even "non-distributional" since it directly estimates a theoretical (conditional) moment from the sample using a method-of moments principle, without making the additional assumptions the logistic regression model does -and each assumption may be a source of misspecification...

These are valid points, but test the validity of a model based on how does it predict a single point of the theoretical distribution?

No, that's mistaken. Assume that we knew the true probabilities. Then, it wouldn't tell us much if the estimated probability from logistic regression was "away from" or "close to" a single true probability.

The inappropriateness of using a single probability increases when we want to pit our model against another estimate, and not the actual probability. As the OP states, with small sample sizes, the accuracy of $\hat p_{1|x_0}$ cannot be guaranteed, irrespective of its "robustness" to misspecification that may haunt the logistic regression model.

We should be able to compare how our model does by predicting many different probabilities, not just one. We should form the estimated probability distribution and then measure the distance of it from the true one, by some suitable metric.

With large test sample sizes, each based on a given value for the regressor, then, indeed we could more validly evaluate the predictions of our model against the many $\hat p_{1|x_j}$ estimated probabilities.

share|improve this answer
    
Interesting thoughts, but there is a difference between the two. By law of large numbers, we know that $\hat{p}_{1|x_0}$ converges to the true probability. However, for logistic regression, there is no such result. –  user765195 Aug 21 at 4:37
    
Are you saying that the maximum likelihood estimator is inconsistent in the context of logistic regression? –  Alecos Papadopoulos Aug 21 at 4:42

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.