Validating a logistic regression for a specific $x$

Question

I have a logistic regression model for 0/1 binary response data that is built from samples $(x_1,Y_1),\dots,(x_m,Y_m)$, where $x_1,\dots,x_m$ are, fixed, nonrandom, real values and $Y_1,\dots,Y_m$ are the corresponding binary response values. So, my model is: $$\log\bigg(\frac{\text{P}(Y=1|x)}{1-\text{P}(Y=1|x)}\bigg) = \alpha + \beta x.$$

Using our data, we estimate $\alpha$ and $\beta$ to be $\hat{\alpha}$ and $\hat{\beta}$.

Now, someone wants to test my model. They bring in $n$ new values of $Y$, say, $\hat Y'_1,...,\hat Y'_n$, all corresponding to a particular value of $x$, say $x = x_0$. They claim that if my model is correct, the proportion of $\hat Y'_i$'s that are equal to 1 must be close to $$\frac{1}{1+\exp(\hat{\alpha} + \hat{\beta} x_0)},$$ (say within a 95% confidence interval for this quantity, which we can obtain using the asymptotic normality of $\hat{\alpha}$ and $\hat{\beta}$ and their asymptotic covarriance matrix). But we know that even if the model is perfectly correct and $$\text{P}(Y = 1 | x_0) = \frac{1}{\exp(\hat{\alpha} + \hat{\beta} x_0)},$$ the rate of convergence of $(\hat Y'_1+\ldots+\hat Y'_n)/n$ to $\text{P}(Y = 1|x_0)$ is of order $1/\sqrt{n}$ (please correct me if I'm wrong on this). Therefore, even a sample of size 100 (which would be considered quite large) isn't really large enough for the sample proportion to be close to its presumed expected value that the logistic regression model claims.

How do we really test the claim that the model is correct using this newly obtained data? I've looked around quite a bit, and the Bayesian approach seems to be to use simulation to create a prediction interval for the sample proportion of a sample of size $n$ (see for instance chapter 7 in Gelman and Hill's regression book), but I haven't been able to find anything in the frequentist world to address this question. I'd very much appreciate any comments, suggestions, or pointers to the literature.

Answer 1

By its construction the logistic regression models predicts probabilities,

$$\hat P(Y =1 \mid X=x_0) = \frac{1}{1+\exp{\{\hat{\alpha} + \hat{\beta} x_0\}}}$$

The proportion of $Y=1$ in the test sample of size $n$, with all $x$'s equal, is a different estimator of the same conditional probability, denote it $\hat p_{1|x_0}$.

Why the performance of the logistic regression model should be judged against $\hat p_{1|x_0}$? What $\hat p_{1|x_0}$ has in favor of it, so as to function as a test for the validity/adequacy of the logistic regression approach?

Well, one could argue that it is non-parametric and even "non-distributional" since it directly estimates a theoretical (conditional) moment from the sample using a method-of moments principle, without making the additional assumptions the logistic regression model does -and each assumption may be a source of misspecification...

These are valid points, but test the validity of a model based on how does it predict a single point of the theoretical distribution?

No, that's mistaken. Assume that we knew the true probabilities. Then, it wouldn't tell us much if the estimated probability from logistic regression was "away from" or "close to" a single true probability.

The inappropriateness of using a single probability increases when we want to pit our model against another estimate, and not the actual probability. As the OP states, with small sample sizes, the accuracy of $\hat p_{1|x_0}$ cannot be guaranteed, irrespective of its "robustness" to misspecification that may haunt the logistic regression model.

We should be able to compare how our model does by predicting many different probabilities, not just one. We should form the estimated probability distribution and then measure the distance of it from the true one, by some suitable metric.

With large test sample sizes, each based on a given value for the regressor, then, indeed we could more validly evaluate the predictions of our model against the many $\hat p_{1|x_j}$ estimated probabilities.