# Correct conditional expectation via logistic regression but terrible AUC

Suppose you have a binary random variable $$Y$$, and several other random variables $$X_1,...,X_p$$. Your goal is to "predict $$Y$$ using $$X_1,...,X_p$$." So, you go ahead and fit logistic regression, which is trying to estimate $$P(Y=1 | X_1,...,X_p)$$. However, when you evaluate the results of your logistic regression, you find that the AUC is terrible.

One possibility is that your model is "bad," i.e. your estimates of the coefficients for $$X_1,...,X_p$$ are incorrect. Perhaps you did not have enough samples, or the model was specified incorrectly etc. However, it also occurs to me that your model could be "perfect," but simply that the $$X_1,...,X_p$$ are not "very predictive."

By this, I mean that suppose $$P(Y=1)=0.1$$, but when you condition on the random variables $$X_1,...,X_p$$, the (true) probability does not change very much. For example, $$0.05\leq P(Y=1|X_1=x_1,...,X_p=x_p) \leq 0.2$$

for all $$x_1,...,x_p$$. In other words, $$Y$$ is almost independent of $$X_1,...,X_p$$. So, even if the logistic regression model perfectly learns the conditional expectation, the AUC could still be very bad!

Is there a name for this situation? And is there a way to detect it? How can I tell that my conditional expectations are "correct," even though my predictive ability is so poor? Or am I not even phrasing the problem correctly statistically?

• It does not follow from $$0.05\leq P(Y=1|X_1=x_1,...,X_p=x_p) \leq 0.2$$ that the AUC must be bad. If all $Y=1$ have higher $P(Y=1|X)$ than all $Y=0$, the AUC is 1. Another way to state this is that if all the positives are ranked higher than all of the negatives, the AUC is 1. See: stats.stackexchange.com/questions/145566/… – Sycorax Apr 30 '19 at 15:01
• @Sycorax: do you want to post your comment(s) as an answer? Better to have a short answer than no answer at all. Anyone who has a better answer can post it. – Stephan Kolassa Apr 30 '19 at 15:08
• It's not clear to me that my comment rises to the level of an answer because the main part of the question seems concerned with naming and resolving a certain kind of deficiency in logistic regression. If the reason is that "$Y$ can't be effectively modeled with $X$" perhaps this is a duplicate of stats.stackexchange.com/questions/222179/… but otherwise answers will turn on how to make a good predictive model using logistic regression (and whatever the name for this phenomenon is). – Sycorax Apr 30 '19 at 15:12
• @Sycorax, I think you are right...my example is bad. Here is a better one, I think. Let $p=1,$ let $X$ be binary. Further, suppose that $P(Y=1|X=0) = 0.1$ and $P(Y=1|X=1)=0.2$. Now, suppose that after fitting on my training data, my logistic regression function $f(X)$ is "accurate," in that $f(0) = 0.1$ and $f(1) = 0.2$. I claim that you cannot have a perfect AUC on the test set. You would only get two possible "predictions," 0.1 and 0.2, and no threshold you set would be perfectly accurate. Clearly, not all $Y=1$ have higher $P(Y=1|X)$ than $Y=0$. – The_Anomaly Apr 30 '19 at 16:32
• @The_Anomaly in that contrived example you are correct, an AUC is bounded by a certain theoretical value and analyses may "beat it" only by accident. It's an artifact of probabilistic teaching that we actually believe in "completely random" outcomes. Except in quantum physics, inherent randomness is not really believed to be a thing. You could predict the outcome of a coin flip knowing the torque, velocity, angle, air pressure, surface tension, etc, but instruments are too imperfect to catch all that. – AdamO Apr 30 '19 at 16:55

One must first ask if it is a "problem" at all. My viewpoint is that of a chaotician. I would look at the set of $$X_1, \ldots, X_p$$ and say that set of regressors is too small, and consequently my $$n$$ is too small too. If I needed to have an AUC of 1, I would get it by having an infinite set of regressors and of samples, and I could deterministically predict the outcome. Just because all of $$X_1, \ldots, X_p$$ predictors are statistically significant in the logistic model does not mean they comprise a complete set of predcitors (or even a good set at that). You likely (always) only have a subset of the ones you need, and there's always room for more.