4
$\begingroup$

For simplicity's sake, let's suppose a binary classification problem, with a perfect 50% of probability for each of the classes, and a SkLearn's SVC model.

Let's assume that the classes in the vector Y cannot be predicted out of the values from the matrix X, because they are unrelated, but we don't know that. So we continue collecting observations, hoping that good predictions will be able to be made at some point.

While observations tend to infinite, should probability of the classes (obtained from predict_proba) converge towards that 50%?

If probabilities doesn't converge, does it mean that may be some form of correlation?

Any thoughts are well appreciated. Thanks in advance.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Are the classes balanced 50:50 in the sampling distribution? $\endgroup$
    – Galen
    Commented Sep 7, 2023 at 17:39
  • 2
    $\begingroup$ It is worth noting that "unrelated" and "uncorrelated" are not the same. There can be a relationship without that relationship being correlation. $\endgroup$
    – Dave
    Commented Sep 7, 2023 at 18:11
  • 2
    $\begingroup$ "Unrelated" seems to be an elliptical way of telling us $X$ is irrelevant, so let's ignore it. Your question therefore concerns what happens in the long run when you predict the results of random flips of a fair coin. The answer is given by various laws of large numbers as well as the Central Limit Theorem. $\endgroup$
    – whuber ♦
    Commented Sep 7, 2023 at 18:29
  • 1
    $\begingroup$ @whuber That's a good point! But remember that, at the beggnining, you don't konw if X is relevant or not. So, how to know? How predict proba would loook like if it's not? $\endgroup$
    – Juan Flautista De Torrepacheco
    Commented Sep 7, 2023 at 18:35
  • 1
    $\begingroup$ It doesn't matter what you know. All that matters is how $Y$ behaves. If you employ useless information, such as $X,$ then you might make poor predictions. But when $Y$ is independent of all your information and has equal chances of being in one of two states, it's a flip of a fair coin and so is the agreement between your prediction and the each outcome of $Y.$ $\endgroup$
    – whuber ♦
    Commented Sep 7, 2023 at 18:38

1 Answer 1

Reset to default
3
$\begingroup$

You can force an outstanding in-sample accuracy by overfitting to the training data. However, if the features are unrelated to the outcomes, that means the feature distributions are identical for both classes (otherwise, they would be related to the outcomes).

If the feature distributions are identical for the two categories, they you have no hope of reliably distinguishing between the two categories. Given a feature vector, you never have any insight into the category to which it belongs, beyond the fact that one category might generally be more common (which you've excluded as a possibility by saying the categories are represented equally).

You can be a bit more formal about this by using Bayes' theorem. Your model is predicting $P(Y=1\vert X=x)$ for some feature vector $x$, that is, the probability that $Y$ belongs to category $1$ given that the features are the $x$ that they are.

$$ P(Y=1\vert X=x) = \dfrac{ P(X = x\vert Y = 1)P(Y = 1) }{ P(X = x) } $$

When the distributions have identical distributions for categories $0$ and $1$, then $P(X = x\vert Y = 1) = P(X = x)$, and your predicted probability is just the overall probability of category $1$, which is $0.5$ in the balanced case. There is always a $50/50$ chance that the observation belongs to either category.

Improve this answer
$\endgroup$
10
  • 1
    $\begingroup$ Somehow I never get bored of Bayes' rule. (+1) $\endgroup$
    – Galen
    Commented Sep 7, 2023 at 17:52
  • $\begingroup$ Could you please add some information about predict_proba works? Then I'll mark your answer as accepted. Thanks! $\endgroup$
    – Juan Flautista De Torrepacheco
    Commented Sep 7, 2023 at 18:15
  • $\begingroup$ @JuanFlautistaDeTorrepacheco The predict_proba method returns probability predictions instead of hard categories, but you seem to know that. What else would you want to see about predict_proba? $\endgroup$
    – Dave
    Commented Sep 7, 2023 at 18:16
  • 1
    $\begingroup$ @JuanFlautistaDeTorrepacheco What does that have to do with predict_proba? $\endgroup$
    – Dave
    Commented Sep 7, 2023 at 18:49
  • 1
    $\begingroup$ @JuanFlautistaDeTorrepacheco The beauty of this is that it doesn’t depend on the model. What I wrote is true whether you use an SVM approach, neural networks, logistic regression, or anything else. // However, nothing guarantees that the in-sample predictions are going to score a particular way. If you’re not careful, you will wind up overfitting your model to the training data. $\endgroup$
    – Dave
    Commented Sep 8, 2023 at 10:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.