Binary classification and target-label proportion

Question

Suppose that we have a binary classification problem with a vector y = [1 1 0 1 0 0 1 ... 0] having the proportion: prop = (# of ones)/(# of samples) = 0.7. Now, suppose that I fit any classifier clf and use it to predict the probability of every sample of the data to be classified as 1. Would the average probability coincide with the proportions of ones (i.e., would the average probability be 0.7)? Why?

Thank you

Answer 1

The obvious case is when your classifier returns zeros and ones and has perfect accuracy, then the predictions and target values are the same $y_i = \hat{y}_i \;\forall\, i$, then also $\tfrac{\sum_{i=1}^n y_i}{n} = \tfrac{\sum_{i=1}^n \hat{y}_i}{n}$. When it woundn't have perfect accuracy, this won't have to be the case.

We say that estimator is unbiased, when on average the predictions are equal to predicted values $E[\hat{\theta}_n] = \theta$. If you are aiming at predicting the probabilities ($\theta$) rather then the labels ($y$), then probabilities on average equal to empirical proportion is in fact statement about bias of the estimator.

If the classifier returns probability-like scores, it is also related to probabilities being well-calibrated. If they are, then they adequately predict individual probabilities. When talking about averaging the predicted probabilities $\hat{p}(y|\mathbf{X})$, you most likely mean averaging them given the features $\mathbf{X}$ distributed as $f$:

$$ E_{\mathbf{x} \sim f}\big[\,\hat{p}(y|\mathbf{X})\,\big] = \sum_i \,\hat{p}(y|\mathbf{X}_i) \;f(\mathbf{X}_i) $$

Notice that in case of perfectly calibrated probabilities, i.e. $p(y|\mathbf{X}) = \hat{p}(y|\mathbf{X})$, by the law of total probability we obtain

$$ \sum_i \,\hat{p}(y|\mathbf{X}_i) \;f(\mathbf{X}_i) = p(y) $$

We want the predicted probabilities to be consistent with the true probabilities (and so, with empirical probabilities), that is why plotting predicted probabilities vs binned probabilities is a popular diagnostic plot for predictive models.

See also Properties of logistic regressions and Should predicted probabilities from Logistic Regression correspond with percentages?.