I'm reading Understanding Machine Learning by Shai and Shai. In it, the Bayes Optimal Predictor is defined as

$$f_{\mathcal{D}}(x) = \mathbb{1}[\mathbb{P}[y = 1 | x] \geq 1/2]$$

Where $\mathcal{D}$ is any probability distribution over $\mathcal{X} \times \{0,1\}$.

This seems to be true of all binary classifiers. Can this be applied to other classes of tasks (e.g., regression)?

Intuitively it would make sense for the regression case to be

$$f_{\mathcal{D}}(x) = c\cdot\mathbb{1}[\mathbb{P}[y = c | x] \geq 1/2],$$

but I'm not sure this is true.


The model that you proposed wouldn't work for a continuous random variable because the probability that it takes any particular value is exactly 0. To be more precise, for all $c \in \mathbb R$ $\mathbb P(Y = c) = 0$.

But by using densities we can do something that's similar in spirit. What you really are looking for is to predict $Y$ with the value that it's most likely to take, and certainly some regions of $\mathbb R$ are more likely than others. This is exactly what the density gets at.

Now for some random variable $Y$ let's propose the following rule: predict $Y$ with $c$ where $c = \textrm{argmax}_{z \in \mathbb R} f_Y(z)$.

What happens if $Y \sim Bern(p)$? Well, the density of $Y$ is now $f_Y(z) = p^z (1-p)^{1-z}$ so this exactly coincides with the discrete rule that you proposed. So this density maximization (i.e. maximum likelihood) method is a generalization of this discrete decision rule.

Now let's see how linear regression fits in this framework. The standard model is $Y = X\beta + \varepsilon$ with $\varepsilon \sim \mathcal N(0, \sigma^2 I_n)$. This means that we are assuming that the $Y_i$ are independent and $Y_i \sim \mathcal N(X_i^T \beta, \sigma^2)$. The likelihood of $Y$ is therefore $$ f_Y(\vec y) = \frac{1}{(2\pi\sigma^2)^{n/2}}\exp{\left( \frac{-1}{2\sigma^2} \sum_{i=1}^n (y_i - X_i^T\beta)^2\right)} $$

and we want to choose $\beta$ that maximizes this. Note that by taking the log this is exactly the same as minimizing $\sum_i (y_i - X_i^T \beta)$, i.e. the RSS.

So the point of this is that it's quite a reasonable thing to maximize the density, and that this is exactly what OLS does.

One caveat: you don't want to blindly maximize the density because then you'll overfit. If we take $Y_i \sim \mathcal N(\mu_i, \sigma^2)$ and want to estimate all $n$ $\mu_i$, we'll maximize the likelihood by setting $\hat \mu_i = y_i$. This is the most likely model possible! But it's also complete garbage.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.