0
$\begingroup$

I have read a lot of stuff on the relation between minimizing a loss function / maximizing the likelihood / choose a centrality measure of the posterior (Bayesian estimation); but I cannot see a clear connection, in one point specifically.

Suppose I have a system taking input $x$ and giving output $y$ (I have in mind a typical supervised learning problem). I assume the system is described by the following equations $$ y = f_w(x) + \varepsilon\ ,$$ where $\varepsilon$ are iid r.v.'s.

Let us focus on regression, namely $y \in \mathbb{R}$.

Suppose that we have access only to some observations of the r.v.'s $x$ $y$, namely: $$ \mathcal{T} = \{(x_i, y_i)_{i=1,\ldots,N}\}\ ;$$ a.k.a. the training set.

The "frequentist" approach is to fix parameters $w$ by maximizing the loglikelihood, i.e. $$ \log P(y_{1:N} | w, x_{1:N}) = \sum_i \log P(y_i | w, x_i)\ .$$ It is very easy to see that if $\varepsilon$ is gaussian (and thus the likelihood is gaussian as well), then maximizing the likelihood is equivalent to minimize the empirical quadratic loss: $$ \sum_i (y_i - f_w(x_i))^2\ .$$ Notice that the empirical loss minimization is actually only an approximation to what one would really want to do in the loss-minimization approach, namely to minimize $$ E(\text{loss}(y, f_w(x)))\ ,$$ which reduces to the previous one when replacing the data distribution $p(x,y)$ to the empirical distribution given $\mathcal{T}$.

A very similar thing can be done in the classification case, with Bernoulli distributions and cross-entropy loss.

Now with Bayes.

In the Bayes setting, we find the posterior distribution (of the parameters given the observations) $$ P(w | \mathcal{T}) $$ and, in principle, one may use that posterior in anything involving the estimate $f_w(x)$, without ever fixing a specific $w$.
However, suppose I want to use the loss minimization approach, and to see it "in the Bayesian setting". The idea would be to minimize the expected loss. But, in the Bayesian setting, the expected loss is $$ E(\text{loss}(y, f_w(x))) \approx E_{w | \mathcal{T}} E_{x,y | w}\text{loss}(y, f_w(x)) = \int \text{loss}(y, f_w(x)) p(x,y | w) p(w | \mathcal{T})dxdydw\ (*).$$

Ok, now I am eventually able to state my problem:

we know that MAP bayesian estimate (with uniform prior) is equivalent to MLE, and we know that MLE is equivalent to quadratic loss minimization; but how can I relate MAP directly to quadratic loss from (*)?

I mean: I am not able to relate loss minimization to Bayesian mode/median/mean point estimates

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.