4
$\begingroup$

Studying machine learning, I've made it to the point where I've exponentiated my L2 Regularization loss function. We began with a simple ordinary least squares loss function, and added a penalty term proportional to the squared weights of the coefficients, as seen below:

starting loss function

Because minimizing the least squared loss function is equal minimizing the negative log-likelihood, we flipped the signs so that maximizing the negative loss function is equal to maximizing the log likelihood. Then we've exponentiated to get rid of the log function, yielding the $\exp{\{-J\}} $term below.

exponentiation

Now I'm told that these represent two gaussians. I have the following two expressions:

likelihood

I know that $J = $ ($-$ log likelihood), thus $ -J =$ (log likelihood), thus $\exp{\{-J\}} = $ likelihood. What I'm confused by is how do the expressions in the first image represent the gaussians below? Or rather, why do/can I add $\frac{1}{2\sigma^2}$ inside the exponentiation and normalize by the constant? I'm missing the connection between them.

Side note: as I was following my material I thought I had the connection, but I got bogged down in some computations and believe I lost sight of the connection here.

Note: This question follows from my other post in which I was trying to prove what the instructor states, which is that the second expression in the $\exp{\{-J\}}$ term represents a Gaussian with $\mu = 0$ and $\sigma^2 = \frac{1}{\lambda}$, but I was getting a different answer. I can prove this about the Prior probability at the bottom, but the instructor said it referring to the second expression in $\exp{\{-J\}}$

$\endgroup$
5
  • 1
    $\begingroup$ I'm completely guessing here - there's not enough context for me to know if this is 100% correct - but it looks like you're trying to find the posterior distribution of $X$ given $Y$? One of the things that you'll get used to over time is that when you are trying to find posterior distributions, it's usually easier to just discard all proportionality constants. What you care about is the essence of the PDF: what is the form of the PDF of the posterior, disregarding the constants - because hopefully the essence of the PDF of the posterior looks like a familiar distribution. $\endgroup$ Jul 10, 2018 at 12:31
  • $\begingroup$ @Clarinetist thank you for your reply. You are probably right but I will attempt to provide more context in the question to be certain. $\endgroup$
    – Hanzy
    Jul 10, 2018 at 12:32
  • $\begingroup$ In other words, you don't need to worry about the $\lambda$, $\sigma^2$, or any of those proportionality constants. Because when you're trying to find the posterior PDF, you will hopefully get a PDF that looks familiar. $\endgroup$ Jul 10, 2018 at 12:32
  • $\begingroup$ @Clarinetist I've added more context, but not sure if what I've added is helpful. But your comment does seem informative regardless, so thank you. $\endgroup$
    – Hanzy
    Jul 10, 2018 at 12:40
  • $\begingroup$ This link - if I'm reading your question correctly - is essentially solving the problem, but it's a lot more dense with the matrix notation. Notice that I used $\propto$ symbols throughout this derivation - so I'm ignoring the constants. $\endgroup$ Jul 10, 2018 at 12:43

1 Answer 1

3
$\begingroup$

These expressions, for the posterior and the (exponentiated) loss function, are not mathematically identical, but in this case they are computationally equivalent, since you are only interested in finding the maximum of the posterior (or minimum of the loss function). Because it doesn't matter whether you find the maximum of $f(x)$ or of $a\times f(x)$: both functions have their maximum at exactly the same value of $x$ (or one will have a minimum, if $a$ is negative), and you typically don't care how high that maximum is.

This also means that if you only know $a\times f(x)$ (i.e. you only know your target function up to a constant of multiplication), that's fine. You don't need to worry about getting the scaling constant right.

This explains why, for the purposes of optimization w.r.t. $w$, we can ignore the normalization constants $\frac{1}{\sqrt{2\pi\sigma^2}}$ and $\sqrt{\frac{\lambda}{2\pi}}$, and so they do not appear in the loss function (because why compute them if they don't matter), but they do appear in the strict mathematical definition of a likelihood and a prior.

That leaves one more discrepancy, which is the appearance of $\frac{1}{2\pi\sigma^2}$ in the likelihood (but not in the loss function) and of $\frac{\lambda}{2}$ in the prior (vs. $\lambda$ in the loss). This has a similar reason, because these values are weights that determine the importance of the prior and of the data points in the likelihood. But you only care about the relative importance of these things w.r.t. one another. And since $\sigma$ is constant for all data points, that means the only relevant parameter that we're left with is the strength of the prior, relative to the likelihood, which we can describe with a single scalar $\lambda$.

So in summary, the loss function retains only the relevant bits of the prior and likelihood, which is why you cannot fully reconstruct the prior and likelihood from the loss. It actually makes more sense to go in the other direction: take the log of the posterior (i.e. the product of likelihood $\times$ prior) and then drop all terms that you don't need for the optimization problem, and you should end up with the loss function you're familiar with.

$\endgroup$
2
  • 1
    $\begingroup$ Thank you, you really have touched upon why I was confused and this answer is excellent. Without too much follow up, should I recognize $ \exp{\{-x^2\}}$ (or any similar variation) as a non-normalized Gaussian always? That is the impression I get, which seems supported by this link. $\endgroup$
    – Hanzy
    Jul 10, 2018 at 13:44
  • $\begingroup$ $\exp(-x^2)$ is a Gaussian function, so yeah in statistical contexts anything with that general form is likely to derive from a Gaussian probability distribution. $\endgroup$ Jul 11, 2018 at 8:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.