# How does this likelihood function log transformation work?

Suppose I have the following Bayesian inference problem:

$$\underset{\{{\theta_i\},\{B^j}\}}{\operatorname{argmax}} P(\{\theta_i\},\{B^j_{il}\}|\{v_{jl}\}) \propto P(\{v_{jl}\}|\{\theta_i\},\{B^j_{il}\}) P(\{B^j_{il}\}) P(\{\theta_i\}) \tag1 \label{eq1}$$

and suppose that given $$\{B^j\}$$ we have the MAP estimate on $$\{\theta_i\}$$:

$$\hat{\theta_i} = \frac{\sum_{jl} B^i_{jl} v_{jl}/\sigma_j^2}{1/\sigma_0^2 + \sum_{jl}B^j_{il}/\sigma_j^2} \text{ for } i = 1, \dots, L.$$

In the paper Bayesian Nonparametric Federated Learning of Neural Networks (2020), the authors then say: " We can now cast [the] optimization [problem] [..] with respect to only $$\{B_j\}$$ [..] Taking natural logarithm we obtain:

$$-\frac{1}{2}\sum_i(\frac{\|\hat{\theta_i}\|^2}{\sigma_0^2} + D\log(2\pi\sigma_0^2) + \sum_{jl} B_{il}^j \frac{\|v_{jl} - \hat{\theta_i}\|^2}{\sigma_j^2}) + \log (P(\{B^j\}) \tag2 \label{eq2}$$

Can someone help me understand how the different terms in \eqref{eq2} are found? Is there a simple formula to take the likelihood expressed in \eqref{eq1} into \eqref{eq2}? (Knowing that the $$\{v_{jl}\}$$ are assumed to be normally distributed)

EDIT: So indeed as @FabianWerner said, the second part of equation (2) $$-\frac{1}{2}\ D\log(2\pi\sigma_0^2) + \sum_{jl} B_{il}^j \frac{\|v_{jl} - \hat{\theta_i}\|^2}{\sigma_j^2}$$ simply comes from writing out the pdf of the normally distributed variable $$v_{jl}$$ and applying common log transformations. I still haven't figured why the first term and the sum over i $$\sum_i(\frac{\|\hat{\theta_i}\|^2}{\sigma_0^2} ..)$$ is here though..

• I have the feeling that it is just using the natural rules for log like log(ab) = log(a) + log(b) and log(a/b) = log(a) - log(b). For example, $\log(...\cdot P(B^j)) = \log(...) + \log(P(B^j))$. That is how the last summand is derived. The rest should really just be those two rules above and filling in the definitions for the other $P$-terms... Jan 27, 2021 at 10:51
• Yes, for the sum I agree. But I don't see where the (-1/2) or the terms inside the parenthesis come from (in particular the norms?) Jan 27, 2021 at 11:02
• Humm... I don't know anything about this particular paper but let's start with the next simple term: $P(\{\Theta_i\}$. What is the definition of that term? According to what they claim it should be something like $\prod_i e^{|\Theta_i|/\sigma_0^2}$... that essentially means that the $|\Theta_i|$ are somewhat normally distributed... My guess (unfortunately, it is a very bad habit of mathematicians to mystify their work by not writing down explicitly what they are doing and by not using references appropriately) is that is goes like this: Jan 27, 2021 at 11:23

Sorry, this is more of a comment than an answer but it didn't fit in the comment field...

In (2) they say that the $$v_{jl}$$ (with respect to something else I don't quite understand without getting into the details of this paper) are normally distributed. Then they say that they assume that the covariance matrices $$\Sigma$$ are just something like a*diag(1,1,...,1). Let's say that we have a random variable $$X\sim N(\mu, \Sigma)$$ then

$$p(x) = \text{const} ~~ e^{-\frac{1}{2} (x-\mu)^T \Sigma^{-1} (x-\mu)}$$

Now let's assume that $$\mu = 0$$ and $$\Sigma = a \cdot \text{diag}(1,1,...)$$ then the expression above simplifies:

$$p(x) = \text{const} \cdot e^{-\frac{1}{2a} x^Tx}$$

If $$x = (x_1, ..., x_d)$$ what is $$x^Tx$$? It is nothing else than $$x_1^2 + ... + x_d^2 = |x|^2$$. Hence,

$$\log(p(x)) = \log(\text{const}) + \left(-\frac{1}{2} \frac{1}{a} |x|^2 \right)$$

which looks pretty much like the expression that they have in the logarithm form of the likelihood...

So I guess that $$P(\{v_{jl}\} | ...)$$ becomes this term above and for some weird reason (I am by far not an expert in this MAP business) this MAP step allows them to drop the term $$P(\{\theta_i\})$$. Maybe after substituting this by an approximation is becomes constant and therefore is irrelevant for the optimization process or something like that...

• Yep I guess it's something along those lines. I don't see why you would assume that $\mu$ = 0 in your example? Jan 27, 2021 at 12:38
• Otherwise you will end up with $|\theta - \text{something}|^2$ instead of just $|\theta|^2$... Jan 27, 2021 at 12:43
• yes I see.. I'll try to see if I can write it out given your hints, thanks (sorry not enough rep points to vote up) Jan 27, 2021 at 12:48
• @user1360448 No worries and good luck :-) Yes, please share what you figured out so that others can also benefit from it. Jan 27, 2021 at 15:13