1
$\begingroup$

Suppose I have the following Bayesian inference problem:

$$\underset{\{{\theta_i\},\{B^j}\}}{\operatorname{argmax}} P(\{\theta_i\},\{B^j_{il}\}|\{v_{jl}\}) \propto P(\{v_{jl}\}|\{\theta_i\},\{B^j_{il}\}) P(\{B^j_{il}\}) P(\{\theta_i\}) \tag1 \label{eq1}$$

and suppose that given $\{B^j\}$ we have the MAP estimate on $\{\theta_i\}$:

$$\hat{\theta_i} = \frac{\sum_{jl} B^i_{jl} v_{jl}/\sigma_j^2}{1/\sigma_0^2 + \sum_{jl}B^j_{il}/\sigma_j^2} \text{ for } i = 1, \dots, L.$$

In the paper Bayesian Nonparametric Federated Learning of Neural Networks (2020), the authors then say: " We can now cast [the] optimization [problem] [..] with respect to only $\{B_j\}$ [..] Taking natural logarithm we obtain:

$$-\frac{1}{2}\sum_i(\frac{\|\hat{\theta_i}\|^2}{\sigma_0^2} + D\log(2\pi\sigma_0^2) + \sum_{jl} B_{il}^j \frac{\|v_{jl} - \hat{\theta_i}\|^2}{\sigma_j^2}) + \log (P(\{B^j\}) \tag2 \label{eq2}$$

Can someone help me understand how the different terms in \eqref{eq2} are found? Is there a simple formula to take the likelihood expressed in \eqref{eq1} into \eqref{eq2}? (Knowing that the $\{v_{jl}\}$ are assumed to be normally distributed)

EDIT: So indeed as @FabianWerner said, the second part of equation (2) $-\frac{1}{2}\ D\log(2\pi\sigma_0^2) + \sum_{jl} B_{il}^j \frac{\|v_{jl} - \hat{\theta_i}\|^2}{\sigma_j^2}$ simply comes from writing out the pdf of the normally distributed variable $v_{jl}$ and applying common log transformations. I still haven't figured why the first term and the sum over i $\sum_i(\frac{\|\hat{\theta_i}\|^2}{\sigma_0^2} ..)$ is here though..

$\endgroup$
3
  • $\begingroup$ I have the feeling that it is just using the natural rules for log like log(ab) = log(a) + log(b) and log(a/b) = log(a) - log(b). For example, $\log(...\cdot P(B^j)) = \log(...) + \log(P(B^j))$. That is how the last summand is derived. The rest should really just be those two rules above and filling in the definitions for the other $P$-terms... $\endgroup$ Jan 27, 2021 at 10:51
  • $\begingroup$ Yes, for the sum I agree. But I don't see where the (-1/2) or the terms inside the parenthesis come from (in particular the norms?) $\endgroup$ Jan 27, 2021 at 11:02
  • $\begingroup$ Humm... I don't know anything about this particular paper but let's start with the next simple term: $P(\{\Theta_i\}$. What is the definition of that term? According to what they claim it should be something like $\prod_i e^{|\Theta_i|/\sigma_0^2}$... that essentially means that the $|\Theta_i|$ are somewhat normally distributed... My guess (unfortunately, it is a very bad habit of mathematicians to mystify their work by not writing down explicitly what they are doing and by not using references appropriately) is that is goes like this: $\endgroup$ Jan 27, 2021 at 11:23

1 Answer 1

0
$\begingroup$

Sorry, this is more of a comment than an answer but it didn't fit in the comment field...

In (2) they say that the $v_{jl}$ (with respect to something else I don't quite understand without getting into the details of this paper) are normally distributed. Then they say that they assume that the covariance matrices $\Sigma$ are just something like a*diag(1,1,...,1). Let's say that we have a random variable $X\sim N(\mu, \Sigma)$ then

$$p(x) = \text{const} ~~ e^{-\frac{1}{2} (x-\mu)^T \Sigma^{-1} (x-\mu)}$$

Now let's assume that $\mu = 0$ and $\Sigma = a \cdot \text{diag}(1,1,...)$ then the expression above simplifies:

$$p(x) = \text{const} \cdot e^{-\frac{1}{2a} x^Tx}$$

If $x = (x_1, ..., x_d)$ what is $x^Tx$? It is nothing else than $x_1^2 + ... + x_d^2 = |x|^2$. Hence,

$$\log(p(x)) = \log(\text{const}) + \left(-\frac{1}{2} \frac{1}{a} |x|^2 \right)$$

which looks pretty much like the expression that they have in the logarithm form of the likelihood...

So I guess that $P(\{v_{jl}\} | ...)$ becomes this term above and for some weird reason (I am by far not an expert in this MAP business) this MAP step allows them to drop the term $P(\{\theta_i\})$. Maybe after substituting this by an approximation is becomes constant and therefore is irrelevant for the optimization process or something like that...

$\endgroup$
4
  • $\begingroup$ Yep I guess it's something along those lines. I don't see why you would assume that $\mu$ = 0 in your example? $\endgroup$ Jan 27, 2021 at 12:38
  • $\begingroup$ Otherwise you will end up with $|\theta - \text{something}|^2$ instead of just $|\theta|^2$... $\endgroup$ Jan 27, 2021 at 12:43
  • $\begingroup$ yes I see.. I'll try to see if I can write it out given your hints, thanks (sorry not enough rep points to vote up) $\endgroup$ Jan 27, 2021 at 12:48
  • $\begingroup$ @user1360448 No worries and good luck :-) Yes, please share what you figured out so that others can also benefit from it. $\endgroup$ Jan 27, 2021 at 15:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.