# Understanding Loss functions in Stacked Capsule Autoencoders

I was reading Stacked Capsule Autoencoder paper published by Geoff Hinton's group last year in NIPS. While reading section 2.1 about constellation autoencoders I couldn't understand how the expression of parts likelihood is calculated. Likelihood expression given in the paper is as follows

$$p(x_{1:M}) = \prod_{m=1}^{M} \sum_{k=1}^{K} \sum_{n=1}^{N} \frac{a_k a_{k,n}}{\sum_{i} a_i \sum_{j} a_{i,j}}p(x_m|k,n)$$

The way I understood it is that it's Gaussian Mixture model. $$a_k$$ represents the presence probability of $$k^{th}$$ capsule, $$a_{k,n}$$ represents the conditional presence probability of $$n^{th}$$ candidate part {$$n^{th}$$ Gaussian class} in $$k^{th}$$ object capsule and $$p(x_m | k,n)$$ represents the probability that $$x_m$$ part capsule belongs to $$n^{th}$$ Gaussian class {$$n^{th}$$ candidate part} of $$k^{th}$$ object capsule. So to find the $$p(x_1)$$ over 'k' object capsules, each having 'n' candidate predictions we marginalize $$p(x_1)$$ over 'k' and 'n' . So shouldn't the likelihood expression be

$$p(x_{1:M}) = \prod_{m=1}^{M} \sum_{k=1}^{K} \sum_{n=1}^{N} a_k a_{k,n} \space p(x_m|k,n)$$ From where does $$\sum_{i} a_i \sum_{j} a_{i,j}$$ is coming in the denominator of expression given in the paper?

• I also noticed this similarity at first but this is not making sense to me. If I write in simpler terms for 2 object capsules ${C_1,C_2}$ each giving two Gaussian classes ${G_1,G_2}$, then it's $P(G_1 | C_1)P(C_1) + P(G_2 | C_1)P(C_1) + P(G_1 | C_2)P(C_2)+ P(G_2 | C_2)P(C_2) = P(G_1 , C_1) + P(G_2 , C_1) + P(G_1 , C_2) +P(G_2 , C_2) = P(G_1) + P(G_2) = P(C_1) + P(C_2) = 1$. – user_3pij Jun 11 '20 at 4:03
• why do you think $P(C_1)+P(C_2)=1$? it's not obvious to me – Aksakal Jun 11 '20 at 4:18
• Yes you are right it's not necessarily 1 (I'll edit it), but except that the expression $P(C_1) + P(C_2)$ is not making sense to me. Why is it there? – user_3pij Jun 11 '20 at 4:29
• I understand $a_k a_{k,n}$ that's why I wrote the equation which I derived. What I don't get is the denominator part. Because all I need to calculate $P(x_m)$ is $a_k a_{k,n} P(X_m|n,k)$ specifically $P(x_m) = a_k a_{k,n} P(X_m|n,k)$. How does it $P(x_m) = \frac{a_k a_{k,n}}{\sum_{i} a_i \sum_{j} a_{i,j}} P(X_m|n,k)$ according to paper? – user_3pij Jun 11 '20 at 14:04
• Am I missing some trivial thing? – user_3pij Jun 11 '20 at 14:05

## 1 Answer

The key here is construction of the mixture. The components of the mixture are on the right hand side $$a_ka_{kj}p(x_{1:M}|a_k,a_{kj})$$. You add them up to get the probability of observing the capsules $$p(x_{1:M})$$ while having all parts.

On the left hand side you start with the same probability but written as $$p(x_{1:m}|\mathrm{parts})p(\mathrm{parts})$$, where the probability of all parts present is $$p(\mathrm{parts})=\sum_k\sum_ja_ka_{kj}$$ and $$p(x_m|\mathrm{parts})$$ is a constructed mixture. That's why you need to normalize the right hand side by $$p(\mathrm{parts})\ne 1$$ to get the sought likelihood $$p(x_{1:m}|\mathrm{parts})$$.

The idea here is not even Bayes theorem, but something like a weighted average. For instance, let's say are calculating the total weight of bus passengers: $$TW=N(\mathrm{male})*W(\mathrm{male})+N(\mathrm{female})*W(\mathrm{female})$$ where N is the number of passengers and W is the average weight of the group. Then you write the same as $$TW=(N(\mathrm{male})+N(\mathrm{female}))*W(\mathrm{all})$$ This gives the mixture weight $$W(\mathrm{all})=\frac{N(\mathrm{male})*W(\mathrm{male})+N(\mathrm{female})*W(\mathrm{female})}{N(\mathrm{male})+N(\mathrm{female})}$$

It's important to normalize because total num of passengers $$N(\mathrm{male})+N(\mathrm{female})$$ is not fixed. Changing only num of males impacts the weights of both males and females. In your case adding an object also changes weights of other components of the mixture probability.

• Thank you for your reply @Aksakal but what do you mean by "while having all the parts" and why we need to do $p(x_m |parts)p(parts)$? Shouldn't there be $p(x_m)$ where you wrote $p(x_{1:m})$? Also in your example you just multiplied and divided by "$N(male) + N(female)$". How this example is related to the problem? And does doing such normalization a regular thing or was it related to the paper? Could you please share some links if I am missing some thing. Really appreciated. – user_3pij Jun 13 '20 at 12:41
• Taking your example to simplify our problem, lets there's I want to calculate the likelihood of 3 passengers (parts in our case) belonging to a bus which have 2 compartments (object capsules in our case) and existence probability of these compartments keeps changing and is given by $a_k$. Each compartment predicts 2 candidate passengers (candidate parts in our case) such that each candidate passenger has its own conditional existence probability $a_{k,j}$ and a $pdf$(Gaussian $pdf$ in our case). – user_3pij Jun 13 '20 at 15:09
• So the $p(passenger) = \sum_{compartments}\sum_{candidates}p(passenger | compartment,candidate)p(candidate|compartment)p(compartment)$. – user_3pij Jun 13 '20 at 15:09
• Now how normalization would come here? – user_3pij Jun 13 '20 at 15:09