0
$\begingroup$

It is said that $p \left( \theta | y _ { 1 : N } \right) \propto _ { \theta } p \left( y _ { 1 : N } | \theta \right) p ( \theta )$.

And $p \left( \theta | y _ { 1 : N } \right)$ is the posterior, $ p \left( y _ { 1 : N } | \theta \right)$ is the likelihood, and $p ( \theta )$ is the prior.

Suppose we have a model M, and its weights are W. To my understanding:

  1. As for $p \left( \theta | y _ { 1 : N } \right) \propto _ { \theta } p \left( y _ { 1 : N } | \theta \right) p ( \theta )$, I think $\theta$ is $W=\{w_1, w_2, ..., w_n\}$, $p(\theta)$ is a distribution for $w_i$, $ p \left( y _ { 1 : N } | \theta \right)$ is the accuracy, and $p \left( \theta | y _ { 1 : N } \right)$ is the ideal weights distribution. Are my understandings correct?

  2. As for bayesian inference for $p \left( \theta | y _ { 1 : N } \right)$, first we select a distribution $q(.)$ for $w_i$, and the goal is to find the parameters(mean, variance, ...) for this distribution so that it is very close to the ideal weights distribution $p \left( \theta | y _ { 1 : N } \right)$. Using KL-divergence

    $\mathrm { KL } ( q ( \cdot ) \| p ( \cdot | y ) ) := ... := \log p ( y ) - \int q ( \theta ) \log \frac { p ( \theta , y ) } { q ( \theta ) } d \theta$ , it is down to maximum $ \int q ( \theta ) \log \frac { p ( \theta , y ) } { q ( \theta ) } d \theta$. But how to solve this? I see some materials talking about SVI, but still do not understand this well. Where can I find some codes for SVI? |......| If me, I will first use Adam to optimize all the $w_i$, and then recompute the mean and variance for the distribution $q(.)$, after resampling, use Adam again to optimize these new $w_i$, ... . Is this a feasible method?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.