2
$\begingroup$

This is a simple Bayesian inference problem, where we are trying to infer some weight parameter $w$. Our posterior distribution is $$ P\propto \exp\left(-\frac{1}{\sigma^2} w^Tw\right) \exp\left(-f(w)\right)$$ $$ = \exp\left(-\frac{1}{\sigma^2} w^Tw-f(w)\right)$$ where the first term expresses Guassian prior (let's say $g(w)=\frac{1}{\sigma^2}w^Tw$). The second term expresses likelihood.

The first term scales with the number of elements ($N$) in vector $w$, and let's assume that the second term does not scale as fast as the first term and its scaling can only be estimated through numerical experiments. I believe it's logical to say that at very large $N$, the first term will dominate, and the posterior will be basically Gaussian. However, I am struggling to come up with a numerical analysis that will determine the threshold for $N$. What can I do?

I first thought about finding a maximum a posteriori estimation of $w$ (call it $w^*$), and see at which $N$, $g(w^*)$ is much greater than $f(w^*)$. But soon realized this doesn't make sense, because if the Gaussian term dominates, $g(w^*) \rightarrow 0$. Is this a correct argument?

The second thing I tried was using $w$ whose elements are i.i.d. $\mathcal{N}(0,\sigma^2)$, and see at which $N$, $g(w)$ is much greater than $f(w)$. I think this is a way to go, but I cannot clearly explain why this is right. Could anyone help?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.