This is a simple Bayesian inference problem, where we are trying to infer some weight parameter $w$. Our posterior distribution is $$ P\propto \exp\left(-\frac{1}{\sigma^2} w^Tw\right) \exp\left(-f(w)\right)$$ $$ = \exp\left(-\frac{1}{\sigma^2} w^Tw-f(w)\right)$$ where the first term expresses Guassian prior (let's say $g(w)=\frac{1}{\sigma^2}w^Tw$). The second term expresses likelihood.

The first term scales with the number of elements ($N$) in vector $w$, and let's assume that the second term does not scale as fast as the first term and its scaling can only be estimated through numerical experiments. I believe it's logical to say that at very large $N$, the first term will dominate, and the posterior will be basically Gaussian. However, I am struggling to come up with a numerical analysis that will determine the threshold for $N$. What can I do?

I first thought about finding a maximum a posteriori estimation of $w$ (call it $w^*$), and see at which $N$, $g(w^*)$ is much greater than $f(w^*)$. But soon realized this doesn't make sense, because if the Gaussian term dominates, $g(w^*) \rightarrow 0$. Is this a correct argument?

The second thing I tried was using $w$ whose elements are i.i.d. $\mathcal{N}(0,\sigma^2)$, and see at which $N$, $g(w)$ is much greater than $f(w)$. I think this is a way to go, but I cannot clearly explain why this is right. Could anyone help?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.