I was doing maximum a posteriori estimation of the variance $v\equiv \sigma^2$ using samples $\{x_n\}$ from a normally distributed random variable $X \sim \mathcal{N}(0,1)$, where the variance has an inverse Gamma prior distribution.
The posterior for $N$ samples is proportional to this:
$$ \begin{split} p(v|\{x_n\},a,b) &\propto \prod_{n=1}^N \mathcal{N}(x_n | 0, v) \Gamma^{-1}(v | a/2, b/2)\\ &= \prod_{n=1}^N \left[ (2\pi)^{-1/2} v^{-1/2} e^{-\frac{x_n^2}{2v}} \times \frac{(b/2)^{a/2}}{\Gamma(a/2)} v^{-a/2-1} e^{-\frac{b}{2v}} \right] \end{split} $$
Wait, this is wrong! It should actually be:
$$ \begin{split} p(v|\{x_n\},a,b) &\propto \left[ \prod_{n=1}^N \mathcal{N}(x_n | 0, v) \right] \times \Gamma^{-1}(v | a/2, b/2)\\ &= \left[ \prod_{n=1}^N (2\pi)^{-1/2} v^{-1/2} e^{-\frac{x_n^2}{2v}} \right] \times \frac{(b/2)^{a/2}}{\Gamma(a/2)} v^{-a/2-1} e^{-\frac{b}{2v}} \end{split} $$
Essentially, I thought the product $\prod_{n=1}^N$ was over products of the Normal and inverse gamma distributions, but that's incorrect. When I took the log of the first (incorrect!) posterior and maximized it anyway, I got this:
$$ \hat{\sigma}^2 \equiv \hat{v} = \frac{ \sum_n x_n^2 + N \cdot b }{(a + 3) N} \neq \frac{ \sum_n x_n^2 + b }{a + 2 + N} $$
The last expression is the correct MAP, and it's of course not the same as what I got.
Question: is the estimate that I got by making this mistake in the posterior absolute nonsense?
As far as I can tell, I unintentionally did this:
- Mistakenly assumed this probabilistic model for pairs of random variables $(x_n, v)$, where $x_n$ are my data, but $v$ is not observed: $$ p(x_n, v | a,b) = \mathcal{N}(x_n | 0,v) \times \Gamma^{-1}(v | a/2,b/2) $$
- Wrote the likelihood (assuming independence of all pairs): $$ p\left( \left\{(x_n,v)\right\}_{n=1}^N \middle| a,b \right) = \prod_{n=1}^N p(x_n, v | a,b) $$ Same as the first formula in my question, but now this is not a posterior, but rather a joint distribution.
- And maximized the (log-)likelihood w.r.t that $v$ to obtain: $$ \hat{v} = \arg\max_v \prod_{n=1}^N p(x_n, v | a,b) $$
Is such a process even valid? Did I estimate any meaningful quantity by doing this? Since the variance $v$ is not observed, this seems similar in spirit to compound distributions, but here I'm not integrating over $v$ and instead maximizing with respect to it...