# BIC in practice with gaussian distribution

I am considering a gaussian distribution: $$\begin{equation} y \sim N(net(x,w), \sigma^2). \end{equation}$$ $$net()$$ is just the output of some neural net with weights $$w$$ and input $$x$$.

The log-likelihood is $$\begin{equation} \log L = -\frac{n}{2} (\log(2\pi) + \log(\sigma^2)) - \frac{1}{2\sigma^2} \sum (y_i - net(x,w))^2 \end{equation}$$ and the BIC is \begin{align} BIC &= -2 \log L + \log(n) \cdot d \\ &= n(\log(2\pi) + \log(\sigma^2)) + \frac{1}{\sigma^2} \sum (y_i - net(x,\hat{w}))^2 + \log(n) \cdot d\\ &\approx n/\sigma^2 (MSE + \frac{\log(n)\cdot d \cdot \sigma^2}{n}), \end{align} where $$d$$ is the number of parameters. What I wonder is, how do I estimate $$\sigma^2$$ in practice? My intuition was to estimate it with usual MLE which is the MSE, i.e. $$\hat{\sigma}^2 = 1/n \sum(y_i - net(x, \hat{w}))^2$$, but then the first term would just cancel out... And does the variance count as a parameter in $$d$$? I am really confused how to use this in practice.

I don't see any particular reason for not estimating $$\sigma$$ as in any Gaussian process, with sample standard deviation given a sample of outputs from the neural network for some fixed $$x, \omega$$
But make sure $$\sigma$$ is indeed independent from those!
And yes, $$\sigma$$ is a parameter in its own right!
• Thank you! If I am getting you right, this is exactly the formula to estimate $\sigma^2$ that I have written down (which is the MSE). So you say that is okay that the is $BIC = n + \log(n) \cdot d$? That seems weird to me – msloryg Jun 4 '19 at 16:10