BIC in practice with gaussian distribution

Question

I am considering a gaussian distribution: \begin{equation} y \sim N(net(x,w), \sigma^2). \end{equation} $net()$ is just the output of some neural net with weights $w$ and input $x$.

The log-likelihood is \begin{equation} \log L = -\frac{n}{2} (\log(2\pi) + \log(\sigma^2)) - \frac{1}{2\sigma^2} \sum (y_i - net(x,w))^2 \end{equation} and the BIC is \begin{align} BIC &= -2 \log L + \log(n) \cdot d \\ &= n(\log(2\pi) + \log(\sigma^2)) + \frac{1}{\sigma^2} \sum (y_i - net(x,\hat{w}))^2 + \log(n) \cdot d\\ &\approx n/\sigma^2 (MSE + \frac{\log(n)\cdot d \cdot \sigma^2}{n}), \end{align} where $d$ is the number of parameters. What I wonder is, how do I estimate $\sigma^2$ in practice? My intuition was to estimate it with usual MLE which is the MSE, i.e. $\hat{\sigma}^2 = 1/n \sum(y_i - net(x, \hat{w}))^2$, but then the first term would just cancel out... And does the variance count as a parameter in $d$? I am really confused how to use this in practice.

Answer 1

I don't see any particular reason for not estimating $\sigma$ as in any Gaussian process, with sample standard deviation given a sample of outputs from the neural network for some fixed $x, \omega$

But make sure $\sigma$ is indeed independent from those!

And yes, $\sigma$ is a parameter in its own right!