BIC in practice with gaussian distribution

Question

I am considering a Gaussian distribution:

\begin{equation} y \sim N(\text{net}(x,w), \sigma^2). \end{equation}

where $\text{net}()$ is just the output of some neural net with weights $w$ and input $x$. The log-likelihood is

\begin{equation} \log L = -\frac{n}{2} (\log(2\pi) + \log(\sigma^2)) - \frac{1}{2\sigma^2} \sum_i (y_i - \text{net}(x_i,w_i))^2 \end{equation}

and the BIC is

\begin{align} BIC &= -2 \log L + \log(n) \cdot d \\[12pt] &= n(\log(2\pi) + \log(\sigma^2)) + \frac{1}{\sigma^2} \sum_i (y_i - \text{net}(x_i,\hat{w}_i))^2 + \log(n) \cdot d \\[6pt] &\approx \frac{n}{\sigma^2} \bigg( \text{MSE} + \frac{\log(n)\cdot d \cdot \sigma^2}{n} \bigg), \end{align}

where $d$ is the number of parameters.

Now, I want to estimate $\sigma^2$. My intuition was to estimate it with usual MLE which is the MSE, i.e.

$$\hat{\sigma}^2 = \frac{1}{n} \sum_i (y_i - \text{net}(x_i, \hat{w}_i))^2$$

but then the first term would just cancel out. Does the variance count as a parameter in $d$?

How do I estimate $\sigma^2$?

Answer 1

It would be preferable to profile out $\sigma^2$ earlier. From

$$\log L = -\frac{n}{2}\log(2\pi)-\frac{n}{2}\log\sigma^2 -\frac{1}{2\sigma^2}\textrm{RSS}$$ you can differentiate with respect to $\sigma^2$ and solve to get $$\hat\sigma^2=\frac{\text RSS}{n}$$ as the maximiser

Plugging that it then gives the profile likelihood $$\log L = -\frac{n}{2}\log(2\pi)-\frac{n}{2}\log\frac{\text RSS}{n} -\frac{n}{2}$$ which is free of $\sigma^2$. Dropping terms that don't depend on your model choice you have $$\log L = \frac{-n}{2}\log\frac{\text RSS}{n}$$ and $$\mathrm{AIC}= n\log\frac{\text RSS}{n} + 2p$$ and $$\mathrm{BIC}= n\log\frac{\text RSS}{n} + p\log n$$

Answer 2

I don't see any particular reason for not estimating $\sigma$ as in any Gaussian process, with sample standard deviation given a sample of outputs from the neural network for some fixed $x, \omega$

But make sure $\sigma$ is indeed independent from those!

And yes, $\sigma$ is a parameter in its own right!