I am considering a Gaussian distribution:

\begin{equation} y \sim N(\text{net}(x,w), \sigma^2). \end{equation}

where $\text{net}()$ is just the output of some neural net with weights $w$ and input $x$. The log-likelihood is

\begin{equation} \log L = -\frac{n}{2} (\log(2\pi) + \log(\sigma^2)) - \frac{1}{2\sigma^2} \sum_i (y_i - \text{net}(x_i,w_i))^2 \end{equation}

and the BIC is

\begin{align} BIC &= -2 \log L + \log(n) \cdot d \\[12pt] &= n(\log(2\pi) + \log(\sigma^2)) + \frac{1}{\sigma^2} \sum_i (y_i - \text{net}(x_i,\hat{w}_i))^2 + \log(n) \cdot d \\[6pt] &\approx \frac{n}{\sigma^2} \bigg( \text{MSE} + \frac{\log(n)\cdot d \cdot \sigma^2}{n} \bigg), \end{align}

where $d$ is the number of parameters. What I wonder is, how do I estimate $\sigma^2$ in practice? My intuition was to estimate it with usual MLE which is the MSE, i.e.

$$\hat{\sigma}^2 = \frac{1}{n} \sum_i (y_i - \text{net}(x_i, \hat{w}_i))^2$$

but then the first term would just cancel out... And does the variance count as a parameter in $d$? I am really confused how to use this in practice.

I am considering a Gaussian distribution:

\begin{equation} y \sim N(\text{net}(x,w), \sigma^2). \end{equation}

where $\text{net}()$ is just the output of some neural net with weights $w$ and input $x$. The log-likelihood is

\begin{equation} \log L = -\frac{n}{2} (\log(2\pi) + \log(\sigma^2)) - \frac{1}{2\sigma^2} \sum_i (y_i - \text{net}(x_i,w_i))^2 \end{equation}

and the BIC is

\begin{align} BIC &= -2 \log L + \log(n) \cdot d \\[12pt] &= n(\log(2\pi) + \log(\sigma^2)) + \frac{1}{\sigma^2} \sum_i (y_i - \text{net}(x_i,\hat{w}_i))^2 + \log(n) \cdot d \\[6pt] &\approx \frac{n}{\sigma^2} \bigg( \text{MSE} + \frac{\log(n)\cdot d \cdot \sigma^2}{n} \bigg), \end{align}

where $d$ is the number of parameters.

Now, I want to estimate $\sigma^2$. My intuition was to estimate it with usual MLE which is the MSE, i.e.

$$\hat{\sigma}^2 = \frac{1}{n} \sum_i (y_i - \text{net}(x_i, \hat{w}_i))^2$$

but then the first term would just cancel out. Does the variance count as a parameter in $d$?

How do I estimate $\sigma^2$?

I am considering a gaussian distribution: \begin{equation} y \sim N(net(x,w), \sigma^2). \end{equation} $net()$ is just the output of some neural net with weights $w$ and input $x$.

The log-likelihood is \begin{equation} \log L = -\frac{n}{2} (\log(2\pi) + \log(\sigma^2)) - \frac{1}{2\sigma^2} \sum (y_i - net(x,w))^2 \end{equation} and the BIC is \begin{align} BIC &= -2 \log L + \log(n) \cdot d \\ &= n(\log(2\pi) + \log(\sigma^2)) + \frac{1}{\sigma^2} \sum (y_i - net(x,\hat{w}))^2 + \log(n) \cdot d\\ &\approx n/\sigma^2 (MSE + \frac{\log(n)\cdot d \cdot \sigma^2}{n}), \end{align} where $d$ is the number of parameters. What I wonder is, how do I estimate $\sigma^2$ in practice? My intuition was to estimate it with usual MLE which is the MSE, i.e. $\hat{\sigma}^2 = 1/n \sum(y_i - net(x, \hat{w}))^2$, but then the first term would just cancel out... And does the variance count as a parameter in $d$? I am really confused how to use this in practice.

I am considering a Gaussian distribution:

\begin{equation} y \sim N(\text{net}(x,w), \sigma^2). \end{equation}

where $\text{net}()$ is just the output of some neural net with weights $w$ and input $x$. The log-likelihood is

\begin{equation} \log L = -\frac{n}{2} (\log(2\pi) + \log(\sigma^2)) - \frac{1}{2\sigma^2} \sum_i (y_i - \text{net}(x_i,w_i))^2 \end{equation}

and the BIC is

\begin{align} BIC &= -2 \log L + \log(n) \cdot d \\[12pt] &= n(\log(2\pi) + \log(\sigma^2)) + \frac{1}{\sigma^2} \sum_i (y_i - \text{net}(x_i,\hat{w}_i))^2 + \log(n) \cdot d \\[6pt] &\approx \frac{n}{\sigma^2} \bigg( \text{MSE} + \frac{\log(n)\cdot d \cdot \sigma^2}{n} \bigg), \end{align}

where $d$ is the number of parameters. What I wonder is, how do I estimate $\sigma^2$ in practice? My intuition was to estimate it with usual MLE which is the MSE, i.e.

$$\hat{\sigma}^2 = \frac{1}{n} \sum_i (y_i - \text{net}(x_i, \hat{w}_i))^2$$

but then the first term would just cancel out... And does the variance count as a parameter in $d$? I am really confused how to use this in practice.