0
$\begingroup$

I am considering a gaussian distribution: \begin{equation} y \sim N(net(x,w), \sigma^2). \end{equation} $net()$ is just the output of some neural net with weights $w$ and input $x$.

The log-likelihood is \begin{equation} \log L = -\frac{n}{2} (\log(2\pi) + \log(\sigma^2)) - \frac{1}{2\sigma^2} \sum (y_i - net(x,w))^2 \end{equation} and the BIC is \begin{align} BIC &= -2 \log L + \log(n) \cdot d \\ &= n(\log(2\pi) + \log(\sigma^2)) + \frac{1}{\sigma^2} \sum (y_i - net(x,\hat{w}))^2 + \log(n) \cdot d\\ &\approx n/\sigma^2 (MSE + \frac{\log(n)\cdot d \cdot \sigma^2}{n}), \end{align} where $d$ is the number of parameters. What I wonder is, how do I estimate $\sigma^2$ in practice? My intuition was to estimate it with usual MLE which is the MSE, i.e. $\hat{\sigma}^2 = 1/n \sum(y_i - net(x, \hat{w}))^2$, but then the first term would just cancel out... And does the variance count as a parameter in $d$? I am really confused how to use this in practice.

$\endgroup$
0
$\begingroup$

I don't see any particular reason for not estimating $\sigma$ as in any Gaussian process, with sample standard deviation given a sample of outputs from the neural network for some fixed $x, \omega$

But make sure $\sigma$ is indeed independent from those!

And yes, $\sigma$ is a parameter in its own right!

$\endgroup$
  • $\begingroup$ Thank you! If I am getting you right, this is exactly the formula to estimate $\sigma^2$ that I have written down (which is the MSE). So you say that is okay that the is $BIC = n + \log(n) \cdot d$? That seems weird to me $\endgroup$ – msloryg Jun 4 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.