Log-Likelihood Computation for AIC & BIC

Question

Considering $n$ observations that an be modelled by a Gaussian error model and two nested motion models with $p = 4$ and $p = 7$ parameters, I want to compute the log likelihoods $L$ given the Maximum Likelihood Estimates (MLE) of the two models to perform a likelihood ratio test or a comparison based on $\text{AIC}$ and $\text{BIC}$.

There are two formulas arising in the literature that express the log likelihood in these scenarios and are usually considered in the Gaussian error scenario, both applyable in the same way to compute $\text{AIC}$ and $\text{BIC}$, in the following exemplary for the $\text{AIC}$. First one is given for a known $\sigma$ as

$\text{I}:\text{AIC} = \sum_i \frac{\text{RSS}}{\sigma^2} + 2p + \text{const}$.

The other one is given for an unknown $\sigma$ as

$\text{II}:\text{AIC} = n \log{\frac{\text{RSS}}{n}} + 2p + \text{const}$,

where the estimated $\hat{\sigma}^2 = \frac{\text{RSS}}{n}$ is determined as a MLE.

In my scenario I have the choice to estimate $\sigma$ for my data with $n \approx 1500$ points because it is not known or I use synthetic data and add a known amount of Gaussian noise. My question is now which formula should I use when and in which scenarios are they actually valid?

Since the estimate for $\hat{\sigma}$ is only valid for large $n$, why is the emperical standard deviation not used? Because the difference is usually very small?

Answer 1

Formula II isn't a large-sample approximation; it's the exact AIC (well, up to additive constants)

The Gaussian likelihood for a single observation is $$L\beta,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\frac{-(y-x\beta)^2}{2\sigma^2}$$ so the loglikelihood is $$\ell(\beta,\sigma)= -\frac{1}{2}\log 2\pi - \frac{1}{2}\log\sigma^2 -\frac{1}{2}\frac{(y-x\beta)^2}{\sigma^2}$$ Adding up for $n$ observations: $$\ell_n(\beta,\sigma)= -\frac{1}{2}n\log 2\pi - \frac{1}{2}n\log\sigma^2 -\frac{1}{2}\frac{\sum_i (y_i-x_i\beta)^2}{\sigma^2}$$

Now, we want to compare the AIC between two models at the MLE, not at some arbitrary $(\beta,\,\sigma^2)$. The MLE for $\beta$ doesn't depend on $\sigma^2$ and if you plug it in, you get the sum in the last term replaced by the RSS $$\max_\beta \ell_n(\beta,\sigma^2)= -\frac{1}{2}n\log 2\pi - \frac{1}{2}n\log\sigma^2 -\frac{1}{2}\frac{\mathrm{RSS}}{\sigma^2}$$ For any value of $\sigma$, this is the highest value the loglikelihood can take (the profile loglikelihood).

Next, we do the same thing for $\sigma^2$. The loglikelihood is maximised at $\sigma^2=\mathrm{RSS}/n$. We're not talking here about whether that's a good estimator, we're just trying to maximise the loglikelihood so we can compare models.

Plugging in $\hat\sigma^2$ gives us the highest possible value of the loglikelihood for this model and data $$\max_{\beta,\sigma^2}\ell_n(\beta,\sigma^2)= -\frac{1}{2}n\log 2\pi - \frac{1}{2}n\log\mathrm{RSS}+\frac{1}{2}n\log n -\frac{1}{2}n$$ Now if we multiply by $-2$ and add $2p$ we get

$$AIC = 2p-2\max_{\beta,\sigma^2}\ell_n(\beta,\sigma^2)= 2p+n\log 2\pi + n\log\mathrm{RSS}-n\log n +n$$ or up to constants either $$AIC = 2p-2\max_{\beta,\sigma^2}\ell_n(\beta,\sigma^2)= 2p + n\log\mathrm{RSS}+ \textrm{constants}$$ $$AIC = 2p-2\max_{\beta,\sigma^2}\ell_n(\beta,\sigma^2)= 2p + n\log\frac{\mathrm{RSS}}{n}+ \textrm{different constants}$$

There's nothing here that's a large-sample approximation, and while you might well use the values of $\beta$ and $\sigma^2$ that maximise $\ell$ as estimators, this argument doesn't depend at all on how good they are as estimators.

If $p$ is large compared to $n$ you might want to use some sort of corrected AIC instead, but this is the AIC.