0
$\begingroup$

Considering $n$ observations that an be modelled by a Gaussian error model and two nested motion models with $p = 4$ and $p = 7$ parameters, I want to compute the log likelihoods $L$ given the Maximum Likelihood Estimates (MLE) of the two models to perform a likelihood ratio test or a comparison based on $\text{AIC}$ and $\text{BIC}$.

There are two formulas arising in the literature that express the log likelihood in these scenarios and are usually considered in the Gaussian error scenario, both applyable in the same way to compute $\text{AIC}$ and $\text{BIC}$, in the following exemplary for the $\text{AIC}$. First one is given for a known $\sigma$ as

$\text{I}:\text{AIC} = \sum_i \frac{\text{RSS}}{\sigma^2} + 2p + \text{const}$.

The other one is given for an unknown $\sigma$ as

$\text{II}:\text{AIC} = n \log{\frac{\text{RSS}}{n}} + 2p + \text{const}$,

where the estimated $\hat{\sigma}^2 = \frac{\text{RSS}}{n}$ is determined as a MLE.

In my scenario I have the choice to estimate $\sigma$ for my data with $n \approx 1500$ points because it is not known or I use synthetic data and add a known amount of Gaussian noise. My question is now which formula should I use when and in which scenarios are they actually valid?

Since the estimate for $\hat{\sigma}$ is only valid for large $n$, why is the emperical standard deviation not used? Because the difference is usually very small?

$\endgroup$
1
$\begingroup$

Formula II isn't a large-sample approximation; it's the exact AIC (well, up to additive constants)

The Gaussian likelihood for a single observation is $$L\beta,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\frac{-(y-x\beta)^2}{2\sigma^2}$$ so the loglikelihood is $$\ell(\beta,\sigma)= -\frac{1}{2}\log 2\pi - \frac{1}{2}\log\sigma^2 -\frac{1}{2}\frac{(y-x\beta)^2}{\sigma^2}$$ Adding up for $n$ observations: $$\ell_n(\beta,\sigma)= -\frac{1}{2}n\log 2\pi - \frac{1}{2}n\log\sigma^2 -\frac{1}{2}\frac{\sum_i (y_i-x_i\beta)^2}{\sigma^2}$$

Now, we want to compare the AIC between two models at the MLE, not at some arbitrary $(\beta,\,\sigma^2)$. The MLE for $\beta$ doesn't depend on $\sigma^2$ and if you plug it in, you get the sum in the last term replaced by the RSS $$\max_\beta \ell_n(\beta,\sigma^2)= -\frac{1}{2}n\log 2\pi - \frac{1}{2}n\log\sigma^2 -\frac{1}{2}\frac{\mathrm{RSS}}{\sigma^2}$$ For any value of $\sigma$, this is the highest value the loglikelihood can take (the profile loglikelihood).

Next, we do the same thing for $\sigma^2$. The loglikelihood is maximised at $\sigma^2=\mathrm{RSS}/n$. We're not talking here about whether that's a good estimator, we're just trying to maximise the loglikelihood so we can compare models.

Plugging in $\hat\sigma^2$ gives us the highest possible value of the loglikelihood for this model and data $$\max_{\beta,\sigma^2}\ell_n(\beta,\sigma^2)= -\frac{1}{2}n\log 2\pi - \frac{1}{2}n\log\mathrm{RSS}+\frac{1}{2}n\log n -\frac{1}{2}n$$ Now if we multiply by $-2$ and add $2p$ we get

$$AIC = 2p-2\max_{\beta,\sigma^2}\ell_n(\beta,\sigma^2)= 2p+n\log 2\pi + n\log\mathrm{RSS}-n\log n +n$$ or up to constants either $$AIC = 2p-2\max_{\beta,\sigma^2}\ell_n(\beta,\sigma^2)= 2p + n\log\mathrm{RSS}+ \textrm{constants}$$ $$AIC = 2p-2\max_{\beta,\sigma^2}\ell_n(\beta,\sigma^2)= 2p + n\log\frac{\mathrm{RSS}}{n}+ \textrm{different constants}$$

There's nothing here that's a large-sample approximation, and while you might well use the values of $\beta$ and $\sigma^2$ that maximise $\ell$ as estimators, this argument doesn't depend at all on how good they are as estimators.

If $p$ is large compared to $n$ you might want to use some sort of corrected AIC instead, but this is the AIC.

$\endgroup$
4
  • $\begingroup$ Given formula 4, I could directly use it with an estimated sigma which would result in another weighting of the RSS. Since the RSS is differenlty weighted in the last two formulas, what are the reasons to pick one or the other? $\endgroup$ – Miau Aug 25 '20 at 12:39
  • $\begingroup$ If you mean formula 1, you could, but it would still be the formula for the AIC with known $\sigma^2$, so it would be an approximation and you might as well use either the actual AIC or (for large $p$) perhaps some other criterion. $\endgroup$ – Thomas Lumley Aug 26 '20 at 0:07
  • $\begingroup$ Thank you so far. You use the estimator $\hat{\sigma}^2 = \mathrm{RSS}/n$. When comparing two models, $\hat{\sigma}^2$ has to be determined based on either $\mathrm{RSS}_1$ or $\mathrm{RSS}_2$. Your formula is based on the fact that the $\mathrm{RSS}$ in the AIC formula is the same as for estimating $\hat{\sigma}^2$. If you just plug in the $\mathrm{RSS}_i$ of the respective model $\mathcal{M}_i$, you get different estimates for $\hat{\sigma}^2$ in each score $\mathrm{AIC}_1$ and $\mathrm{AIC}_2$. Use the model with more parameters to estimate $\hat{\sigma}^2$ or use two different values? $\endgroup$ – Miau Aug 29 '20 at 14:40
  • $\begingroup$ No matter how many models you are comparing, the derivation I gave is for the maximum of the loglikelihood under each model. So, if you need to think in terms of estimators (which I don't recommend), you use different estimators for the two models. On the other hand, If you're using your first formula, $\sigma^2$ is a known constant and so is the same for all models, and if you're actually estimating it, you need to use the same estimator for all models $\endgroup$ – Thomas Lumley Aug 29 '20 at 22:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.