1
$\begingroup$

I know very similar questions get asked a lot - but I would like to make sure that I am not missing something here:

Suppose we have the simple model:

y = mx + c + s*e

where e ~ N(0,1) and m, c and s are model parameters.

In the case where we are holding 's' fixed the number of parameters used to calculate the AIC is 2 (m and c).

The reason for the question is that I have seen it written that the 'noise term' should be counted - but I presume that is really referring to the 's' parameter in this example, and so should not be included if it is being held fixed. The 'e' RV does not itself contribute anything.

Again, apologies for the similarity to other questions - just want to confirm this specific point.

Thanks -

$\endgroup$
0
$\begingroup$

In your example $s$ seems to be the standard deviation of the residuals. In that case it should be counted as a parameter and it doesn't make much sense to held it fixed. Let say you compare (on the same dataset) your model with a more complex model, e.g. $y = mx^2 + mx + c + se$, and to an even simpler model $y = c + se$. The predictions of the more complex model (that is $\hat y = mx^2 + mx + c$) will be on average closer to the actual values of $y$, while the predictions of the simpler model will be on average more off. Thus, the standard deviation of the residuals will be smaller for the more complex model, and larger for the simpler one. If you held it fixed then for both models the set of parameters is not the one that maximizes the likelihood of the data and therefore you cannot use the AIC (which is calculated from the maximum value of the likelihood function and the number of free parameters).

Note also that the AIC is only a measure of the relative quality of statistical models (its absolute value doesn't mean much). In this example counting it or not in the calculation of the AIC amounts to adding a constant, 2, to the AIC of all three models. Therefore the results of the model comparisons, e.g. expressed in terms of AIC differences, would be exactly the same

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ OK - interesting point thanks. Your answer made me realize that I was considering 'se' to not be generated by some sort of measurement noise, but rather from some underlying source with known variance (hence no need to fit). Maybe a slightly unusual case, but in which case my original comment statements would be correct? - Thanks $\endgroup$ – James Jul 31 '18 at 22:58
  • $\begingroup$ I don't think it makes much sense. What criterion do you use to estimate the parameters $m$ and $c$? Least squares? $\endgroup$ – matteo Jul 31 '18 at 23:10
  • $\begingroup$ Given the way in which it is defined, $s$ seems to represent the average distance between the model predictions $\hat y$ and the actual value of $y$, so it isn't really measurement noise, nor some noise in the generative process that generated the data. $\endgroup$ – matteo Jul 31 '18 at 23:17
  • $\begingroup$ I think I am not clear on what the problem is with knowing a priori what s is (in a toy example). Let's say I construct a machine that takes a variable as input (x), multiplies it by some value m, and adds a value c plus a value scaled by s from a Normal distribution (and spits outs result: y). Because of the way I constructed the machine, I am very certain about s, but not so certain about m or c (in fact c may be missing altogether - to make the problem relevant to AIC). I want to estimate these unknowns from data that I have measured from the system, and figure out if c really is missing. $\endgroup$ – James Aug 1 '18 at 13:21
  • $\begingroup$ Let's say I am doing estimation by maximizing likelihood (from what I remember that comes out the same as least squares in this simple case). BTW, didn't notice earlier the paragraph where you say that the AIC differences would be the same - thanks for that. Ultimately I am just trying to get at whether the noise process itself contributes something to the AIC. $\endgroup$ – James Aug 1 '18 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.