# What is the dimension (k) of these regression models?

I am attempting to use Akaike's Information Criterion to select the most appropriate model for some data. This means I need to find the likelihood of my data under various models, compute the AIC score for each model, and then select the model with the lowest AIC score. To find AIC, I would use:

$$AIC = 2k−2\log(L)$$

I want my question to be as clear as possible, so I'm going to go step-by step in calculating the likelihood of the data, simply by quoting from Barlow (1993).

The data are a set of (x,y) pairs, exactly as on p. 93 of Barlow (1993), $\{(x_i,y_i)\}$.

"The $x_i$ are known exactly, whereas the $y_i$ have been measured, each with some known resolution $\sigma_i$. Suppose that $y$ is given by some function $f$ of $x$, which also depends on a parameter (or, in general, several parameters) $a$, and that $a$ is what we want to estimate. This is the ideal $y$, however, and the actual measurements have been smeared by the resolution.

"Invoking the central limit theorem to show that the distributions of the measured $y$ values about their ideals is Gaussian, the probability of a particular $y_i$, for a given $x_i$, is

"$P(y_i;a) = \frac{1}{\sigma_i\sqrt{2\pi}}e^{-[y_i-f(x_i;a)]^{2}/2\sigma_i^{2}}$

"The logarithm of the likelihood for the complete data set is thus

"$\ln{L}=-\frac{1}{2}\sum[\frac{y_i-f(x_i)}{\sigma_i}]^2-\sum\ln{\sigma_i\sqrt{2\pi}}$ "

I have three models to select among:

$$y_A = m_1x$$ $$y_B = m_1x^{2}$$ $$y_C = m_1x^{m_2}$$

where $m_i$ is a parameter that I estimate using Least Squares minimization in KaleidaGraph.

I'm clear on how to calculate the likelihood $L$ of the data under each of these three models, following the parameter estimation. Based on responses to an earlier version of this question, there may be some disagreement/confusion over how I account for measurement error. I have, I think, assumed that $s_i = \sigma_i$. If this is wrong, please explain.

I am unsure of which values to use for $k$. For $y_A$ and $y_B$ I have estimated only one parameter, so does this mean that $k_A=1$ and $k_B=1$ ? Does $k_C=2$ ?

Suppose the models included lower order terms as well, eg

$$y_A = m_1 + m_2x$$ $$y_B = m_1 + m_2x + m_3x^{2}$$ $$y_C = m_1 + m_2x^{m_3}$$

When selecting from this set, would I use $$k_A = 2$$ $$k_B = 3$$ $$k_C = 3$$

References: Barlow, R., 1993. Statistics: A Guide to the use of Statistical Methods in the Physical Sciences

• Since none of your models appear to allow for an error term your choice should be totally straightforward. Likely at most one model will fit exactly, and most probably none will - with no errors, only exact fits 'fit the model' at all. A more interesting question would result after either adding an error term, or writing the right hand sides as parameters in some distribution for $y$, or failing that, at least putting $\operatorname{E}()$'s on the $y$ variables. Commented Apr 7, 2013 at 23:26

Your models are missing a random component. If it's $\ldots + \epsilon$ where $\epsilon\sim\mathcal{N}(0,\sigma^2)$, then you're not counting another parameter, $\sigma$. This doesn't matter when it's in all the models you're comparing, as you just look at differences in AIC, but it does matter when it isn't, or when you're using the second-order version, AICc. But you've counted the other parameters - the $m$s - correctly.

For example, if you have observations indexed $1, \ldots, i, \ldots, n$, then for each observation Model A might be

$$y_i = m_1 + m_2x_i + \epsilon_i \qquad \text{where }\epsilon_i\sim\mathcal{N}(0,\sigma^2)$$

So there are three parameters to estimate: $m_1$, $m_2$, & $\sigma$.

[In response to your latest edit: So each $y_i$ is the average of several observations with the same covariate pattern, the observations have Gaussian error & you've thus estimated error variances $\sigma_i$.

(1) There's a distinction between "fixed" or "known" parameters, which are set according to prior knowledge, and "free" or "unknown" parameters, which are estimated from the data. If you've made an independent estimate of $\sigma_i$ for each $y_i$ it should be included in the bias correction term of the AIC. (1a) However the absolute values of AIC are of no interest, only differences between the AIC of different models. For AIC (first-order at least) that implies you only need take into account differences in the number of estimated parameters for each model. (Not that it's hard to count how many parameters you've estimated, but it explains why people can ignore variance & intercepts in casual talk about model comparison & not go wrong.)

(2) It's unusual to fit independent estimates of error variance for each $y_i$. Are you quite sure (a) you want to, (b) that's what you in fact did?]

• Perhaps I need to clarify the question. Each $y$ value is an average of $n$ measurements, with sample standard deviation $s$. If I use $s$ to estimate $\sigma$ for each $y$ value, then have I not fixed the $\sigma$ parameter? I'm interested in choosing between three models, each of which predicts an average measurement, which I have called $y$, based on the independent variable $x$. Commented Apr 11, 2013 at 16:01
• That actually made it less clear for me. At any rate, every $\sigma$ you estimate is a free parameter & not fixed. If you forget about some free parameters it doesn't matter for calculating differences in AIC between models provided you forget about the same number of parameters in each model (& aren't using AICc). Commented Apr 11, 2013 at 18:12
• What if I want to model $\bar{y}$ rather than $y_i$? I want to make sure I am clear about what I want to do. Editing the question to hopefully reflect this... Commented Apr 11, 2013 at 22:41
• (1) Ok, I see the distinction between a fixed parameter from prior knowledge, versus a parameter estimated from the data, and counting the latter case in the AIC k parameter makes sense. At the same time, I see why as a practical matter, it is the differences in the number of parameters that are most important when comparing models. (2a) I can see how variability in measurement error may be unusual. But I do think that that is the situation I am in. (2b) So, yes, this is part of KaleidaGraph, each data point can have its own associated standard deviation. Thanks for the diligent responses! Commented Apr 12, 2013 at 19:12