I have a dataset $S$ with $D$ features and three fitted linear regression models:
Model1. Ridge regression that is fitted on all $D$ features from $S$.
Model2. Ridge regression that is fitted on some $d < D$ features from $S$.
Model3. Lasso regression that is fitted on all $D$ features from $S$, and only $m < D$ features got non-zero weights (coefficients) after fitting.

I want to use AIC to select the best model. We know that AIC formula for linear regression models is the following:
$$\mathrm{AIC} = 2k + n \log{(\mathrm{RSS}/n)}.$$ where $k$ is the number or estimated parameters (degrees of freedom) and $n$ is the sample size. So we can easily calculate AIC value for all three models.

And I have two questions:
1. Can I compare AIC's values of these models and choose the best one with the lowest AIC?
I thought that the answer is yes but I became confused after reading documentation for AIC function in R package. It claims that models should be fitted on the same data. Whereas my Model2 is fitted on the (technically) different dataset (on the subset of $S$).

2. What is $k$ value for Model3?
It is clear that $k = D + 2$ for Model1 ($D$ estimates for slope parameters + intercept estimate + $\hat \sigma^2_\varepsilon$ estimate) and, similarly, $k=d+2$ for Model2.
But Model3 have only $m$ non-zero slope parameters after fitting. Does it mean that $k=m + 2$ for Model3?

  • 1
    $\begingroup$ Nice question and some good observations from your part (+1). $\endgroup$
    – usεr11852
    Dec 24 '19 at 17:07

Some preliminaries: In LASSO models, the number of non-zero coefficients is an unbiased and consistent estimate for the degrees of freedom of the lasso (see, Zou et al. (2007) "On the "degrees of freedom" of the lasso" for more details). In ridge models, the degrees of freedom are directly related to the singular values of the centred input matrix $X$ (see, Hastie et al. (2009) "Elements of Statistical Learning" in Sect. 3.4.1 for more details). Assuming the matrix $X$ has the SVD $X = USV^T$, the degree of freedom as a function of $\lambda$ are $df(\lambda) = \sum_{j=1}^p \frac{s_j^2}{s_j^2 +\lambda}.$ We can see clearly that for $\lambda \rightarrow 0$ we get $p$ degrees for freedom and for $\lambda \rightarrow \infty$ we get $0$ degrees of freedom.

Based on these and for your questions in particular:

  1. Yes, your observation is correct. M1 and M2 are not fitted on the same data. That said, the documentation mentions this because it aims to stop users from using different variants of the response variables and/or compare models that potentially have different rows of data or sample sizes. That is more obvious if we move from the "RSS"-derived calculation of AIC to the "Loglikehood"-derived one. Assuming a Gaussian log-likelihood as:

$$ \log(L(\theta)) =-\frac{|D|}{2}\log(2\pi) -\frac{1}{2} \log(|K|) -\frac{1}{2}(x-\mu)^T K^{-1} (x-\mu), $$ with $K$ being the covariance structure of our model (and $|K|$ the determinant of it), $|D|$ the number of points in our datasets, $\mu$ the mean response and $x$ our dependent variable. There is nothing invalidating this to be used on different datasets assuming we have the same dependent variable. (Notice though that for BIC we do want nested models so the model covariance structure $K$ are hierarchical). Being quite permissive with ourselves, in M2 we have an "elastic-net-like" situation where certain explanatory variables coefficients are set manually to $0$.

  1. I hope it is clear that the original calculation for $k$ for M1 and M2 are a bit oversimplifying. They need to directly account for $\lambda$. That said, you are correct that for M3, we will use the number of $m$ non-zero coefficients.
  • $\begingroup$ Thanks! So, as I understood, the answer on my first question should be "yes" (we can compare AIC's values of all three models and choose the best one with the lowest AIC). And this is true even for BIC. But I want to ask – what do you mean by "models that potentially have different rows of data"? "Different rows" mean samples from completely different probabilistic distribution? $\endgroup$
    – Rodvi
    Dec 24 '19 at 17:54
  • 1
    $\begingroup$ 1a. Yes. 1b. Yes, but do notice that for BIC we cannot compare models where M1 is not nested within M2. 2. Yes. Example: we want to model $y$, we did an experiment measuring $X_{Oc}$ and $y_{Oc}$ last October, got $N$ samples, we did the same experiment again in November, and recorded $X_{No}$ and $y_{No}$ got another $N$ samples; the fact we have two samples of equal size $N$ measuring the same variable $y$ does not mean we can compare the models $M_{Oc}$ and $M_{No}$ AIC. (The example is a bit simplistic but it can happen if we have access to the same $y$ from different data sources). $\endgroup$
    – usεr11852
    Dec 24 '19 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.