1
$\begingroup$

So I know for a ridge regression model, we need to find an optimal $\lambda$ value.

I also know that we can achieve this by finding an optimal AIC value, that is, we find the $\lambda$ value that minimises the AIC.

The AIC is defined as follows,

$AIC = n\ln(\frac{SS_{res}}{n}) + 2p$

Where p is the effective degree of freedom,

$p = tr(\hat H) = tr(X(X^TX+\lambda I)^{-1}X)$

Combined, we get a function for AIC in terms of $\lambda$.

$AIC = n\ln(\frac{SS_{res}}{n}) + 2tr(X(X^TX+\lambda I)^{-1}X^T)$

Now, this is where my confusion starts. The first term is free of $\lambda$, therefore it is a constant. The second term seems like it's not a quadratic equation (or any polynomial). I tried plotting this in R.

x <- seq(0, 1000, by = 0.1)
k <- #number of parameters#
constant <- 5
variable <- function(n){
   matrix <- solve(t(X)%*%X + (n * diag(k))
   trace <- diag(X %*% (matrix) %*% t(X))
   return(2 * sum(trace))
}
y <- c()
for (term in x){
  y <- c(y, constant + variable(term))
}
plot(x, y)

For simplicity, I put an arbitrary constant for the first term. If you run the code with any X value, you can see that the graph is strictly decreasing, hence there's no optimal AIC. The questions are:

  1. Is $tr(X(X^TX+\lambda I)^{-1}X)$ in any way quadratic (or a polynomial)?
  2. If yes, what seems to be wrong with the code fragment?
  3. If no, is there something wrong with my understanding of $\lambda$ and AIC?
$\endgroup$
2
  • $\begingroup$ Why not use cross validation? $\endgroup$
    – Dave
    Commented Apr 29, 2021 at 14:27
  • $\begingroup$ The purpose of this post is for theoretical knowledge rather than application. $\endgroup$ Commented Apr 29, 2021 at 15:12

1 Answer 1

0
$\begingroup$

I think $SS_{res}$ depends on $\lambda$, as the residuals are different for different values of $\lambda$. That should solve the problem.

$\endgroup$
2
  • $\begingroup$ Ah, so the SSres in the first term is actually the SSres for a ridge regression estimator rather than for an ordinary one? $\endgroup$ Commented Apr 29, 2021 at 15:26
  • $\begingroup$ @expulsio, indeed. $\endgroup$ Commented Apr 29, 2021 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.