So I know for a ridge regression model, we need to find an optimal $\lambda$ value.
I also know that we can achieve this by finding an optimal AIC value, that is, we find the $\lambda$ value that minimises the AIC.
The AIC is defined as follows,
$AIC = n\ln(\frac{SS_{res}}{n}) + 2p$
Where p is the effective degree of freedom,
$p = tr(\hat H) = tr(X(X^TX+\lambda I)^{-1}X)$
Combined, we get a function for AIC in terms of $\lambda$.
$AIC = n\ln(\frac{SS_{res}}{n}) + 2tr(X(X^TX+\lambda I)^{-1}X^T)$
Now, this is where my confusion starts. The first term is free of $\lambda$, therefore it is a constant. The second term seems like it's not a quadratic equation (or any polynomial). I tried plotting this in R.
x <- seq(0, 1000, by = 0.1)
k <- #number of parameters#
constant <- 5
variable <- function(n){
matrix <- solve(t(X)%*%X + (n * diag(k))
trace <- diag(X %*% (matrix) %*% t(X))
return(2 * sum(trace))
}
y <- c()
for (term in x){
y <- c(y, constant + variable(term))
}
plot(x, y)
For simplicity, I put an arbitrary constant for the first term. If you run the code with any X value, you can see that the graph is strictly decreasing, hence there's no optimal AIC. The questions are:
- Is $tr(X(X^TX+\lambda I)^{-1}X)$ in any way quadratic (or a polynomial)?
- If yes, what seems to be wrong with the code fragment?
- If no, is there something wrong with my understanding of $\lambda$ and AIC?