# Which evaluation metric should I choose? AIC or MSE?

I am currently at a total loss, so I hope someone can point me in the right direction regarding my model selection.

The situation
I want to create a linear model that best forecasts my data. I am trying to figure out how many parameters I should enter into the model.
I have tried Lasso regression for this, as it should shrink coefficients of irrelevant variables to zero. However, feeding a different number of parameters to the Lasso regression yields different results. In particular, adding more parameters keeps returning more significant parameters.

I try to vary the number of parameters I let enter in the Lasso regression and compare the optimal models using AIC and MSE. The problem is, they both yield different results. What could be the cause of this?

Also, I read somewhere that since my Lasso regression is selecting an optimal model by minimizing the MSE, I can't compare them using AIC. Is this true?

Lastly, I was wondering if I should calculate AIC using the log-likelihood of the residuals or of the predicted values.

EDIT: For clarification on how I calculate both metrics, I first split the data with the first 70% as the train set, and the last 30% the test set.

Then, on this first 70% I perform Lasso regression with TimeSeriesSplit as a cross-validation method. I obtain an optimal model, and use it to create a prediction of my dependent variable $$y_t$$. Using this prediction, I calculate a likelihood assuming a normal distribution. I use this to calculate the AIC.

For the MSE, I use the model fitted on the first 70% of the data to create out-of-sample predictions, and compare them with my test set (the last 30% of the data).

• Lastly, I was wondering if I should calculate AIC using the log-likelihood of the residuals or of the predicted values. Obtaining the residuals requires you to consider the predicted values, so this does not make sense to me. Could you please clarify? // AIC has a specific definition based on the likelihood. If you calculate another way, your calculation might be legitimate and useful, but it isn’t AIC. // After you do your variable selection, what do you use as the number of parameters in the AIC calculation?
– Dave
Commented Jun 7, 2023 at 11:06
• Why use only one metric when you can use both, giving you more information about the performance of the model. Commented Jun 7, 2023 at 11:09
• @Dave I indeed use predicted values to calculate the log-likelihood. What I do now is get my predicted set of y values and calculate its mean and variance. But I could also calculate residuals and get their mean and variance, as they follow a distribution as well. I'm not sure what to do here.
– eork
Commented Jun 7, 2023 at 11:15
• @MikeHunter That's what I thought at first, but it turns out that different models are better depending on whether I look at the AIC or the MSE. So now I am wondering which one I should look at when choosing the optimal model.
– eork
Commented Jun 7, 2023 at 11:16
• What type of MSE are you computing: in sample or out of sample ("leave-one-out")? There is an old result by Stone (1977) that leave-one-out MSE minimization and AIC minimization are asymptotically equivalent. The proof is based on a Taylor series expansion of the log-likelihood function and thus holds for all models with twice differentiable densities. It should thus not matter much which criterion you use. Commented Jun 7, 2023 at 12:34