# Model selection: Is AIC enough or should one compute the p-value in model selection (and if yes to how to do it?)?

I fitted 2 models with a python package (curve_fit function from scipy.optimize) one linear and one nonlinear. I want to compare those 2 models. I compared those to model by calculating the AIC using the definition given in (p.63 Burnham Anderson Model Selection and Multi-Model Inference Springer 2002), namely: $$AIC = n \log(\hat{\sigma}^2) + 2k$$ where $$\hat{\sigma}^2 = \frac{\sum \hat{\epsilon}_i^2}{n}$$ where $$n$$ is the number of data points, $$k$$ is the number of parameters, $$\epsilon$$ is the error of the prediction. My question is the following: is the value of the AIC enough to decide which models is the best, or do I have to compute a p-value to between the 2 values (and if yes, how to do it with the given approach and without having to use lm() function of R for example).

• How do you want to use your selected model?
– Dave
Jun 27, 2022 at 9:59
• @Dave. I want to use the $k$ hyperparameters on real data
– ecjb
Jun 27, 2022 at 12:09
• To do what with the real data?
– Dave
Jun 27, 2022 at 12:11
• predict a variable in a clinical setting
– ecjb
Jun 27, 2022 at 12:17
• @Dave: I posted a new question there: stats.stackexchange.com/questions/580148/…
– ecjb
Jun 27, 2022 at 14:16

Update + correction

The AIC is a more general concept than what the formula $$n\log(RSS/n) + 2k$$ implies. However, if you make assumptions and get details right, you can use the simplified formula to compare the fits of a linear and a non-linear model to the same data.

• The errors are independent Gaussian with mean zero and equal variance٭.
• The parameters include the variance $$\sigma^2$$. So add 1 to $$k$$ if $$k$$ counts only the parameters optimized by scipy.optimize.curve_fit.
• The AIC estimates how well a model generalizes to new data when the model is well-specified. (This assumption is required to derive the number of parameters penalty from the Fisher information matrix [1].) So both the linear and non-linear model should be reasonable fit to the data in order to make a meaningful comparison.
• The two models are nested. There are arguments for and against. In Selecting Amongst Large Classes of Models Brian Ripley argues that the models have to be nested. In AIC myths and misunderstandings Anderson and Burnham argue that the models need not be nested.
• The same algorithm is used to fit both models. See AIC using nls function.

٭ The equal variance assumption can be relaxed by specifying weights $$w_i$$ such that $$e_i \sim \operatorname{N}(0,w_i\sigma^2)$$ and updating the formula appropriately. The same weights must be used for both models.

You've got the formula for the AIC wrong.

The Akaike information criterion (AIC) is defined as:

\begin{aligned} \operatorname{AIC} = -2 \log\left\{\ell(\hat{\theta} | y)\right\} + 2k \end{aligned} where $$\ell(\hat{\theta} | y)$$ is the likelihood function evaluated at the maximum likelihood estimate (MLE) $$\hat{\theta}$$ of the model parameters $$\theta$$ [1].

If the model is a linear least squares regression, the AIC simplifies to:

\begin{aligned} \operatorname{AIC} = n\log\{\hat{\sigma}^2\} + 2k \end{aligned} where $$\hat{\sigma}$$ is the residual standard error and the number of parameters $$k$$ includes the variance. Burnham and Anderson (2002) defines the AIC on page 60 and the AIC in the least squares case on page 63.

Moreover, the AIC is not a random quantity, so you can't compute a p-value for it.

[1] C. R. Shalizi. Advanced Data Analysis from an Elementary Point of View (2021). Available online.
[2] K. P. Burnham and D. R. Anderson. Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach. Springer, 2002.

• Thank you for your answer @dipetkov. I am not a statistician by training. How is it possible to calculate the likelihood function with this those data?
– ecjb
Jun 27, 2022 at 12:11
• Have no idea since I don't know anything about your models. Not all models have a likelihood function. I suggest you try the following: (In a new question) Describe your data X, your task Y and the models M1 and M2. Then ask how to compare them. I think this is what @Dave is getting at by asking you for details. Instead you've written a question that makes it sounds like you are interested in the general theory of comparing models. Jun 27, 2022 at 12:19
• I upvoted your question and posted a new question there: stats.stackexchange.com/questions/580148/…
– ecjb
Jun 27, 2022 at 14:16
• You still use the wrong formula for AIC in the follow-up question. Jun 27, 2022 at 14:21
• you mean that the correct formula would be the one with the log likelood? My goal would be to calculate AIC with the error and the number of parameters. Is there a way to calculate AIC for non linear function only with those informations
– ecjb
Jul 14, 2022 at 11:34