I am modelling a nonlinear stochastic process and have data to compare model output against. My aim is to obtain an evolution equation of the form,

$$\frac{du}{dt} = f(u,\theta_f)+\alpha(u,\theta_\alpha)\xi(t),$$

($f$: a nonlinear function, $\xi(t)$: Gaussian noise, $\alpha$: a linear function, $\theta_{i}$: parameters) that can satisfactorily reproduce the observed probability distribution of $u$. Therefore, I fit the parameters with an optimisation algorithm that minimizes the mean squared difference between histograms of steady state model output and data. Alternative models are chosen by taking different functional forms of $f$.

For selection between these alternative models, I use the $AIC$. Usually, the $AIC$ is calulated from the variable of interest $u$ itself (see 'Usual procedure'). I calculate the $AIC$ differently, from histograms of the data and model output of $u$ (see 'My approach').

Usual procedure:

In e.g. linear regression, model errors $\hat\sigma^2_\epsilon$ are assumed to be normally distributed, such that,

$$AIC=n \ln(\hat\sigma_\epsilon^2) + 2k,$$

where $\hat\sigma^2_\epsilon$ is obtained by

$$\hat\sigma_\epsilon^2=\frac{1}{n}\sum_{i=1}^{n}[\hat u_{i}(\theta)−u_{i}]^{2},$$

with $\hat u_{i}(\theta)$ model estimates, $u_{i}$ observations and $n$ the number of each.

My approach:

I calculate the $AIC$ based on the histograms of model output and observations, from the mean squared difference mentioned above, i.e.

$$\hat\sigma_\epsilon^2=\frac{1}{m}\sum_{j=1}^{m}\{ h[\hat u_j(\theta)]−h(u_{j})\}^{2}.$$

Here, $h$ is the normalised frequency in bins with bin center $u_j$, $j\in \{1,...,m\}$. Note that $k$ is the number of parameters, i.e. the number of elements in the set $\theta=\theta_f \cup \theta_\alpha$.

My question:

Is this alternative way of calculating the $AIC$ valid and is it justified for my problem? Are there better ways of doing this?


If you use a different formulation of the variance $\sigma^{2}_{\epsilon}$ then technically you are not using Akaike’s IC, as the above mentioned Akaike’s formula for OLS regression comes from the equivalence between MLE and OLS (see for example this). So if you modify the variance formula, then there is no longer a correspondence between Akaike’s and Likelihood and you are calculating an info ratio which is not Akaike’s one. It is rather an alternative, which may be tested to see if it works asymptotically, but we cannot be sure a priori that it will have all the desiderabile properties of Akaike’s one, because it is different.

So if you are interested in the use of the alternative, to be 100% rigorous, you should explore its properties and consistency by numerical simulations (I am referring to these properties, you do not need to buy the article just read the abstract to understand what I am referring to, there are a number of free sources for this). Otherwise use Akaike’s with its standard formulation of the variance.

| cite | improve this answer | |
  • $\begingroup$ Thanks - I will read up about the theory behind MLE and how it is used for the AIC. Is the standard formulation you refer to $AIC=−2 \log(\cal{L}(\hat θ|y))+ 2k$ ? $\endgroup$ – Bertram Aug 29 '19 at 21:54
  • $\begingroup$ @Bertram yes the standard formulation is the one you mentioned. However the formula that you have attached for linear regression is also fine for linear regression because in that specific case it is equivalent. As long as you keep calculating the variance in the usual way. If you derogate, then you cannot be 100% sure that the new criterion under your approach may lead to the right specification. It may be better or worse. But to be rigorous you should test it empirically on simulated data. This holds any time you try to modify a given ratio without literature having tested it. $\endgroup$ – Fr1 Aug 30 '19 at 2:00
  • $\begingroup$ This is to be rigorous, like for an academic paper. $\endgroup$ – Fr1 Aug 30 '19 at 2:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.