Nesting is when all of the models tested can be derived by eliminating parameters from a parent model. Non-nesting is when the models contain parameters that are not in a set with subset(s) format.
I suspect that AIC is more limited than its more optimistic proponents suggest. First, please consider that goodness of fit is not always a useful regression target. For example, suppose that we want to model how a drug is eliminated from blood plasma. Almost to the exclusion of other formulas, the pharmaceutical industry and the FDA would recommend using a sum of exponential terms expression. Sums of exponential terms (SET) functions are ubiquitous and often given as $C_{\text{SET}}(t)=\sum_{i=1}^n c_i \,e^{-\lambda_i\,t}$. Their density functions have not received much attention, as the industry and the FDA are blissfully unaware of statistical considerations:
\begin{equation}\label{eq:SET}
\text{ED}_n(t;\lambda_{1,2,3...n})=\sum_{i=1}^n\lambda_i \,p_i \,e^{-\lambda_i\,t},
\end{equation}
where $\sum_{i=1}^n p_i= 1$, the $\lambda_i$ are decay coefficients, the term constants of SET functions relate to the ED$_n$ scale parameters as $c_i=\kappa \lambda_i p_i$, and concentration is $C_{\text{SET}}(t)=\kappa\, \text{ED}_n$, where $\kappa=\text{AUC}(\text{SET})_0^{\infty}$. Note that ED$_n$ and SET only have scale parameters. Unfortunately, there are no shape parameters to aid in fitting disposition curves, the fitting of curve shapes, and extrapolation. There is no location parameter. It is typical, in the industry to choose $n=2$ for the above equation yielding $\text{ED}_2(t;\lambda_1,\lambda_2)=\lambda_1 \,p_1 \,e^{-\lambda_1\,t}+\lambda_2 \,p_2 \,e^{-\lambda_2\,t}$, where $p_1+p_2=1$.
This merited a comment by @whuber "... It is indeed possible to parameterize the ED$_2$ family with a scale parameter and two shape parameters; for instance, one could take $\lambda_1$ as an inverse scale parameter, leaving $\lambda_2>\lambda_1$ and $p_1$ as shape parameters. I cannot see any connection whatsoever between such considerations and derivatives unless "derivative" means something unusual in this context...." The response to which was "There is indeed an inefficient mutability of shape for an ED$_2$. Using multiple parameters to emulate a shape parameter is inefficient in the sense that the full range of shapes that a solitary shape parameter offers is not properly rendered.... One can reduce the error of fitting to zero by using a sufficient number of parameters... Consider overfitting a curve with a polynomial. One can reduce the error of fitting to zero by using a sufficient number of parameters. However, unless the physics of the problem is coincidentally an exact polynomial shape, that perfect goodness of fit is meaningless in terms of extrapolation, and the fit may be "wiggly" between the samples fit. That is, overfitting does not tell us what a good model is, and if one does not consider what the slope is between or among samples, the model itself may have achieved a pyrrhic goodness of fit."
Indeed, ED$_2$ is inflexible enough that exact solutions for four time-samples are sometimes complex field, i.e., not real and not physical. In one study of 413 subjects, eight results (1.9%) with four time-samples solutions had unphysical exponential coefficients
Now let us consider a non-nested model with respect to that latter equation. The gamma distribution (GD) is given by
\begin{equation}\label{eq:GD}
\text{GD}(t; a,b) =
\,\dfrac{1}{t}\;\dfrac{e^{-b \, t}(b \, t)^{\,a} }{\Gamma (a)} \;\; \;;\hspace{2em}t\geq 0 \;\; \;\;\\
,
%\tabularnewline
\end{equation}
where the gamma function satisfies $\Gamma (a)=\int _0^{\infty } e^{-t} t^{a-1}dt$. The GD is an ED when $a=1$. The GD also has a ($+\infty$) discontinuity at $t=0$ when $0<a<1$. However, that discontinuity is integrable $\left(\mathbb{R}_{\geq 0}\right)$. The gamma distribution has a rate parameter $b$, whereas $\frac{1}{b}$ is the scale parameter. The shape parameter for the GD is $a$. The shape parameter aids in fitting disposition curves and their shapes. There is no location parameter.
The GD and ED$_{n\geq2}$ are not respectively nested because one cannot reduce either model to be equal to the other by choosing particular values for their parameters. Now, let us cut to the chase. The gamma distribution has a shape parameter, and SET formulas do not. As a consequence, SET formulas do not fit the derivative of blood plasma concentration, because they lack shape parameters, and gamma distributions, or their convolutions, do in fact fit the derivatives. In the case of drug persistence in the body, without proper derivative fitting, there is no hope of predicting future plasma concentration of drugs using SET heuristics, whereas fitting of derivatives may permit more exact extrapolation.
When one plots SET derivatives from actual data fits, the result is a wiggly curve, with one bump for each exponential term, which is pathognomonic for overfitting. For another example of a non-nested models exhibiting problematic AIC values, in that case for $n$ of small to moderate size, see this answer.
AIC use only compares departures from goodness of fit of the data itself, and says nothing about how well derivatives are fit. B-spline fitting is an example of fitting both data and an arbitrary number of data derivatives. In the case of drug persistence in the body, AIC is insufficient as a fit criterion, as the useful model will fit not only the data, but also the shape of the data, and an AIC comparison of the non-nested SET and GD models is not sufficient to characterize their respective utility, at least as far as drug concentration in blood plasma is concerned. For further comparison concerning this see this paper. When that paper was under review, one of the reviewers requested an AIC comparison. The authors compared correlation coefficients between the models and the data instead of using AIC, and even that was irrelevant.
In summary, how one compares models is context dependent. That is, the appropriate goal of modelling is often not simple goodness-of-fit. However, it has become, IMHO, an all too frequent a reflex to think that AIC is relevant to any and all circumstances, and that without considering what those circumstances are. The first step in modelling should be, but seldom is, to identify what the goal or goals of modelling is, and to choose only those procedures that are appropriate to those goal(s).
