# Why the $R^2$, AIC and BIC criteria give different “best model” for models of equal complexity?

I'm modeling growth curve of several ecosystems with respect to their rainfall-productivity relationship using a simple linear regression

$\text{ANPP}(t)=a+b\cdot\text{Rain}(t)$

and a modified version of the Brody Model

$\text{ANPP}(t)=a\cdot\left[1-e^{(-b\cdot\text{Rain}(t))}\right]$,

i.e. for each ecosystem, the two models are fitted to observed data and compared to each other.

I would like to know why the $R^2, AIC$ and $BIC$ criteria disagree for many ecosystems about the "best model" knowing that the Brody and the the linear regression are both 2-parameter models (i.e. in this case the complexity is not an issue for $R^2$)?

In the above formula, $\text{Rain}(t)$ denotes the accumulated rainfall from time to to time t and to compute the $R^2$, I used the following formula:

r2 <- mss/(mss+rss)
mss <- sum((fitted(model) - mean(fitted(model)))^2)


My guess is that the models needs to have the same form (which is not the case here: linear form vs exponential form) ) or nested so that their model selection with frequentist or Bayesian approaches such as the $AIC$ and $BIC$ criteria will give the same best result with a simple maximizing fit approach such as the $R^2$.

• I believe that for models of equal complexity R^2, AIC, and BIC should all give the same answers. There are some rather delicate issues with R^2 for nonlinear models ... – Ben Bolker Jun 12 '13 at 0:46
• @Ben That would be correct for Gaussian distributions of errors. Otherwise, $R^2$ will give different information than AIC/BIC. – whuber Jun 12 '13 at 11:58
• @Gung Sorry, I'm new here – Armel Jun 12 '13 at 17:25
• @whuber I don't understand why are you saying that R^2 should give different information. What I read is that R^2 doesn't take into account the complexity (number of parameters of the model) and in my case, the linear and Brody are both two-parameters models. – Armel Jun 13 '13 at 4:07

I would avoid the question altogether as secondary. The issue is the choice between two models, a linear fit and an exponential approach to an asymptote. The linear model may possibly be an acceptable fit depending partly on the range of your data, but a negative intercept implies negative productivity for zero rainfall and a positive intercept implies positive; the first is impossible and the second implausible; nevertheless the model might be tolerable with warnings.

Conversely, the exponential is evidently chosen to have the limiting behaviour of approaching the origin, but how well it fits in practice depends on whether the idea of an asymptote is correct and (simply, but crucially) on the impact of numerous other controls on productivity.

The central question of which model fits better should be clear from plotting data and model fits. Plotting residuals as well would do no harm.

There is scope for disagreement here, but I prefer to approach $R^2$ as the square of the correlation between observed and predicted. This is totally consistent with its use in linear regression and relatively easy to explain to non-statistical users. Either way, be aware that outside linear regression the many different definitions of $R^2$ may not agree.

There is some minor confusion in your question. Whether people use any kind of $R^2$, AIC or BIC does not really stamp them as frequentist or Bayesian. There could be qualifications and details attached to that, but arguments over frequentist and Bayesian approaches to inference arguably have no bearing on your question.

(LATER) It has become apparent that the observations are points on two curves, productivity so far this season and rainfall so far this season. I suppose this was implicit in the term "growth curve" but it was not obvious to me and causes severe qualifications to my original comments.

Cumulation imparts a dependence structure which renders any inferential machinery for $R^2$, AIC and BIC invalid, unless somehow calculations took the cumulation into account. Thus all inferential bets are off unless the cumulation is corrected for in the model assessment. (I've seen allusion to this problem of correlating cumulative curves in the glaciology literature.)

I was supposing that (rainfall, productivity) pairs for each location were from different seasons, which still leaves the possibility of season to season dependence, which in practice is likely to be weaker. To spell out the point, it seems that each location is being characterised by data from one season.

@Armel expresses disagreement with my statement that a curve of the form $a\cdot(1-\exp[-b\cdot\text{Rain}(t)])$ can not be S-shaped, but does not give any reasons. I am referring to the shape of the curve on a plot with Rain on one axis; the point then is one of elementary calculus that such a curve has no inflexion. Here what I understand by S-shaped is that an inflexion exists at which curvature changes sign. (The relationship between productivity and time could be much more complicated geometrically, depending on the dependence of cumulative rainfall on time.)

• yes I have several locations and I performing the two models for each of those location and the number of values (whic correpond to the number of measurement of rainfall and productivity) differ from a location to another one. What kind of plant growth model do you have in mind when you talk about model with qualitatively correct limiting behaviour? In my previous comment (who was edited by the moderator ), I said that we expected a S-shape curve. – Armel Jun 12 '13 at 17:31
• Your present exponential has no inflexion point and is not S-shaped. I meant simply that productivity tends to 0 from above as does rainfall. – Nick Cox Jun 12 '13 at 17:35
• When the time increases (for a specific wet season), the cumulative rainfall Rain(t) increases also and the productivity will finally level off towards an upper asymptote which (in the case of the brody model that I used) is the parameter a. I disagree with you when you said that this is not a S-shape model – Armel Jun 13 '13 at 3:24
• See additions to my answer. – Nick Cox Jun 13 '13 at 7:24
• You are rigth Nick. The Brody model doesn't have an inflexion point. I made this confusion because in many papers, it was considered as a S-shape model. I modify my question to clarify that I used the cumulative rainfall instead of the instantaneous rainfall. – Armel Jun 17 '13 at 21:37