Recently I got a recommendation from a reviewer of my article to calculate AIC (or BIC) for the brnn model. However, there is no straightforward way of doing that. Could anyone suggest how to do it?

alligator = data.frame(
  lnLength = c(3.87, 3.61, 4.33, 3.43, 3.81, 3.83, 3.46, 3.76,
               3.50, 3.58, 4.19, 3.78, 3.71, 3.73, 3.78),
  lnWeight = c(4.87, 3.93, 6.46, 3.33, 4.38, 4.70, 3.50, 4.50,
               3.58, 3.64, 5.90, 4.43, 4.38, 4.42, 4.25))

model_brnn = brnn(lnWeight ~ lnLength, data = alligator)

This is not a package I'm very familiar with, so I'll describe my thought process generally -

To calculate AIC, you need to specify a likelihood function - likelihood functions using sum-of-squares are pretty common, and the residuals sum-of-squares is included in the output as out$Ed (according to ?brnn). The also include "effective number of parameters" as out$gamma. If you used weights or other features, you should take those in to account.

In R, if you implement a logLik method for brnn, AIC will use it automatically:

logLik.brnn <- function(object) structure(
  -(object$n / 2) * log(object$Ed),
  nobs = object$n,
  df = object$gamma,
  class = 'logLik'

Then you can compare two models using AIC, etc:

model_brnn2 = brnn(lnWeight ~ lnLength, data = alligator, neurons=2)
model_brnn4 = brnn(lnWeight ~ lnLength, data = alligator, neurons=4)


This is probably good enough for a random revise-and-resubmit, but could probably use some deeper thought - you could incorporate the coefficients in the likelihood, etc.

Also be sure to set the RNG seed, I get slightly different results each run otherwise.

  • $\begingroup$ Unless the conditions are met for proper application of AIC, BIC, I would take it with a grain of salt. Are they? $\endgroup$ – Carl Jan 4 '18 at 4:59
  • $\begingroup$ Agreed - I just wanted to demonstrate the mechanics of calculating AIC in R for an add-on package - Jerry is on his own to specify the likelihood function and interpret the AIC correctly. I'm not familiar enough with brnn to say, for example, that changing neurons won't lead to extra rows being dropped and leading to invalid comparisons. $\endgroup$ – Neal Fultz Jan 4 '18 at 16:54

Typically, reviewers ask for AIC and BIC without caring whether or not it is meaningful. What I do in those situations is provide meaningful analysis as an answer to the reviewer's request without elaboration. For example,

Reviewer 1: Please provide AIC (BIC) to show model comparison.

Author response: Adjusted $R^2$ of the logarithms of the data were compared between methods. Significance testing showed blah blah blah...

  • $\begingroup$ Thank you, this would probable satisfy reviewers. But I would still like to know how to go for AIC… is it actually possible? $\endgroup$ – JerryTheForester Jan 3 '18 at 7:38
  • 1
    $\begingroup$ Under certain constrained conditions, yes. Do these apply to brnn? Not sure. For one thing, residual structure has to be accounted for and maximum likelihood applied to at least two different models of the same data. This question reduces to "Is brnn a maximum likelihood method, and, are the residuals in a recognizable pattern for which that maximum likelihood model is designated." Quite a mouthful that. $\endgroup$ – Carl Jan 3 '18 at 8:23
  • $\begingroup$ Again, thank you very much. I believe now it is my task to explore the rest and come up with the answer. $\endgroup$ – JerryTheForester Jan 3 '18 at 8:42
  • $\begingroup$ If you can find this out definitively, come back here and put it in as an answer. My guess is that it does not apply, but then, it is just a guess. May help to look at other AIC and BIC Q/A on this site, there are lots. $\endgroup$ – Carl Jan 3 '18 at 9:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.