4
$\begingroup$

I want to use AIC to compare three candidate models (labeled by m), each having K_m parameters. However, I have M datasets over which I can make the comparison. My ultimate goal is to report the "relative goodness" of each of the three models for a single fit. How to I make use of the multiple datasets?

One idea is to find the Akaike weights (See Anderson and Burnham, 2002) for each dataset and average the weights over all the M datasets (perhaps, weighting by the number of points in each dataset?).

Another approach would be to say that, for model m, I have a single M*K_m parameter model that I fit over the M datasets. In this case the AIC value, ACIC_net, is the sum of the AIC values for a fit to each of the M datasets. In this case, I would use AIC_net to compare the three models.

How should I proceed?

Improve this question
$\endgroup$
1
  • $\begingroup$ Robbie's approach makes sense to me. Though I wonder if you have already ruled this out since you don't mention it? Is the model nonlinear? You have variance issues with different datasets (indeed, even with different number of parameters) in nonlinear models that can affect model comparability. That may be reason enough to stack the datasets together - you could include a variable for the type of dataset and any metadata about the datasets that could be explanatory. $\endgroup$
    – robin.datadrivers
    Commented Dec 5, 2014 at 15:50

3 Answers 3

Reset to default
7
$\begingroup$

You can basically only compare AIC scores when you use the same dataset. Within the same dataset you can't even compare AIC scores if let's say one model is fitted on 70 records and the other model on 69 records (because of a missing value in one of the variables).

Can't you combine all the datasets into one single large dataset? Then you can compare your different models without any problems while using AIC scores.

Improve this answer
$\endgroup$
4
$\begingroup$

Coming from the field of behavior analysis (specifically matching law and theory), Navakatikyan (2007) displays a way to calculate AICc and BIC over multiple datasets:

$$AICc=\sum_{i=1}^N \left(n_i \cdot log_e\frac{RSS}{n_i}\right) + 2KN\left(\frac{n_t}{n_t-KN-N}\right)$$

and

$$BIC=\sum_{i=1}^N \left(n_i \cdot log_e\frac{RSS}{n_i}\right) + KNlog_e(n_t)$$

where $N$ = number of data sets, $i$ = the index of the data set (thus $n_i$ = number of data points in the $i^{th}$ dataset), $K$ = number of parameters for the model, and $n_t$ = total number of data points combined over the sets.

He cites these equations from a personal communication with a B. McArdle, whom I presume is Dr. Brian McArdle at University of Auckland (Associate Professor, Department of Statistics).

Here's the citation in APA format:

Navakatikyan, M. A. (2007). A model for residence time in concurrent variable interval performance. Journal of the Experimental Analysis of Behavior, 87, 121-141. doi: 10.1901/jeab.2007.01-06

Improve this answer
$\endgroup$
0
$\begingroup$

If the datasets are about the same size, then I think it's ok to average Akaike weights. Otherwise, I would fit each model on the "global dataset" because you can't just weight the goodness of fit statistics by the number of points.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.