1
$\begingroup$

I have 1200 data points and I'm building 6 different time series models. One with 1200 data, another model with 1100, 1000,..., 700 and then I would choose the best model but I don't know how to select between different data lengths. I'm doing so because my data have time based trend and I don't know how many data should I use to make the models.

I've tried to use $$AIC=-2\log(L)+2k$$

where $k$ = # of Parameters, $L$ = Likelihood, $N$ = Data Length

To avoid AIC always selecting the model with minimal data length, I've used the logarithm in the base N, but it still prefers to choose the model with lower N.

Improve this question
$\endgroup$
1
  • $\begingroup$ Possible duplicate: stats.stackexchange.com/questions/251056/…. As per the comments, there are subtleties involved in this endeavor. It would be much easier for you if you could simply train the different models on the same data, rather than datasets of different size. Is this not an option? $\endgroup$
    – jwimberley
    Commented Jan 9, 2017 at 2:02

1 Answer 1

Reset to default
3
$\begingroup$

You can't use AIC to compare models that were fit to different data. Comparing AICs amounts to determining which model results is less information loss relative to the true model. When you have different data sets, in addition to the scale issues you point out, there may be different "true" models, so the comparison doesn't make sense regardless of whatever you're doing to fidget with the base of the logarithm.

If you really must compare these models, I suggest some absolute measure of fit, such as an $R^2$ type measure (i.e. squared correlation between the fitted values and the true values) or mean squared error. If the data are binary you could use area under the ROC curve, which has a nice interpretation (the probability that a randomly selected "positive" will have a larger predicted value than a randomly selected "negative")

Improve this answer
$\endgroup$
5
  • $\begingroup$ yes the prediction is binary, I'll try to measure the area under the ROC (subject that I did not know). Higger is better? $\endgroup$
    – Bernardo Braga
    Commented Jan 9, 2017 at 12:50
  • $\begingroup$ I found this BriorScore wich I guess is more adequate of what I'm doing: en.wikipedia.org/wiki/Brier_score $\endgroup$
    – Bernardo Braga
    Commented Jan 9, 2017 at 13:46
  • $\begingroup$ That basically looks like mean squared error. Seems reasonable. $\endgroup$
    – gammer
    Commented Jan 9, 2017 at 14:21
  • $\begingroup$ But there will be no problems with this huge difference between data lengths? the model should 'unfaithfully' adapts better with lower data. I know I'm dividing by N but still... $\endgroup$
    – Bernardo Braga
    Commented Jan 9, 2017 at 15:18
  • $\begingroup$ No, I think it's fine. The MSE estimates produced from the larger data sets will be more accurate of course, but they are still comparable, unlike the AICs. You can try to get uncertainty estimates for the error rates and incorporate that into your comparison if you want. $\endgroup$
    – not_bonferroni
    Commented Jan 9, 2017 at 18:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.