I recently read a paper in which the authors claim that in order to compare the forecasting performance of two non-nested models, models A and B, a valid procedure is to fit models A and B on the same data set, and compare the average likelihood of the fitted models computed on the hold-out data set. All that is required is that models A and B are expressed as probability densities for the same variable. I am using the language of the paper: strictly speaking the quantities being compared are fitted models evaluated at data points, not likelihoods. Accepting this likelihood as the forecasting accuracy metric, this method for checking forecast accuracy has some intuitive appeal, although I have misgivings. Can these likelihoods be meaningfully compared without some sort of normalization? The out of sample probabilities will not add up to one. I'm not able to come up with a straightforward example in which this test would give a spurious result.

update: I was able to produce a simple example in which direct comparison of out of sample likelihoods gave a misleading result: let the dependent variable y be a linear trend plus normally distributed error term, and let the explanatory variable x be a linear trend plus independent normally distributed error term. Generate 100 points for y and x. Model A is a linear regression, while for model B I used a regression with Student t-distributed errors (with two degrees of freedom). I trained on the first 50 points and tested on the second set of 50 points, and repeated with training and test sets interchanged. I repeated with three choices of variance for the data generating process. Model B gave higher average out of sample likelihoods in all cases. This example is a bit contrived but does illustrate my concern.

  • $\begingroup$ Can you list the paper in question? $\endgroup$ – gung - Reinstate Monica Dec 21 '12 at 3:56
  • 1
    $\begingroup$ Gung,the paper is "Model Selection Criteria Using Likelihood Functions and Out-Of-Sample Performance", by Norwood, Ferrier, and Lusk. The url is ageconsearch.umn.edu/bitstream/18947/1/cp01no01.pdf $\endgroup$ – user18010 Dec 21 '12 at 15:59
  • $\begingroup$ If you use a probability density function PDF (for example Gaussian Normal PDF), then, the probabilities will not add to one. However, you would be able to compare different values of PDF and give a decision on the query. Finally, you need CDF in case you want the probabilities to add to 1. I am not sure of the way the probability is computed in the paper but you may check. $\endgroup$ – soufanom Dec 23 '12 at 3:41
  • $\begingroup$ please see the update for an example $\endgroup$ – user18010 Jan 14 '13 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.