1
$\begingroup$

I have two probabilistic model, $A$ and $B$. I am using MLE to estimate the parameters in both models. Then I run the following 10-fold cross validation. I randomly chopped the data set into 10 chunks, in each iteration, I use MLE to fit the 2 models on the training data set and get my estimates, so I will have 2 training Loglikihoods for 2 models. Then, I apply these 2 models with their estimates of the parameters to the test data set and get 2 test loglikihoods.

I go through all 10 iterations, then I could (1) take the average of 10 test loglikihoods for each model, then compare these 2 average; Or I could (2) sum all the 10 test loglikihoods for each model, then compare the sum.

I think in order to tell one model is better than the other, both of these 2 methods are equivalent. However, I feel that the first one, i.e., take the average is more meaningful, while the 2nd one (i.e., take the sum) is just mathematically equivalent but not very meaningful. But I couldn't tell why.

Can someone provide some comments?

$\endgroup$
0
$\begingroup$

Mathematically, this is equivalent since the number of folds is the same for both models.

log likelihoods are per se not a very intuitive (though in some cases valid) scale for comparison of models. Therefore, it does not matter very much in your case I would think.

In other cases, it is plain misleading to sum performance metrics over cross validation folds. 10 error rates of 2% in each fold are not a 20% error rate on the data set and ten 98% accuracies are not one 980% accuracy.

$\endgroup$
  • $\begingroup$ I have a question: if I use the error rate as performance metric, then usually, the in sample training error rate will be less than the test error rate right? But if I use loglikelihood (LL) as performance metric, then due to the difference size of the training and test data set, the magnitude of the test LL will be much smaller than the training LL, right? Then how can I create a fair comparison of training Vs. test using LL just like using error rate? $\endgroup$ – KevinKim Oct 28 '16 at 19:26
  • $\begingroup$ You might divide both by the number of samples in the set I suppose. $\endgroup$ – David Ernst Oct 28 '16 at 19:27
  • $\begingroup$ Good point, and if I did everything correctly, it should be also the case that the training LL/# data should be "better" than the test LL/# data right? $\endgroup$ – KevinKim Oct 28 '16 at 19:33
  • $\begingroup$ That's how it usually works. I'm not an expert on log likelihood though. $\endgroup$ – David Ernst Oct 28 '16 at 19:35
  • $\begingroup$ Is there a terminology of this procedure you mentioned, i.e., a statistic divided by the number of observations? Something like "standardized" or "rescaled"? $\endgroup$ – KevinKim Oct 29 '16 at 1:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.