I'm reading a paper that compares different probabilistic models using the log likelihood of held-out data. This is just... wrong, correct? There's no meaningful way to compare the LL between two different models? If I'm correct, what is the right way to do this comparison?

  • 1
    $\begingroup$ Take a look at the proper scoring literature, where this is done extensively (evaluating the predictive likelihood on test data leads to the scoring rule sometimes called the "log-score"). $\endgroup$ – Chris Haug Jun 19 '20 at 19:16

You have to find out what they mean by "held-out".

Usually, the "held-out" set is a validation set, a separate dataset that is used to estimate the test set error. For example, it is useful to estimate the test set error to do model selection on different hyperparameters and/or early stopping*. That is very common because ML algorithms often have many hyperparameters that are not fit during learning (the "K" in K-means, the regularisation coefficient in logistic regression, the number of iterations for iterative algorithms in general...).

However, "held-out" literally means "taken out of" so depending on the context, I could see how the authors of the paper could denote the test set by it.

You can trust the comparison if the authors do not use the held-out set for learning parameters or choosing hyperparameters. That is the right way to do this comparison.

* early stopping can be seen as nothing more than hyperparameter search on the duration of training.

  • $\begingroup$ Comparing LLs for similar models for things like choosing hyperparameters or model complexity (e.g. "Do I add this hierarchical component?") make sense. What if these are completely different models? For example, does it make sense to compare the LL of probabilistic PCA with the LL of a VAE? $\endgroup$ – gwg Jun 19 '20 at 16:59
  • $\begingroup$ It does make sense, why not? $\endgroup$ – bomzh Jun 19 '20 at 17:02
  • $\begingroup$ My instinct was because the likelihood is un-normalized. For example, the likelihood depends on the dataset size. But I suppose this doesn't matter when comparing on a test set of fixed size. $\endgroup$ – gwg Jun 19 '20 at 17:34
  • $\begingroup$ If you know the test set size, you can always average the LL over the test set size. I don't know why you would have several test sets. If the test sets are drawn from the same distribution, it is wasteful: you could merge them and have a better estimate of the true generalisation error, or simply add more training/validation data. If they are not drawn from the same distrib, they measure different things (OOD generalisation for example) and it is not fair to compare models on different test sets. $\endgroup$ – bomzh Jun 19 '20 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.