Evaluating a discriminative model is relatively easy: compare the predictions with ground truth, using cross-validation.

Unfortunately this strategy can't be used for generative models. Surely this problem has been tackled already?

  • 1
    $\begingroup$ You can look at the likelihood of a test sample under the generative model. $\endgroup$ – Arthur B. Nov 13 '14 at 15:31
  • $\begingroup$ @ArthurB.Could you detail a little bit? How do you go from the likelihood of a sample to a measure of goodness for the model? $\endgroup$ – static_rtti Nov 14 '14 at 9:01
  • 1
    $\begingroup$ The likelihood of a sample is a measure of goodness of the model, it's the best measure there is. $\endgroup$ – Arthur B. Nov 14 '14 at 14:58

Discriminative algorithms model P(Class|variables), whereas generative algorithms model P(Class,variables) = P(Class|variables)* P(variables). Hence, by modelling the joint distribution of the variable space, generative algorithms model the underlying process that 'created' your data.

My point in starting with this first paragraph is to note that generative algorithms have discriminative properties. Therefore, the same method of evaluating the predictive performance:

"compare the predictions with ground truth, using cross-validation."

applies to generative models, as well as discriminative ones.

However, as you imply, we can additionally asses the ability of the generative algorithms in modelling the underlying process that generates data. A commonly used group of metrics for this is "information theoretic scores" that derive from the idea of likelihood (log-likelihood). Below are some well-known information theoretic scores:

1- log-likelihood (LL) score

2- minimum description length (MDL) score

3- minimum message length (MML) score

4- Akaike Information Criterion (AIC) score

5- Bayesian Information Criterion (BIC) score

Note that 2, 3, 4, and 5 use some complexity penalisation factor over the LL score. This is good practice to combat over-fitting.

| cite | improve this answer | |
  • 1
    $\begingroup$ I am glad it has been helpful! $\endgroup$ – Zhubarb Nov 22 '14 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.