44
$\begingroup$

With respect to the unsupervised learning (like clustering), are there any metrics to evaluate performance?

$\endgroup$
  • $\begingroup$ possible duplicate of Evaluation measure of clustering (without having truth labels) $\endgroup$ – steffen Dec 9 '13 at 9:42
  • 4
    $\begingroup$ I think this question is more general that that one, so I am voting to leave this open. $\endgroup$ – Peter Flom Dec 9 '13 at 11:36
  • $\begingroup$ I have had the same question you have, and have some (as of yet not completely read) references are relevant: A Survey on Internal Validity Measure for Cluster Validation L.Jegatha Deborah, R.Baskaran, A.Kannan and Cluster Validity Measurement Techniques Ferenc Kovács, Csaba Legány, Attila Babos $\endgroup$ – kasterma Dec 9 '13 at 15:52
41
$\begingroup$

In some sense I think this question is unanswerable. I say this because how well a particular unsupervised method performs will largely depend on why one is doing unsupervised learning in the first place, i.e., does the method perform well in the context of your end goal? Obviously this isn't completely true, people work on these problems and publish results which include some sort of evaluation. I'll outline a few of the approaches I'm familiar with below.

A good resource (with references) for clustering is sklearn's documentation page, Clustering Performance Evaluation. This covers several method, but all but one, the Silhouette Coefficient, assumes ground truth labels are available. This method is also mentioned in the question Evaluation measure of clustering, linked in the comments for this question.

If your unsupervised learning method is probabilistic, another option is to evaluate some probability measure (log-likelihood, perplexity, etc) on held out data. The motivation here is that if your unsupervised learning method assigns high probability to similar data that wasn't used to fit parameters, then it has probably done a good job of capturing the distribution of interest. A domain where this type of evaluation is commonly used is language modeling.

The last option I'll mention is using a supervised learner on a related auxiliary task. If you're unsupervised method produces latent variables, you can think of these latent variables as being a representation of the input. Thus, it is sensible to use these latent variables as input for a supervised classifier performing some task related to the domain the data is from. The performance of the supervised method can then serve as a surrogate for the performance of the unsupervised learner. This is essentially the setup you see in most work on representation learning.

This description is probably a little nebulous, so I'll give a concrete example. Nearly all of the work on word representation learning uses the following approach for evaluation:

  1. Learn representations of words using an unsupervised learner.
  2. Use the learned representations as input for a supervised learner performing some NLP task like parts of speech tagging or named entity recognition.
  3. Assess the performance of the unsupervised learner by its ability to improve the performance of the supervised learner compared to a baseline using a standard representation, like binary word presence features, as input.

For an example of this approach in action see the paper Training Restricted Boltzmann Machines on Word Observations by Dahl et al.

$\endgroup$
  • 11
    $\begingroup$ +1 "how well a particular unsupervised method performs will largely depend on why one is doing unsupervised learning in the first place" pretty much sums it up. Don't go looking for a some magic number that you can somehow use to justify a given result without actually interpreting the result. $\endgroup$ – Marc Claesen Dec 9 '13 at 15:43
  • 1
    $\begingroup$ I would also add that using a supervised approach as a proxy to how well an unsupervised approach works doesn't require the discovery of new features. For example, clustering doesn't learn new features, yet clustering is often used to improve the prediction accuracy of a supervised learner, with the added benefit of explaining why this may be so. For example, k-means clustering can produce k predictions that are each improved by way of exploiting the discovered structure and compression from clustering. See ttic.uchicago.edu/~shubhendu/Papers/clustering_bagging.pdf $\endgroup$ – Cybernetic Oct 8 '16 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.