Performance metrics to evaluate unsupervised learning

Question

With respect to the unsupervised learning (like clustering), are there any metrics to evaluate performance?

Answer 1

In some sense I think this question is unanswerable. I say this because how well a particular unsupervised method performs will largely depend on why one is doing unsupervised learning in the first place, i.e., does the method perform well in the context of your end goal? Obviously this isn't completely true, people work on these problems and publish results which include some sort of evaluation. I'll outline a few of the approaches I'm familiar with below.

A good resource (with references) for clustering is sklearn's documentation page, Clustering Performance Evaluation. This covers several method, but all but one, the Silhouette Coefficient, assumes ground truth labels are available. This method is also mentioned in the question Evaluation measure of clustering, linked in the comments for this question.

If your unsupervised learning method is probabilistic, another option is to evaluate some probability measure (log-likelihood, perplexity, etc) on held out data. The motivation here is that if your unsupervised learning method assigns high probability to similar data that wasn't used to fit parameters, then it has probably done a good job of capturing the distribution of interest. A domain where this type of evaluation is commonly used is language modeling.

The last option I'll mention is using a supervised learner on a related auxiliary task. If you're unsupervised method produces latent variables, you can think of these latent variables as being a representation of the input. Thus, it is sensible to use these latent variables as input for a supervised classifier performing some task related to the domain the data is from. The performance of the supervised method can then serve as a surrogate for the performance of the unsupervised learner. This is essentially the setup you see in most work on representation learning.

This description is probably a little nebulous, so I'll give a concrete example. Nearly all of the work on word representation learning uses the following approach for evaluation:

1. Learn representations of words using an unsupervised learner.
2. Use the learned representations as input for a supervised learner performing some NLP task like parts of speech tagging or named entity recognition.
3. Assess the performance of the unsupervised learner by its ability to improve the performance of the supervised learner compared to a baseline using a standard representation, like binary word presence features, as input.

For an example of this approach in action see the paper Training Restricted Boltzmann Machines on Word Observations by Dahl et al.

Answer 2

The most voted answer is very helpful, I just want to add something here. Evaluation metrics for unsupervised learning algorithms by Palacio-Niño & Berzal (2019) gives an overview of some common metrics for evaluating unsupervised learning tasks. Both internal and external validation methods (w/o ground truth labels) are listed in the paper.

Hope this helps!