Disclaimer: I'm looking for a bit of help as I'm only a simple neuroscientist and even working out what to google in this area is a tricky prospect. Here goes:

I have a set of data (3d positions in the brain). This can be allocated to known brain areas. What I want to know is - do these positions cluster nicely in to previously described brain regions, or not?

As an analogy: Imagine you have the longitude and latitude of every home in Europe. You want to understand where people live. You can simply look up the country or state/county/district within a country in which any given home is located.

If you run a cluster analysis, you'll find clusters like London that correspond entirely to one country - UK - but to multiple counties within the country - Essex, Hertfordshire etc. The city of Basel is nominally a Swiss city, but with suburbs in France and Germany. So in these cases, the cluster (the city) won't correspond well to a single classification (the country). In contrast, a city such as Bath is located in the UK, and also entirely within one subregion - Somerset

I'm looking for a way to quantify this discordance. To be clear, I don't want to train a supervised ML algorithm to recapitulate the classification, but rather to find out how an unsupervised clustering matches up.



I'm a machine learning scientist turned neuroscientist, so hopefully we'll be able to sort something out. There are basically two options here:

Option A: Direct cluster similarity estimation

There are some algorithms that can give you a direct similarity measure between two clusterings of the same data (the real anatomical regions on one side, the outcome of your unsupervised algorithm on the other). With this option you wouldn't know which region each cluster corresponds to, but you would get an absolute measure of similarity.

There are several approaches, but a simple one is to just calculate the mutual information between them -- the higher it is, the more similar the clusterings are.

Here are a couple of papers: this one with some simple and effective methods and this one with a review and comparison of several approaches.

Option B: Classification via clustering

Alternatively, you can split the process in two parts: 1) find a mapping between your true labels and your unsupervised cluster memberships; and 2) calculate how well those match as a standard classification evaluation. The advantage of this option is that you get a better grasp on what your unsupervised algorithm is doing, the disadvantage is that it's not as principled as the end-to-end solutions from option A.

Let's look at (2) first. There are piles of literature published on this which I can't possibly fit in a SE answer. I'll point you to the relevant Wikipedia section and informally suggest the Rand index as a reasonable candidate, but of course there are many more.

Now back to (1). If you can afford it (i.e. you're handling a relatively small number of categories), the exhaustive brute force approach is to just try all possible combinations and pick the one that maximises your metric of choice from step (2). If that's too expensive, you can do something simpler like majority voting: For each cluster your unsupervised algorithm spits out, pick the label that is most highly represented in that cluster and assign that label to the whole cluster.

Of course there are plenty variables and constraints that can heavily alter the problem, such as whether your unsupervised clustering algorithm gives you probabilities or "hard" cluster assignments, or whether you want to enforce that each label be represented by exactly one cluster. Hopefully now you have a few more keywords you can try searching to find what you need.

  • $\begingroup$ Great stuff. Many thanks! Option A is interesting - to continue my analogy, it would give you a single metric as to whether people in Europe cluster better by country than in Asia. But B is where I think my need is - external evaluation seems to be the sort of thing I'm looking for. The only issue is that I'm not looking for a true mapping from labels to membership (B1). Rand index and the others listed on that page require a concept of true negative/positive, which is linked to this - but which I can't define. But purity looks like a good place to start - thanks! $\endgroup$ – secondlevel May 25 '17 at 13:22
  • $\begingroup$ Glad it helped! I'm afraid you might be misunderstanding A -- you can't use it to compare people and country clustering in Europe and Asia because for them to work the clusterings have to be defined on the same data points. If you don't want a mapping between labels and membership then you're basically in option A by necessity. I would suggest you to try mutual information, which AFAIK does exactly what you want (and is in the External evaluation Wikipedia page). $\endgroup$ – Pedro Mediano May 25 '17 at 13:38
  • $\begingroup$ Also, if the answer actually answered your question, could you accept it? That will help people landing in this page in the future. $\endgroup$ – Pedro Mediano May 25 '17 at 13:39
  • 1
    $\begingroup$ +1 but I am not sure I understand the distinction between A and B in your answer. In particular, Rand index that you list in (B) is just a measure of similarity between two clusterings (or between an unsupervised clustering and known classes such as brain areas in this case) which is what you say is the goal of (A). CC to @secondlevel (Rand index does not require any "tru neg/pos" that you cannot define; Rand index can be computed between any two clustering results). $\endgroup$ – amoeba says Reinstate Monica May 25 '17 at 14:31
  • $\begingroup$ @amoeba Indeed Rand index could also be considered as a direct clustering similarity measure as described in (A). In (B2) I was mostly referring to similarity metrics that are not invariant to label permutation, such as classification rate. In that case you do need a reasonable mapping from cluster memberships to labels to obtain a meaningful comparison. $\endgroup$ – Pedro Mediano May 25 '17 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.