1
$\begingroup$

I'm working on a problem, in which the learning algorithm consistently produces high accuracy on the different samples it is tested on, but generally has a rather unsatisfactory adjusted Rand index (that is, negative or close to 0).

However, I'm unable to find a valid explanation to this. That is, the model is predicting with high accuracy, but it is still worse than random assignment in general? I doesn't seem to add up to me.

I apologise for not being able to provide more detail on what I'm working on, but I hope this would be enough to open a discussion around this topic.

$\endgroup$
0
$\begingroup$

The adjusted Rand Index is a harsh measure, since every clustering error is counted many times, once for each other object that the erroneously classified object is paired with. It can be a useful training objective but it's not the one I would report to the client! KL divergence is a reasonable metric.

However, your question makes me wonder: are you making a classifier or a clusterer? If a classifier, then accuracy is a useful metric (but you must be careful if the sets are unbalanced - few people currently have cancer, so a test that says everyone is healthy is very accurate but also useless). But then the Rand Index would not tell you much.

If it is a clusterer then the Rand Index is useful, but there is no such thing as accuracy. Accuracy is "the model assigns the right label" in a classifier. In a clustering problem, you are looking for clusters, but there is no natural label for each cluster.

$\endgroup$
1
  • $\begingroup$ Indeed the problem I'm working on in computer vision can be seen as a classification of pixels, but also a clustering of pixels; both of which may be valid perspectives to study the problem from. Thank you for your answer, it makes sense to me now. I will check the KL divergence too! $\endgroup$
    – Jakinduria
    May 19 at 8:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.