2
$\begingroup$

I have subjects with about 200 features each. These feature vectors are stored in a vector database, where similarity searching with Euclidean distance is used to find subjects that are similar to a given vector.

Unfortunately, I have no ground truthing data. Do I have any options for validating that my similarity scores are accurate? I've considered using clustering algorithms to get an alternative view of which vectors are similar/dissimilar, but because something like KNN also uses distance metrics I don't feel like I'd actually be validating anything. Apologies for asking such an open-ended question, but I would appreciate any thoughts here. Thanks in advance.

$\endgroup$

1 Answer 1

3
$\begingroup$

No. Given only a set of pairwise distances, there is no way to discern whether or not those distances actually reflect reality, or represent any kind of useful measure.

Suppose you give me a set of pairwise distances and want to know if they are good or not. Regardless of the distances you gave me or anything about their distribution or numeric values, I could tell you that the distances perfectly reflect "similarity" as I'd like to define it, or I could tell you that your result was actually computed from random numbers and is totally useless as a measure of similarity. As a very simple example, suppose I tell you the distances between Cities A, B, and C - nothing about those numbers can tell you if those distances are accurate or not. At best you could evaluate if you even have a distance metric or not (like if we find that the A-C distance is greater than A-B plus B-C).

If you have some kind of sample labeling, you could compute some clustering metrics like the silhouette, leveraging the cluster assumption that samples of the same class should be "more similar" than samples of different classes. Here, the class label is used as a sort of gold standard, though, as it's being used as a measure which the derived distances should match. Without any clue as to which samples should be more or less similar, we have no means of evaluating the accuracy of the distance metric.

$\endgroup$
2
  • $\begingroup$ Thanks for the reply. This is unfortunately what I assumed would be the case. If I were to hand-label data, I may be able to get something somewhat accurate. Would the idea from there be to calculate silhouette scores to see if my label clusters are properly differentiated? $\endgroup$
    – T_d
    Commented May 29 at 17:03
  • $\begingroup$ @T_d That's the idea. If you were to compute silhouette using the distance metric and known labels, you should ideally see that cases of the same class are more similar than cases of different classes. If your average silhouette metric comes out negative, it indicates between-class similarity is higher than within-class - that could be a problem with the distance metric, or with the class labeling, or with the underlying assumption that class members are in fact similar to one another. $\endgroup$ Commented May 29 at 17:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.