I'm using Euclidean distance as a metric to compare two sentences for similarity while clustering them using my custom incremental KMeans algorithm. The current threshold value I'm using is 0.7 which seems to work well in the data which I have. I'm just wondering if there is a standard threshold value which will work optimally in all the cases?
1
-
1$\begingroup$ indeed there is, and it is that same threshold that you have already found: that which seems to work well with the data. $\endgroup$– John MaddenCommented Apr 4 at 1:38
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
There is no standard threshold, the threshold choice will have to relate to our particular application. It will also be strongly affected by any pre-processing step we take. Unless there is a particular reference application we follow and recreate on our data, comparing two values is most likely nonsensical.
-
$\begingroup$ I'm seeking sources such as articles or research papers that support the establishment of thresholds for my data. Since my dataset comprises various forms of communication like messages, chats, and emails, and also it changes overtime, I need to ascertain appropriate thresholds. Do you have any articles or research papers that provide guidelines or thresholds for similarity scores based on different types of data, such as chats, emails, etc.? For instance, a threshold of 0.6-0.7 for chat data and 0.7-0.75 for email data. $\endgroup$– sanjay MCommented Apr 5 at 6:12