1
$\begingroup$

I'm developing some software to allow users to perform various kinds of clustering on some data using a pairwise distance matrix (k-medoids is the main method). I would like to allow the user to tune a parameter p that changes how sensitive the method is to outliers, and I've found a particular log transformation of the distances d that is doing what I want, at least when p = 20: y = log_10(p*d + 1). It acts to "suppress" large values, such that the algorithm differentiates little between medium and large distances; IE the penalty to include an outlier in a cluster is lessened.

However, no matter the value of p the function never approaches the original distances. I want the clustering to be performed on the regular distances when p is at its "resting" value (maybe 1, maybe 0, depending on the transformation function used), and I want the clustering to be less and less sensitive to outliers as p is set further and further from this resting value.

Does anyone have a suggestion for such a transformation? Distances will always be positive, but can be very small and can be zero. I don't care if the function has a horizontal asymptote or not; I'm not particularly fussy beyond the idea that it approaches no transformation at some particular value.

Below is a little mock-up graph showing what I'm looking for. The dotted line is no transformation (p = 1), and the suppression becomes more extreme as p is increased to 4.

Mock graph

$\endgroup$

1 Answer 1

0
$\begingroup$

In case anyone comes across this and is interested, the function I ended up using was:

y = (p*d + 1)^(1/p) - 1

This works to change the behavior of my clustering, such that outliers are more permissible in clusters when p > 1.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.