1
$\begingroup$

I am using a KNN anomaly detection approach, where the distance to my nearest neighbor is an indication for an anomaly.

I am wondering how I can normalize the score between 0 and 1. I can use a test dataset without anomalies to get the normal data distribution. When I calculate the z-score it's not bounded between 0 and 1. The sigmoid function applied on the z-score returns too high values.

Is there some statistical approach based on the data distribution that returns a probability value that my distance value (or z-score) is an outlier?

Improve this question
$\endgroup$
4
  • $\begingroup$ why not score / max(score) ? That'll be in 0-1 (a distance is always positive) $\endgroup$
    – user603
    Commented Oct 16, 2018 at 20:39
  • $\begingroup$ because I don't know before what is my max value as there can always be a new anomaly which has a higher value. $\endgroup$
    – MikeHuber
    Commented Oct 16, 2018 at 20:52
  • 1
    $\begingroup$ How do you know that the scores arising from the sigmoid function are too high? $\endgroup$
    – Dave
    Commented May 4, 2022 at 1:33
  • $\begingroup$ "Is there some statistical approach based on the data distribution that returns a probability value that my distance value (or z-score) is an outlier?" -- how do you define "outlier"? $\endgroup$
    – Tim
    Commented May 5, 2023 at 9:22

1 Answer 1

Reset to default
0
$\begingroup$

If you make the assumption that the z-scores are normally distributed, it is easy to convert the score to the probability of encountering a point that far away or farther p(x >= abs(z)). For example, in R you could write: Score = 2 * (1 - pnorm(abs(z)))

Improve this answer
$\endgroup$
1
  • $\begingroup$ Even if the data are Gaussian (which is often doubtful when one is tempted to remove outliers) the z-score do not follow a Gaussian distribution. I guess it would be closer to student distribution but this is not even the case. $\endgroup$
    – TMat
    Commented May 5, 2023 at 5:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.