I have an ECDF of values that do not follow a particular distribution (thought they are slightly normal, they are not). And I wish to determine if a new observed value is significant or an outlier or not. How can I do this?

For example, I have the following distribution:

Value         % of Observations
-4                   3%
-3                   3.5%
-2                   4%
-1                   6%
0                   12%
1                    5%
2                    5%
3                    4%
4                    1%

With different distributions you set particular cutoffs or thresholds to signify an outlier, e.g. 3$\sigma$ for a normal distribution, but that doesn't help you classify ordinary values like a $2$ in the above case. Only 5% of observed values are $2$ but it's still quite common relative to the rest.

Is there some way to quantify the "outlierness" of a value? For example, if the value $10$ was observed I could say it is greater than $99\%$ of values possibly making it an outlier. However this won't work for non-outlier values, e.g. the value $0$ is greater than $\approx50\%$ of all observations but this doesn't tell me that $0$ is the most common value.

Note: I am not interested in fitting a particular distribution or anything. I just have a large dataset for which an ECDF can be evaluated and I want to know if a new observed value fits into this typically or is an extreme value.

  • $\begingroup$ Your 'data' are impossible to interpret without knowing proportions of values outside $(-4,4)$ and without any idea of sample size. // Many distributions typically show 'outliers' (according to various criteria). Some characteristically show a large number of 'outliers' in any sample of moderate size. An observation should be deleted from a sample only if there is external evidence that it is impossible or an obvious error. (Examples: Man 5 meters tall, negative concentration, documented as data entry error or due to failure of measurement device.) $\endgroup$
    – BruceET
    Oct 5 '19 at 18:58
  • $\begingroup$ This is a familiar question: search our site for "prediction interval." $\endgroup$
    – whuber
    Oct 5 '19 at 19:52
  • $\begingroup$ @BruceET I am not looking to delete observations and I have thousands of observations making up the above distribution. I just want to get some idea of how rare/extreme a new observation is relative to the thousands others I've seen already. $\endgroup$
    – guy
    Oct 5 '19 at 22:44
  • $\begingroup$ @whuber can you give a bit more info. Prediction intervals are more used in regression as far as I know and for prediction purposes. I am not trying to predict anything. I am just trying to compare how unlike or like a new observation is to a distribution that I already have. $\endgroup$
    – guy
    Oct 5 '19 at 23:38
  • 1
    $\begingroup$ A little like that. That's about the best you can do unless you have a more detailed model of how the values might be changing over time, in which case the concept remains the same but you would construct the prediction interval using a regression or time series procedure or whatever is appropriate for the model. $\endgroup$
    – whuber
    Oct 6 '19 at 17:12

You can tell how extreme a new value is by comparing it to the distribution in the ways hat you already are doing.

But that doesn't tell you the probability that it came from this distribution. The data alone can't tell you that, unless you have an alternate distribution and know that the new value comes from one or the other.

  • $\begingroup$ Right I think I wish I could do a hypothesis test but I can't since I don't have an alternative distribution. $\endgroup$
    – guy
    Oct 6 '19 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.