I have an ECDF of values that do not follow a particular distribution (thought they are slightly normal, they are not). And I wish to determine if a new observed value is significant or an outlier or not. How can I do this?
For example, I have the following distribution:
Value % of Observations ... -4 3% -3 3.5% -2 4% -1 6% 0 12% 1 5% 2 5% 3 4% 4 1% ...
With different distributions you set particular cutoffs or thresholds to signify an outlier, e.g. 3$\sigma$ for a normal distribution, but that doesn't help you classify ordinary values like a $2$ in the above case. Only 5% of observed values are $2$ but it's still quite common relative to the rest.
Is there some way to quantify the "outlierness" of a value? For example, if the value $10$ was observed I could say it is greater than $99\%$ of values possibly making it an outlier. However this won't work for non-outlier values, e.g. the value $0$ is greater than $\approx50\%$ of all observations but this doesn't tell me that $0$ is the most common value.
Note: I am not interested in fitting a particular distribution or anything. I just have a large dataset for which an ECDF can be evaluated and I want to know if a new observed value fits into this typically or is an extreme value.