1
$\begingroup$

A list of numbers and I want to remove the extremes on both ends.

The standard deviation is calculated: 26.3 (rounded to 1 decimal)

Originals
[1.1,87.8,97.2,6.8,8.1,10.8,4.9,8.1,1.9,2.8,1.2,1.2,8.7,5.8,2.1,1.9,1.4,1.2,6.6,1.4,1.3,1.5]

Each Original v.s. Standard deviation
[0.04,3.34,3.7,0.26,0.31,0.41,0.19,0.31,0.07,0.11,0.05,0.05,0.33,0.22,0.08,0.07,0.05,0.05,0.25,0.05,0.05,0.06]

For this case, I want to define the numbers of below criteria as outlier:

  1. either greater than 3 times of standard deviation
  2. or small than 0.05 times of standard deviation

enter image description here

Is this a reasonable way to consider and define outlier?

(This proposed method will be applied to lists of numbers may/not normally distributed.)

Thank you.

$\endgroup$
  • 2
    $\begingroup$ (1) Possible that no outliers should be removed. They may give important info about what's happening. (2) I think it's a bad idea to use SD for outlier ID because SD itself is heavily influenced by outliers. (Maybe OK to check each observations by some SD method, provided that observation is removed before SD is calculated. (3) Boxplot outliers use IQR = Q3 - Q1 to measure variability; IQR is relatively robust against outliers, so that method may generally be better. (4) Why remove outliers? Obvious data entry error? Eqpt failure? Or they just annoy you? $\endgroup$ – BruceET Oct 18 at 5:21
  • 1
    $\begingroup$ @BruceET, thank you for details. (3) is a brilliant point. (2) is a teaching to me. for (4), there're suspicious data entry errors (not able to trace root cause) so I want to conduct outlier removal. $\endgroup$ – Mark K Oct 18 at 5:50
  • $\begingroup$ Data analysis needs to know more than that you have "a list of numbers". Two values look higher on any graph worthwhile, but what you do know of their context if anything? The two values are close to 100 and the others are much closer to 0; are they percents and if not what could high values be? $\endgroup$ – Nick Cox Oct 18 at 12:35
  • $\begingroup$ @Nick Cox, thank you for the comment. This sample is taken from a case, where human entry error and misuse of input happened. $\endgroup$ – Mark K Oct 18 at 13:50
  • $\begingroup$ I believe you but that doesn't really impart much flavour here. $\endgroup$ – Nick Cox Oct 18 at 14:11
2
$\begingroup$

Graphical comment. If I've correctly captured your data, here is the boxplot from R. Your title says 'both ends', but I see boxplot 'outliers' only in the upper tail. Do you have any explanation how they might have arisen.

boxplot(x, horizontal=T, col="skyblue2", pch=19)

enter image description here

Only two 'outliers' (no tied ones).

length(boxplot.stats(x)$out)
[1] 2

Even small samples from an exponential distribution often show 'outliers' in the right tail, which are a natural 'feature' of exponential distributions. Would be a mistake to remove them from the sample. The population SD below is $\sigma=12.$ The SD of all 22 observations is about 11.4 (already a slight underestimate of $\sigma.$ The SD of the 20 'non-outliers' is only about 5.2 (serious underestimate).

set.seed(123)
y = rexp(22, 1/12)
boxplot(y, horizontal=T, col="skyblue2", pch=19 )
sd(y)
[1] 11.38477
sd(sort(y)[1:20])
[1] 5.205213

enter image description here

For example, certain members of the Weibull family of distributions and members of the Pareto family have even heavier right tails, hence more 'outliers' on the high side.

$\endgroup$
  • $\begingroup$ thank you for the answer. The question was about to find a way also suitable for other lists, where extremely small numbers are in. for example, [0.01,0.005,2,15,12,6.8,8.1,10.8,4.9,8.1,7,2.8] $\endgroup$ – Mark K Oct 18 at 6:50
  • 2
    $\begingroup$ Based on my experience, it seems to me you're altogether too willing to remove 'outliers'. Saying you have data entry errors and you're unable to find the cause seems self-contradictory. If you're talking about the equivalent of a man 12' (4m) tall, then that's an outlier I'd remove. Not familiar with your data, so don't want to judge. Or if the dataset says an IQ score is 191 and the actual test paper has 91 written on it. But don't expect even honest real data to be 'tidy'--like the data in many elementary textbooks.) $\endgroup$ – BruceET Oct 18 at 7:05
1
$\begingroup$

Outliers are generally defined as:

  • values lower than Q1 - (1.5 * IQR) on the lower end
  • values higher than Q3 + (1.5 * IQR) on the top end

More detail here: https://newonlinecourses.science.psu.edu/stat200/lesson/3/3.2

$\endgroup$
  • $\begingroup$ thank you for your help to the question. Hope you don't mind I choose another which provide more details to the topic. :) $\endgroup$ – Mark K Oct 18 at 7:07
  • 1
    $\begingroup$ Not a problem at all. $\endgroup$ – william3031 Oct 18 at 7:40
  • $\begingroup$ More wrong than right, I'd say, for a bundle of reasons. (1) Outliers might exist in many different spaces (different dimensions -- univariate, bivariate, multivariate --, categorical data too, etc.) so on that ground alone this is hardly general. (2) Precise rules of thumb for outliers miss the main point, which is that identifying outliers requires background knowledge and a statistical sense of what is expected. One size fits all criteria mislead more than they help. (3) This is a perversion of the original intent in Tukey's work, at most a criterion for flagging data points to think about. $\endgroup$ – Nick Cox Oct 18 at 9:51
  • $\begingroup$ (4) "generally defined" depends on a literature survey, and it's my reading against yours, but I wouldn't summarize general statistical advice in this way. Much better advice in the question and comments of @BruceET. $\endgroup$ – Nick Cox Oct 18 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.