0
$\begingroup$

For example, the lists can be something like: $$\{1.123213, 5.154543, 2.134121, 7.34534, 12.223432, 8.16571, 100.45645, 222.423\}$$ I want to remove $\{100.45645, 222.423\}$.

$$\{232.123213, 323.154543, 232.134121, 123.34534, 222.223432, 8545.16571,\\ 4335.45645, 1222.423\}$$ I want to remove $\{8545.16571, 4335.45645, 1222.423\}$.

The size is not fixed, and the range may vary. Are there any ways to remove these "abnormally large" values from my lists?

$\endgroup$
2
  • 2
    $\begingroup$ A method advised by Tukey as a way of flagging "outliers" was to compare their values to 1.5*IQR above and below Q3 and Q1 respectively. With 25% of your sample consisting of "outliers", I don't think this method (or other methods) will be reliable. I generally don't advocate automating such processes since data can be weird and "outliers" can dramatically increase your power to detect trends. If you have data quality issues, flagging outliers can be good practice, but you may want to focus on feeding in higher quality data and reviewing it more closely. $\endgroup$ – AdamO Mar 20 '14 at 17:32
  • 1
    $\begingroup$ An outlier is only an outlier with respect to some model (less formally, some expectation about what you should see). You have given examples of what you regard as outliers in two cases, but I see no clear way to generalize. In that second case is 800 an outlier? Is 500? If the 323 was 380, or 260 would that change your mind about 500? $\endgroup$ – Glen_b Mar 20 '14 at 20:20
3
$\begingroup$

I've added the tag to your question, as that's how I understand your problem. Plenty of methods exist for outlier detection; I'll focus on this, since I assume deletion of detected outliers is trivial for you.

Tukey's fences are the simple, conventional statistics I've been taught to use for univariate outliers. I'd be eager to upvote any better answers here and welcome any further critique of this common method. Fence stats are, as I've said, very simple: the upper and lower bounds are defined by the inter-quartile range, multiplied by 1.5, which is then added to the 3rd quartile and subtracted from the 1st. E.g., in R:

  1. Your first vector of data, X:
    • Upper fence: 1.5*(quantile(X,.75)-quantile(X,.25))+quantile(X,.75) $=79$
      • This identifies the two outliers you intended, 100.45645 and 222.423.
    • Lower fence: quantile(X,.25)-1.5*(quantile(X,.75)-quantile(X,.25)) $=-40$
  2. Your second vector of data, Y:
    • Upper fence: 1.5*(quantile(Y,.75)-quantile(Y,.25))+quantile(Y,.75) $=4657$
      • This is more conservative than you were, only identifying 8545.16571 as an outlier.
    • Lower fence: quantile(X,.25)-1.5*(quantile(X,.75)-quantile(X,.25)) $=-2427$

The lower fences seem unneeded, but I've retained them for demonstration.

I see @AdamO has already raised a good point about your data and why this method may have failed. What justification do you have for calling two and three elements of your 2×8 observations abnormally large? If your data are sampled from larger populations of interest, do you really know what is normal for those populations' distributions? Consider this sample of a random normal distribution with $\mu=$ X's median and X's $SD$ after squaring (set.seed(1); sort(round(rnorm(8,median(X),sd(X))^2))): $$\{496,1143,1745,2143,3262,3400,4373,17923\}$$ To reiterate, this is just a simulated sample of a squared normal distribution. There are no outliers in the square roots of these data (I checked), but Tukey's upper fence identifies 17923 as an outlier in the squared distribution. Here's the probability density function () of this distribution to give you a feel for what it looks like (plot(density(rnorm(100000,median(X),sd(X))^2),xlim=c(1500,45000))):

You're probably not going to want to trust Tukey's upper fence with a strongly, positively skewed (skewness $=2.8)$ distribution like this (somewhat similar to a $\chi^2$ distribution BTW, but probably more skewed and kurtotic than most). Tukey's upper fence identifies the top 7.5% of observations from this distribution as outliers, which is a bit much to shave off the top, and not really justified, considering that these observations are sampled from the same distribution as all the non-"outliers".

Outliers should generally be suspected of belonging to different distributions before disregarding them, and ideally you'd have more than just a univariate statistical basis for such a suspicion. For example, if you're sampling the local baseline of tectonic activity, it's probably safe to say you've sampled from a different distribution during moments when a herd of elephants is stampeding past your detector (which, let's say isn't designed to control for elephant stampedes). If these observations are univariate outliers (or bivariate outliers, considering elephant stampede? as a second binary variable you'd probably want to record, and hopefully wouldn't observe too often), I have no reason to tell you not to exclude them...but if you haven't seen any elephants lately, your high observations might not be proper outliers regardless of what Tukey's fences say. You should seek stronger justifications. If you only have these data and no idea what else might be affecting them, and if they're paired observations, you could consider methods of multivariate outlier detection, but I doubt this will be a big improvement.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.