3
$\begingroup$

I have around a million sets of 8 datapoints. I would like to rank these sets of 8 data points. I am trying to identify sets where there is a single extreme value and the other 7 datapoints are of similar value. For example, $\{0.2, 0.19, 0.27, 0.17, 0.3, 0.21, 0.22, 0.98\}$ would be considered a high-ranking set. Values can be positive or negative.

It is important that the single extreme value is larger than the other values. i.e. an extreme negative value where all other points are of similar positive value is not considered a high-ranking set.

I am currently using the variance of each set but I was hoping somebody could lend some insight into perhaps a more appropriate technique.

Thanks

Improve this question
$\endgroup$
1
  • $\begingroup$ How extreme the one high outlier is seems to be important. What about using the distance from the lower quartile to the high outlier? If you want to implement that idea in R, look at boxplot.stats. // It is true that the distance from the high outlier from the mean would have a heavy influence on the sample SD. // Suggest you try both ideas and see which matches your intuition for best 'ranking`. (If it's down to SD and variance, suggest you use SD instead of variance to stay with original units.) $\endgroup$
    – BruceET
    Commented Aug 20, 2019 at 5:52

2 Answers 2

Reset to default
0
$\begingroup$

You may use confidence intervals for your need, and to count the number of points that are only out of the upper bound (as you are interested only in the positive extremes). This method uses the variance that you've intuitively used but takes into account the side/sign of the extremes.

You can also calibrate the size of the confidence interval for your specific needs (i.e the volume of extremeness) by switching the confidence term with a different multiplier with the standard deviation to control the upper bound.

There are some theories that are used in the field of statistical process control, that are trying to supply tools for similar needs as yours. These technics are designed to find outliers/faults in the domain of manufacturing processes. Examples for such technics are: Control Charts, Six-Sigma which are designed to a similar use with few samples per set.

All of these technics, are taking into account the number of samples used and their standard deviation (some are assuming normality and others not).

The Box-Plot is also another visual technic to describe the extremes and it is using less assumption on the samples population to set the outliers bounds (percentile-based limits).

You may also use a mixed-method for your need: using the median as the centre of the confidence interval, and a fixed size for the interval size (which defines the limit to be an outlier) instead of the std/variance.

In a case that you are having more samples per set of points, you may use other moments of the set of samples such as their Skewness to detect the right tail.

Improve this answer
$\endgroup$
3
  • $\begingroup$ What parameter would you compute confidence intervals for and, given that confidence intervals don't answer questions about either outliers or ranking, why would that approach be relevant or reliable? $\endgroup$
    – whuber ♦
    Commented Aug 20, 2019 at 11:12
  • $\begingroup$ I've suggested the CI as an approach for filtering values between 2 boundaries. With some modifications to the CI such as not using alpha but a constant size for the boundaries to detect the extreme values (you could base it on the sample size and the std if you have a larger sample size, as done with the 6-Sigma approach) $\endgroup$
    – Romid
    Commented Aug 20, 2019 at 14:10
  • $\begingroup$ The CI is not relevant: it answers a completely different question and has none of the properties needed for this purpose. You might have some kind of a tolerance interval in mind, but it isn't clear. $\endgroup$
    – whuber ♦
    Commented Aug 20, 2019 at 15:10
0
$\begingroup$

A simple approach would be the following:

  • Remove the largest value out of the eight in your set.
  • Compute a measure of central location and a measure of dispersion for the remaining eight.
  • Calculate how many dispersions the largest value is away from the center.

This approach compares the largest value to the spread of the others. Two sets of eight could have the same largest value, but different spreads in the other seven, so they should be assigned different scores. Which this approach does automatically.

For a measure of the center, you could use the mean, or a trimmed mean (removing the smallest and the largest value from your seven), or the median.

For a measure of dispersion, you could use the standard deviation, or the range of the seven data points, or a trimmed range, like the distance between the second smallest and the second largest of your seven data points.

As BruceET writes, I would recommend that you play around with a couple of alternatives to see which ones match your intuition.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Interesting. I ended up using a similar more basic apporach of essentially comparing the difference between the largest value and the mean of the reamining seven values. Is this too naive? If I understand correctly, you're suggesting measuring this "difference" in units used to describe the dispersion of the remaining seven. I assume your approach would be better because it's possible that two sets have the same max value, and the same mean for the remaining 7 values, but the two sets have different variance. To which your technique would prioritize the set with less variance? $\endgroup$
    – Harry Stuart
    Commented Aug 20, 2019 at 12:50
  • $\begingroup$ It kind of depends on what your data are actually measuring, and on what you are trying to do. Suppose you might have seven data points between 4.9 and 5.1, and an eighth point at 7; and a second set with points between 1004.9 and 1005.1 and a singleton at 1007. Should these two sets be treated alike? Only you can say. My proposal would treat them alike; comparing the largest value with the mean only would treat them differently. And yes, my approach is predicated on treating sets differently if they agree on the mean and the max, but differ on the variance. $\endgroup$
    – Stephan Kolassa
    Commented Aug 20, 2019 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.