Outlier detection in very small sets

Question

I need to get as accurate as possible a value for the brightness of a mainly stable light source given twelve sample luminosity values. The sensor is imperfect, and the light can occasionally "flicker" brighter or darker, which can be ignored, hence my need for outlier detection (I think?).

I've done some reading up on various approaches here and can't decide on which approach to go for. The number of outliers is never known in advance and will often be zero. Flicker is generally a very large deviation from the stable brightness (enough to really mess with any average taken with a big one present), but not necessarily so.

Here's a sample set of 12 measurements for completeness of the question:

295.5214, 277.7749, 274.6538, 272.5897, 271.0733, 292.5856, 282.0986, 275.0419, 273.084, 273.1783, 274.0317, 290.1837

My gut feeling is there are probably no outliers in that particular set, although 292 and 295 look a little high.

So, my question is, what would be the best approach here? I should mention that the values come from taking the euclidean distance of the R G and B components of the light from a zero (black) point. It would be programmatically painful, but possible, to get back to these values if required. The euclidean distance was used as a measure of "overall strength" as I'm not interested in the color, just the strength of output. However, there's a reasonable chance that the flickers I mentioned have a different RGB composition to the usual output.

At the moment I am toying with some sort of function that will repeat until a stable membership of allowed measures is reached by:

1. Finding the standard deviation
2. Putting everything outside say 2 SDs into an ignore list
3. Recalculating the average and SD with the ignore list excluded
4. Re-deciding who to ignore based on the new average and SD (assess all 12)
5. Repeat until stable.

Is there any value in that approach?

All comments gratefully accepted!

Answer 1

Outliers in small samples can always be very tricky to detect. In most cases actually I would advocate that if you feel that your data are not bluntly corrupted, an "outlierish" value might not be problematic and its exclusion might be unreasonable. Probably using robust statistical techniques will be more sensible and closer to a middle-ground solution. You have a small sample; try to make every sample point count. :)

Regarding your suggested approach: I would not hastily enforce a normality assumption to your data with a 68-95-99.7 rule on them (as you seem to somehow do with your 2SD heuristic rule). Chebyshev's inequality for once assumes a 75-88.9-93.8 rule on them which is clearly less rigid. Other "rules" also exist; the Definition and detection section in the Outlier lemma in wikipedia has a bundle of heuristics.

Here is another one: A free book reference I have come across on the matter, NIST/SEMATECH e-Handbook of Statistical Methods, presents the following idea by Iglewicz and Hoaglin (1993): Use modified $Z$-scores $M$ such that:

$M_i = .6745(x_i-\tilde{x})/MAD$

where $\tilde{x}$ is your median and MAD is the median absolute deviation of your sample. Then assume that absolute values of $M$ above 3.5 are potential outliers. It is a semi-parametric suggestion (as most of them are, the parameter here being the $3.5$). In your example case it would marginally exclude your 295.5 but clearly retain your 292.6 measure... (For what's worth I wouldn't exclude any values out of your example case.)

Again, given you have a really small sample, if you believe that your sample is not obviously corrupted (a human 9'4" tall), I would advise you not to exclude data hastily. Your "suspected outliers" might be uncorrupted data; their use could actually assist rather than harm your analysis.

Answer 2

Point the first - it may be worth going back to rgb color. It's rarely good to throw away data, and the magnitude of the rgb vector isn't the only way to represent brightness - perceived brightness is different, as is value in HSV.

But putting that to one side and dealing with the data you do have, have you considered forming this as a classification problem instead of a modelling one, and doing some machine learning? You have an input, which is a vector with 12 real values in it (the brightness readings). You have an output, which is a vector of 12 binary values (1=inlier, 0=outlier). Get several sets of brightness reading and hand label them yourself, showing which brightness reading in each set is an inlier/outlier. Something like this:

x1 = {212.0, 209.6, 211.5, $\ldots$ , 213.0}, y1 = {1,0,1, $\ldots$,1}

x2 = {208.1, 207.9, 211.2, $\ldots$ , 208.2}, y2 = {1,1,0, $\ldots$,1}

x3 = {223.4, 222.9, 222.8, $\ldots$ , 223.0}, y3 = {1,1,1, $\ldots$,1}

$\ldots$

Then, run the whole lot through a classifier of some sort:

• You could use a single classifier which outputs 12 different binary values - a neural network would let you set this up pretty easily.
• Or, you could use a standard binary classifier (e.g. SVMlite) and train 12 different models, one classifying whether each element of the output is an inlier/outlier.

And you're done! No need to fuss trying to find the 'rule' which separates inliers from outliers yourself. Just get a few sets of data which look sensible and let the machine do that for you :)

~~~

EDIT: Incidentally, your proposed method, where you iteratively fit a gaussian then classify each sample more than 2 standard deviations away as an outlier, looks a lot like an expectation maximisation algorithm. Something like this:

• A single gaussian component (modelling the inliers)
• A uniform background component (the outliers)
• Some prior probability of each that depends in a non-obvious way on the width of the gaussian (the 'classify at 2 standard deviations' rule).
• Hard classification at the expectation step.

If you do go down that route it may be worth googling for EM algorithms and checking what assumptions you're building into your model.

Answer 3

Dixon's Q-test for outliers in very small datasets seems fits well to this kind of situation:

http://en.wikipedia.org/wiki/Dixon%27s_Q_test

http://www.chem.uoa.gr/applets/AppletQtest/Text_Qtest2.htm