0
$\begingroup$

I have a variance test based outliers detection algorithm. The algorithm is exposed with a visual application where the user can configure its parameters namely the multiplier.

The question is what is the range of values for the algorithm multiplier?

the algorithm basically is calculating the mean and standard deviation (sigma) of the data set and then comparing the dataset elements to the upper and lower bounds to flag an element as an outlier or not.

multiplier <- 2.3; 
upper_bound <- mean + multiplier * sigma; 
lower_bound <- mean - multiplier * sigma; 

the evolution of the two function upper_bound (red) and lower_bound(blue) with the multiplier values is as follows

enter image description here

But this graph doesn't give any clue on what range to define. Do you have any idea how to define this range?

$\endgroup$
  • $\begingroup$ Do you know anything about the shape of the distribution of the $Y$-values? $\endgroup$ – Michael M Oct 16 '13 at 17:34
  • $\begingroup$ actually, I don't have this luxury. The application should work for any distribution. $\endgroup$ – MedAli Oct 16 '13 at 17:36
  • $\begingroup$ Then you might go with Chebychev's inequality (the finite sample version) en.wikipedia.org/wiki/Chebyshev%27s_inequality It gives bounds for the probability to deviate more than $k$ standard deviations from the mean. If you know mean and standard deviation, you can derive the range based on this $\endgroup$ – Michael M Oct 16 '13 at 17:39
0
$\begingroup$

To find the outliers, you cannot use the distance of an observation to a model through a rule such as:

$$\frac{|\hat{\mu}-x_i|}{\times \hat{\sigma}},\;i=1,\ldots,n$$

if your estimates of $(\hat{\mu},\hat{\sigma})$ are the classical ones (the usual mean/standard deviation) because the fitting procedure you use to obtain them is itself liable to being pulled towards the outliers (this is called the masking effect).

One simple way to reliably detect outliers however is to use the general idea you suggested (distance from fit) but replacing the classical estimators by robust ones much less susceptible to be swayed by outliers. Below I present a general illustration of the idea. If you give more information about your specific problem I can append my answer to address the particulars of your situation.

An illustration: consider the following 20 observations drawn from a $\mathcal{N}(0,1)$ (rounded to the second digit):

x<-c(-2.21,-1.84,-.95,-.91,-.36,-.19,-.11,-.1,.18,
.3,.31,.43,.51,.64,.67,.72,1.22,1.35,8.1,17.6)

(the last two really ought to be .81 and 1.76 but have been accidentally misstyped).

Using a outlier detection rule based on comparing the statistic

$$\frac{|x_i-\text{ave}(x_i)|}{\text{sd}(x_i)}$$

to the quantiles of a normal distribution would never lead you to suspect that 8.1 is an outlier, leading you to estimate the $\text{sd}$ of the 'trimmed' series to be 2 (for comparison the raw, e.g. untrimmed, estimate of $\text{sd}$ is 4.35).

Had you used a robust statistic instead:

$$\frac{|x_i-\text{med}(x_i)|}{\text{mad}(x_i)}$$

and comparing the resulting robust $z$-scores to the choosen quantiles of a candidate distribution (typically the standard normal if you can assume the $x_i$'s to be symetrically distributed) you would have correctly the last two observations as outliers (and correctly estimated the $\text{sd}$ of the trimmed series to be 0.96).

$\endgroup$
  • $\begingroup$ In my case, I cannot assume that my distribution is normal or is not, and the outlier detection should work for all cases. In fact, I only have an idea about what data set to run the outlier detection on just after the user interaction. The variance test was the most general idea that doesn't require me to have previous knowledge of the distribution of my data. What I understand from your explanation is that I should replace comparing the distance with comparing the statistic, is that what you mean? and what is "med" and "mad" $\endgroup$ – MedAli Oct 16 '13 at 21:19
  • $\begingroup$ med is median and mad is the median absolute deviation. Even then, you will find the the quantile of the normal are tilted to the conservative side (for symmetric distributions) $\endgroup$ – user603 Oct 17 '13 at 7:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.