4
$\begingroup$

I have a sample of length $N$, mostly taken from a Gaussian distribution with unknown mean and variance. Of those $N$ samples, some proportion of them (typically less than 1-2%) are outliers, taken from another Gaussian distribution with a larger mean (larger typically by 3+ standard deviations of the more common distribution). I'm interested in estimating the parameters of the more common "background" distribution so that I can isolate the outliers with some prescribed probability of false alarm.

Using the sample mean/variance are the obvious ways to attack this problem, but they are not sufficiently robust to outliers for my purposes (i.e. the presence of the outliers results in an additive bias in the estimates). I can estimate the mean more robustly using different sample statistics like the median or mode that are more resistant to outlier values. However, I'm not sure how to approach the estimation of the distribution variance in a similar manner.

Is there an accepted approach for such a problem?

Edit: I decided to add some more details in response to the commenters below who suggested perhaps fitting a Gaussian mixture model to the data. I'm not sure whether that would be a good approach or not. I want to estimate the parameters of the background distribution in order to select from two hypotheses:

  1. There are no outliers; the sample consists of a Gaussian distribution with unknown parameters.

  2. There are some outliers, located at unknown locations in the sample. While their exact distribution isn't known, I posit that it is reasonable to approximate it as a Gaussian distribution with a significantly larger mean than the background. If this hypothesis is true, I would like to identify the locations of all of those outliers.

So, I guess I was a bit misleading in the original question. I should have said that the sample "may contain some proportion of outliers."

In the case where there are no outliers of interest, I'm not sure that a GMM would give me good results. My goal is to identify the parameters of the underlying Gaussian distribution so that I can identify outliers with a known, controlled type I error probability. I'm going to look for some more information on robust methods for estimating the distribution's scale.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ Do you know for sure that the outliers are taken from another Gaussian distribution? Do you have any hypothesis on the variance of this distribution, eg is it equal to the variance of the “background” distribution? Fitting a Gaussian mixture would be a solution. $\endgroup$
    – Elvis
    Jan 9, 2013 at 20:26
  • $\begingroup$ Maybe useful en.wikipedia.org/wiki/Robust_statistics $\endgroup$
    – Ziyuan
    Jan 10, 2013 at 0:05
  • $\begingroup$ (+1) On Elvis comment, if you know the distribution of the outliers, fitting a gaussian mixture would be a good solution. $\endgroup$
    – ThePawn
    Jan 10, 2013 at 4:23
  • $\begingroup$ @ziyuang: Thanks for the link. From an initial examination, that might be exactly what I'm looking for, specifically "robust measures of scale.' $\endgroup$
    – Jason R
    Jan 10, 2013 at 16:18

3 Answers 3

Reset to default
2
$\begingroup$

Multivariate Case

Some very applicable research has been done by Rousseeuw et. al. See the paper here that deals with the Minimum Covariance Algorithm (the paper is very readable, I would recommend reading it).

The papers deals with finding a subset of the data that minimizes the covariance matrix (hence find the data that is most Normal-looking). It is very fast, and many libraries are available in python and R

The univariate case

Although the above deals with multivariate data, there does exist formulas for the minCovDet problem for the univariate case. See this question's answer for some details.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ Could you please explain how this multivariate approach you reference should be applied to this univariate problem? $\endgroup$
    – whuber ♦
    Jan 9, 2013 at 20:32
  • $\begingroup$ ah, I misinterpreted the question, edits to reflect this. $\endgroup$
    – Cam.Davidson.Pilon
    Jan 9, 2013 at 22:06
0
$\begingroup$

As far as I know, there are some tests for outliers. One of them is actually described in this paper. It's based on median absolute deviation of the sample elements. There's also more sophisticated methods like chi square, cochran, dixon tests. It's actually depends on what you're using, but for R, I would recommend outliers package.

After you get your outliers removed you can calculate mean or variance in usual way with less or no bias.

Improve this answer
$\endgroup$
1
  • $\begingroup$ There will still be bias, because only a portion of the contamination distribution will typically be identified as outlying. $\endgroup$
    – whuber ♦
    Jan 9, 2013 at 20:31
0
$\begingroup$

Well, for a very simple outlier filter, here's some R code:

x = rnorm(500)
bp = boxplot(x, plot = FALSE)

if (length(bp$out) > 0)
        x = x[-match(bp$out, x)]
Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ The question is not about how to identify or exclude outliers: it is about how to estimate the parameters of the "background" distribution. Although the two may be related, a lot more work needs to be done once you believe you have an outlier detection method: then you have to assess the bias and efficiency of the resulting parameter estimates (presumably based on the dataset without the outliers). Please notice that most of the contamination distribution will not be identified as outlying because about half its values will be less than 3SD above the background mean. $\endgroup$
    – whuber ♦
    Jan 9, 2013 at 20:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.