Multivariant, multi class outlier detection of single points

Question

The problem:

I have a set of means and covariance matrices of a number of multivariant normal distributions, each having a class label. Then I get single data points, one after the other, to which I want to assign one of the class labels or the label 'None', if it is unlikely that the point belongs to any of the given classes (could be drawn from the corresponding distribution).

My idea:

Simplified to the one-dimensional case I would compute the distance of the point to each mean (in terms of corresponding standard deviation units) and assign the label of the closest mean or 'None' if all distances exceed the e.g. 2-sigma boundary (to catch 95% of the probability).

What I have so far:

All I got so far comes from this book. There the same idea is applied to multi-dimensional data using the Mahalanobis distance, but with the difference that you cannot say 'outlier if distance is greater than some fixed value' because you have different deviations in (and between) each dimension. Instead the authors use the set of points (or to be more precise the size of the set of), for which they want to decide which points are outliers, to estimate the rejection distance threshold using a chi-squared distribution. The problem is that this only seems to be reasonable for large sets of points, what directly leads to my questions:

1. Can I apply the same approach if my point set is of size 1 and still get good results?

2. If not, can the approach be adapted to the situation where you have single points?

3. If not, is there a way to label single observation given the parameters of multiple multinomial distributions with an additional rejection class?

Thanks in advance.

Answer 1

Although the book doesn't say so very clearly, I think the question it is trying to answer, in the section that you cited, is what it even means for a point in the tail of a Gaussian distribution to be considered an outlier in the first place.

In order to make things a bit more concrete, consider the following. In a series of random draws from a one-dimensional Gaussian, roughly 99.73% of the points that you draw will lie within 3 sigma of the mean. But suppose you have a sample size of $n=10^{6}$ draws. This means that 997,300 points will fall within the 3 sigma limit, but 2700 points will fall outside of it. Are those 2700 points outliers? Well, no, not really: when you make a million random draws on a Gaussian, you're basically guaranteed to get at least a few points outside of the 3 sigma limit, and those points are no more weird, unusual or unexpected that the ones which fell inside of the boundary; this is simply a natural consequence of making a million draws. In the way that the book has conceived it, true outliers are only those points which are so far away from the mean (in terms of sigma, or its higher dimensional analogue Mahalanobis distance) that we would have expected to see no points that far into the tails, even given a large sample size. True outliers, in the way that I think the book seems to be defining it, are those which appear not to have even been drawn from a Gaussian pdf at all, but some other pdf with fatter tails.

In this conception, any sample size may be used to define outliers using the same methodology that the authors have suggested, even $n=1$. However, you must take great care to be sure that you are defining $n$ correctly for your particular problem. In your case, I would say that $n$ is the size of your entire data set (i.e., the total number of points that you eventually plan to test), not just the number of points that you are testing within a single test iteration.