# Dataset and outlier question

I'm facing a data dilemma. I would like to have a real data illustration for an outlier detection rule i'm working on. The outlier detection rule targets datasets of continuous (not necessarily multivariate elliptically distributed) variables with $\verb+ncol+\leq \verb+nrow+$ and $\verb+ncol+\leq 15$.

But here is the catch: the actual outliers should be known (as i'm trying to compare the performance of this rule to existing ones).

My initial though was to use a classification dataset (using some observations from class "A" as the data and some observations from $\bar{A}$ as the outliers)....but what guarantees do we have that there are no outliers among the data labeled as belonging to class 'A' (without some guarantee that there are no outliers among the data labeled as members of class 'A' the rest of the comparison procedure would become suspect).

When looking at the papers in the robustness literature i found no examples of real data comparison...so any fresh idea is welcome (both in terms of general methodological ideas as well as actual examples of dataset i would have overlooked where the above caveat could be reasonably excluded).

• What is the matter with simulating datasets?
– whuber
Oct 20 '12 at 19:53
• Expanding on @whuber 's question: If you have real data then you do not know what is an outlier and what isn't; you don't know if something is a data entry error or isn't; you don't know if that person whose height was recorded as 17.8 meters was really 1.78 meters or was some other error. Simulating lets you control all this. Then you can test things and know if you get the right answer. Oct 20 '12 at 20:18
• @PeterFlom:> sure, I have plenty of simulation results. I should have mentioned that they all unequivocally suggest one of the methods is much better than the rest. So naturally, it makes me curious to see if this is also the case in a more "natural" setting. Oct 20 '12 at 20:48
• Look for data sets where one variable is income; these will nearly always have outliers. GSS might be a source. Oct 20 '12 at 20:50
• General Social Survey. They collect data on all sorts of things. Oct 20 '12 at 22:32

There are three approaches:

1. Use synthetic data. Then of course you know the outliers.

2. Add outliers to real data by randomization methods.

3. Use real data that has a rare class, assuming that the rare instances (should be < 1% of the data set, but you can also downsample the class!) are all outliers.

The first option is of course very limited. The chances of overfitting your method are really high; because most likely you will be designing the data set and kind of outliers with the same intuition you used for your algorithm.

The second option still suffers from this. The non-outlier data is real data, but the outliers are of a very specific kind: they stem from a particular type of "manipulation" or "transmission errors".

The third will have the smallest choice of data sets available - because you need a data set that has rare observations. Downsampling sometimes helps, but you will encounter a number of data sets where your algorithm just does not detect the outliers. Plus, this approach is also quite naive: it assumes that the majority class does not contain outliers. In any real data set that I have seen every class will have outliers within the class, too. So do not expect your method to be able to go to 90% on these data sets. If you can improve from 70% to 80%, then your method already works quite well. Anything beyond 80% may be indicative of some bias IMHO.

When reviewing outlier detection papers, I consider any result higher than 0.80 to be suspicious: either the data set was too much designed for the algorithm, the algorithm parameter were systematically tweaked to find the best possible result, or maybe the result is just fake altogether.

• +1. But I think you exaggerate the limitations of (1): you are limited only by your skills and imagination. For instance, you can take two real datasets and form a mixture, using one to "contaminate" the other with [known] "outliers." The singular advantage of (1) is that you can use it to understand--in a precise quantitative way--how the underlying distribution and the amount, size, and distribution of any outliers all affect the results of a statistical procedure. That insight is not possible with method (3) and can be only semi-quantitatively obtained with (2).
– whuber
Jan 15 '13 at 14:44
• I'd call that a variant of 2 (semi-real data sets) what you are suggesting. You do a very specific type of contamination, and assume that there are no natural outliers in the original data. Jan 15 '13 at 15:02