Dataset and outlier question I'm facing a data dilemma. I would like to have a real data illustration for an outlier detection rule i'm working on. The outlier detection rule 
targets datasets of continuous (not necessarily multivariate elliptically 
distributed) variables with $\verb+ncol+\leq \verb+nrow+$ and $\verb+ncol+\leq 15$.
But here is the catch: the actual outliers should be known (as i'm 
trying to compare the performance of this rule to existing ones). 
My initial though was to use a classification dataset 
(using some observations from class "A" as the data and some observations 
from $\bar{A}$ as the outliers)....but what guarantees do we have that there 
are no outliers among the data labeled as belonging to class 'A' (without 
some guarantee that there are no outliers among the data labeled as members
 of class 'A' the rest of the comparison procedure would become suspect).
When looking at the papers in the robustness literature i found no examples 
of real data comparison...so any fresh idea is welcome (both in terms of general methodological ideas as well as actual examples of dataset i would have 
overlooked where the above caveat could be reasonably excluded).
 A: There are three approaches:


*

*Use synthetic data. Then of course you know the outliers.

*Add outliers to real data by randomization methods.

*Use real data that has a rare class, assuming that the rare instances (should be < 1% of the data set, but you can also downsample the class!) are all outliers.
The first option is of course very limited. The chances of overfitting your method are really high; because most likely you will be designing the data set and kind of outliers with the same intuition you used for your algorithm.
The second option still suffers from this. The non-outlier data is real data, but the outliers are of a very specific kind: they stem from a particular type of "manipulation" or "transmission errors".
The third will have the smallest choice of data sets available - because you need a data set that has rare observations. Downsampling sometimes helps, but you will encounter a number of data sets where your algorithm just does not detect the outliers. Plus, this approach is also quite naive: it assumes that the majority class does not contain outliers. In any real data set that I have seen every class will have outliers within the class, too. So do not expect your method to be able to go to 90% on these data sets. If you can improve from 70% to 80%, then your method already works quite well. Anything beyond 80% may be indicative of some bias IMHO.
When reviewing outlier detection papers, I consider any result higher than 0.80 to be suspicious: either the data set was too much designed for the algorithm, the algorithm parameter were systematically tweaked to find the best possible result, or maybe the result is just fake altogether.
