Can one truly fight outliers with more data? Consider a noisy classification problem and a training set where we may have outliers. As we collect more data, we gather more samples from both inliers as well as outliers. So is it then reasonable to assume in the general case that one can technically fight outliers by collecting more data? I have read this claim before, but it seems counter-intuitive that this would be the case.
 A: As with most questions about outliers, I don't think there's an easy answer. It will depend on your situation.
For instance, if you are modeling the relationship between race and income, and, just by chance, Michael Jordan answers your survey, then more data can help because it can clarify the situation, but, since very few people get to "be like Mike" you would need millions of cases to fully remove the effect of having him in your sample.  (And N = 1000 will make it much clearer than N = 100 that income is not remotely normally distributed). 
On the other hand, if one of your variables is height, then more people will smooth things out because, while Jordan is tall, he's not so freakishly tall that he shouldn't show up in a data set. 
Sometime a variable you think is normal isn't, at least not in the population you are dealing with.  I found this with weight in adult humans: It's right skew in many populations.  A large sample shows this.  A small one doesn't.
It will also depend on what method you are using for classification and whether the outlier is like the closest inliers.  If you were, for instance, using trees to classify people into "professional basketball players" and "others" then MJ being an outlier on height would be no problem.  Nor his being an outlier on income.  But if you happened to get a very tall person who was very tall but not a basketball player, that would make the tree weird and more data might not help (or it might - I think it depends on the algorithm).
For cluster analysis, the effect of an outlier (and more data) is likely to be different for single linkage and complete linkage.
And so on.    
A: If your outliers occur because of natural outliers in the distribution, then yes, your estimations will become more stable with more data. Let's say you're using a logistic regression, in a theoretical case where the outcome depends on an observed variable with some normally distributed noise. Outliers will come in the noise, and accidentally you hit a value in the noise that's three standard deviations outside of the normal distribution. Then this will have an effect on the estimated intercept and coefficient in your model, and this will be much stronger if you don't have much data. With more data, the effect of these accidents will average out. 
But this is a very theoretical case. If your outliers arise from something more weird, or something that you didn't include in the model but is actually structural, or maybe they are arbitrarily large, then more data might not be able to save your model. 
A: Outliers are those samples that deviate from the pattern(s) of most data. Usually, the number of outliers are far less than normal samples. So, for each new sample, it is more likely that it is a normal sample than outlier. Thus I think, gathering more data can help fight outliers. 
