Long Tail Distribution and importance in classification/prediction problems? I came across this interview question online: Explain what a long tailed distribution is and provide three examples of relevant phenomena that have long tails. Why are they important in classification and prediction problems? And I didn't really know how to answer the part about the classification/prediction problems? If I was to guess, I would say something along lines of ensuring the training data is stratified if I am draw samples from such a distribution? Would that be correct?
 A: There's a use of "long tail" in classification that is closely related to the use popularised in marketing. The book "The Long Tail" argued that there were books, movies, etc, that individually were in very low demand but collectively were in high demand, and that this would be important for businesses such as Amazon that could afford to have very large numbers of distinct items on their virtual shelves.  The long-tailed distribution in this context is the distribution of demand over categories, ordered by decreasing demand.
In classification with large numbers of classes, the 'long tail' problem occurs when there is a substantial aggregate probability for classes that individually have very low probability.  Good classification accuracy would require good prediction for these classes, which is hard because there will be few training examples.  The goal is to, somehow, improve prediction in the low-data classes by transfer of information from the high-data classes. Simple things to do would be to oversample the 'tail' classes or to try to pool them into higher-probability classes. Some more complicated approaches are described here
A: Prediction problems:  Assume a situation you can model with some kind of regression(-like) model. An error distribution with long tails will tend to produce many outliers, that is, observations far from the bulk of the data. That makes predictive modelling harder, and could indicate need for robust methods. For some examples Building a predictive model, regression with a long right tail,   Fat Tailed risks: do they get fatter when we linearize non-linear systems?   and  this arXiv paper Building robust prediction models for defective sensor data using Artificial Neural Networks.
For classification it is different. The response is categorical, so cannot have long tails. Then long tails must apply to the predictors, and the problems will be different. For the moment I defer that to others.
