People often talk about dealing with outliers in statistics. The thing that bothers me about this is that, as far as I can tell, the definition of an outlier is completely subjective. For example, if the true distribution of some random variable is very heavy-tailed or bimodal, any standard visualization or summary statistic for detecting outliers will incorrectly remove parts of the distribution you want to sample from. What is a rigorous definition of an outlier, if one exists, and how can outliers be dealt with without introducing unreasonable amounts of subjectivity into an analysis?
As long as your data comes from a known distribution with known properties, you can rigorously define an outlier as an event that is too unlikely to have been generated by the observed process (if you consider "too unlikely" to be non-rigorous, then all hypothesis testing is).
However, this approach is problematic on two levels: It assumes that the data comes from a known distribution with known properties, and it brings the risk that outliers are looked at as data points that were smuggled into your data set by some magical faeries.
In the absence of magical data faeries, all data comes from your experiment, and thus it is actually not possible to have outliers, just weird results. These can come from recording errors (e.g. a 400000 bedroom house for 4 dollars), systematic measurement issues (the image analysis algorithm reports huge areas if the object is too close to the border) experimental problems (sometimes, crystals precipitate out of the solution, which give very high signal), or features of your system (a cell can sometimes divide in three instead of two), but they can also be the result of a mechanism that no one's ever considered because it's rare and you're doing research, which means that some of the stuff you do is simply not known yet.
Ideally, you take the time to investigate every outlier, and only remove it from your data set once you understand why it doesn't fit your model. This is time-consuming and subjective in that the reasons are highly dependent on the experiment, but the alternative is worse: If you don't understand where the outliers came from, you have the choice between letting outliers "mess up" your results, or defining some "mathematically rigorous" approach to hide your lack of understanding. In other words, by pursuing "mathematical rigorousness" you choose between not getting a significant effect and not getting into heaven.
If all you have is a list of numbers without knowing where they come from, you have no way of telling whether some data point is an outlier, because you can always assume a distribution where all data are inliers.
You are correct that removing outliers can look like a subjective exercise but that doesn't mean that it's wrong. The compulsive need to always have a rigorous mathematical reason for every decision regarding your data analysis is often just a thin veil of artificial rigour over what turns out to be a subjective exercise anyway. This is especially true if you want to apply the same mathematical justification to every situation you come across. (If there were bulletproof clear mathematical rules for everything then you wouldn't need a statistician.)
For example, in your long tail distribution situation, there's no guaranteed method to just decide from the numbers whether you've got one underlying distribution of interest with outliers or two underlying distributions of interest with outliers being part of only one of them. Or, heaven forbid, just the actual distribution of data.
The more data you collect, the more you get into the low probability regions of a distribution. If you collect 20 samples it's very unlikely you'l get a value with a z-score of 3.5. If you collect 10,000 samples it's very likely you'll get one and it's a natural part of the distribution. Given the above, how do you decide just because something is extreme to exclude it?
Selecting the best methods in general for analysis is often subjective. Whether it's unreasonably subjective depends on the explanation for the decision and on the outlier.
I don't think it is possible to define an outlier without assuming a model of the underlying process giving rise to the data. Without such a model we have no frame of reference to decide whether the data are anomalous or "wrong". The definition of an outlier that I have found useful is that an outlier is an observation (or observations) that cannot be reconciled to a model that otherwise performs well.
There are many excellent answers here. However, I want to point out that two questions are being confused. The first is, 'what is an outlier?', and more specifically to give a "rigorous definition" of such. This is simple:
An outlier is a data point that comes from a different population / distribution / data generating process than the one you intended to study / the rest of your data.
The second question is 'how do I know / detect that a data point is an outlier?' Unfortunately, this is very difficult. However, the answers given here (which really are very good, and which I can't improve upon) will be quite helpful with that task.
Definition 1: As already mentioned, an outlier in a group of data reflecting the same process (say process A) is an observation (or a set of observations) that is unlikely to be a result of process A.
This definition certainly involves an estimation of the likelihood function of the process A (hence a model) and setting what unlikely means (i.e. deciding where to stop...). This definition is at the root of the answer I gave here. It is more related to ideas of hypothesis testing of significance or goodness of fit.
Definition 2 An outlier is an observation $x$ in a group of observations $G$ such that when modeling the group of observation with a given model the accuracy is higher if $x$ is removed and treated separately (with a mixture, in the spirit of what I mention here).
This definition involves a "given model" and a measure of accuracy. I think this definition is more from the practical side and is more at the origin of outliers. At the Origin, outlier detection was a tool for robust statistics.
Obviously these definitions can be made very similar if you understand that calculating likelihood in the first definition involves modeling and calculation of a score :)
define an outlier as a member of that minimal set of elements which must be removed from a datasetof size n in order to assure 100% compliance with RUM tests conducted at 95% confidence level on all (2^n -1) unique subsets of the data. See Karian and Dudewicz text on fitting data to pdfs using R(Sept 2010) for definition of the RUM test.
Outliers are important only in the frequentist realm. If a single datapoint adds bias to your model which is defined by an underlying distribution predeterimined by your theory, then it is an outlier for that model. The subjectivity lies in the fact that if your theory posits a different model, then you can have a different set of points as outliers.