I will answer your questions in the opposite order in which you asked them, so that the exposition proceeds from the specific to the general.
First, let us consider a situation where you can assume that except for a minority of outliers, the bulk of your data can be well described by
a known distribution (in your case the exponential).
If $x$ has pdf:
$$p_X(x)=\sigma^{-1}\mbox{exp}\left(\frac{-(x-\theta)}{\sigma}\right),\;x>0;\sigma>0$$
then $x$ is said to follow an exponential distribution (the special case where we set $\theta=0$ is called the one-parameter or standard exponential distribution).
The usual MLE estimator of the parameters are [0,p 506]:
$$\hat{\theta}=\min_i x_i$$
and
$$\hat{\sigma}=\mbox{ave}_ix_i-\min_i x_i$$
Here is an example in R
:
n<-100
theta<-1
sigma<-2
set.seed(123) #for reproducibility
x<-rexp(n,rate=1/sigma)+theta
mean(x)-min(x)
the MLE of $\sigma$ is $\approx2.08$.
Unfortunately, the MLE estimates are very sensitive to the presence of outliers. For example, if I corrupt the sample by replacing 20% of the $x_i$'s by $-x_i$:
m<-floor(0.2*n)
y<-x
y[1:m]<--y[1:m]
mean(y)-min(y)
the MLE of $\sigma$ based on the corrupted sample is now
$\approx11.12$(!).
As a second example, if I corrupt the sample by replacing 20% of the $x_i$'s by $100x_i$ (say if the decimal place was accidentally misplaced):
m<-floor(0.2*n)
z<-x
z[1:m]<-100*z[1:m]
mean(z)-min(z)
the MLE of $\sigma$ based on this second corrupted sample is now
$\approx54$(!).
An alternative to the raw MLE is to (a) find the outliers using a robust outlier identification rule, (b) set them aside as spurious data and (c) compute the MLE on the non spurious part of the sample.
The most well known of these robust outlier identification rule is the med/mad rule proposed by Hampel[3] who attributed it to Gauss (I illustrated this rule here). In the med/mad rule, the rejection threshold are based on the assumption that the genuine observations in your sample are well approximated by a normal distribution.
Of course, if you have extra information (such as knowing that the distribution of the genuine observations is well approximated by a poisson distribution as in this example)
there is nothing to prevent you from transforming your data and using the baseline outlier rejection rule (the med/mad) but this strikes me as a bit awkward to transform the data to preserve what is after all an ad-hoc rule.
It seems much more logical to me to preserve the data but adapt the rejection
rules. Then, you would still use the 3 step procedure I described in the first link above, but with rejection threshold adapted to the distribution you suspect the good part of the data has. Below, I give the rejection rule in situations where the genuine observations are well fitted by an exponential distribution. In this case, you can construct good rejection thresholds using the following rule:
1) estimate $\theta$ using [1]:
$$\hat{\theta}'=\mbox{med}_ix_i-3.476\mbox{Qn}(x)\ln2$$
The Qn is a robust estimate of scatter that is not geared towards symmetric data. It is widely implemented, for example in the R package robustbase. For exponential distributed data, the Qn is multiplied by consistency factor of $\approx3.476$, see [1] for more details.
2) reject as spurious all observations outside of [2,p 188]
$$[\hat{\theta}',9(1+2/n)\mbox{med}_ix_i+\hat{\theta}']$$
(the factor 9 in the rule above is obtained as the 7.1 in Glen_b's answer above, but using a higher cut-off. The factor (1+2/n) is small sample correction factor that was derived by simulations in [2]. For large enough sample sizes, it is essentially equal to 1).
3) use the MLE on the non spurious data to estimate $\sigma$:
$$\hat{\sigma}'=\mbox{ave}_{i\in H}x_i-\mbox{min}_{i\in H}x_i$$
where $H=\{i:\hat{\theta}'\leq x_i \leq 9(1+2/n)\mbox{med}_ix_i+\hat{\theta}'\}$.
using this rule on the previous examples, you would get:
library(robustbase)
theta<-median(x)-Qn(x,constant=3.476)*log(2)
clean<-which(x>=theta & x<=9*(1+2/n)*median(x)+theta)
mean(x[clean])-min(x[clean])
the robust estimate of $\sigma$ is now
$\approx2.05$ (very close to the MLE value when the data is clean).
On the second example:
theta<-median(y)-Qn(y,constant=3.476)*log(2)
clean<-which(y>=theta & y<=9*(1+2/n)*median(y)+theta)
mean(y[clean])-min(y[clean])
The robust estimate of $\sigma$ is now
$\approx2.2$ (very close to the value we would have gotten without the outliers).
On the third example:
theta<-median(z)-Qn(z,constant=3.476)*log(2)
clean<-which(z>=theta & z<=9*(1+2/n)*median(z)+theta)
mean(z[clean])-min(z[clean])
The robust estimate of $\sigma$ is now
$\approx2.2$ (very close to the value we would have gotten without the outliers).
A side benefit of this approach is that it yields a subset of indexes of
suspect observations which should be set aside from the rest of the data, perhaps to be studied as object of interest in their own right (the members
of $\{i:i\notin H\}$).
Now, for the general case where you do not have a good candidate distribution
to fit the bulk of your observations beyond knowing that a symmetric distribution won't do, you can use the adjusted boxplot[4]. This is a generalization of the boxplot that takes into account a (non parametric and outlier robust) measure of skewness of your data (so that when the bulk of the data is symmetric is collapses down to the usual boxplot). You can also check this answer for an illustration.
- [0] Johnson N. L., Kotz S., Balakrishnan N. (1994). Continuous Univariate Distributions, Volume 1, 2nd Edition.
- [1] Rousseeuw P. J. and Croux C. (1993).
Alternatives to the Median Absolute Deviation.
Journal of the American Statistical Association, Vol. 88, No. 424, pp. 1273--1283.
- [2] J. K. Patel, C. H. Kapadia, and D. B. Owen, Dekker (1976).
Handbook of statistical distributions.
- [3] Hampel (1974). The Influence Curve and Its Role in Robust Estimation. Journal of the American Statistical Association
Vol. 69, No. 346 (Jun., 1974), pp. 383-393.
- [4] Vandervieren, E., Hubert, M. (2004) "An adjusted boxplot for skewed distributions". Computational Statistics & Data Analysis
Volume 52, Issue 12, 15 August 2008, Pages 5186–5201.
1.5*IQR
definition of an outlier is not universally accepted. Try to unload your question and expand on the problem you're trying to solve. $\endgroup$