Determine range that contains given ratio of data points

Question

I'd like to determine the "representative" range for a series of data points. For example, I'd like to know the range where 98% (or any other given ratio) of points are located. I could have used the inter-quantile range (1%-99% in this case), but in case of seriously skewed distributions this gives "too much attention" to long tails of the distribution.

I'd better consider the following idea: imagine a histogram built for that series of points. Maximum of the histogram is considered to be 100%. Now draw the horisontal line at 1% mark and find the leftmost and rightmost intersections of the histogram curve with this line. Those intersections would give the left and right boundaries of the scale that contain most of the points.

Does this approach have a commonly defined name? Is this approach appropriate for determining the "proper" range of the data?

Answer 1

Your intuition is good, yet it is clearer to work with a smooth probability distribution $p(x)$ rather than an histogram or an empirical distribution. I will assume for simplicity that the distribution is unimodal. The problem of finding intervals for a one-dimensional distribution may also be generalised to that of finding domains for the probability density $p(\mathbf{x})$ of a random vector $\mathbf{X}$ in dimension $d$.

Given a probability level $1-\alpha$ (e.g., $1-\alpha = 0.95$), many domains exist with this probability. Among these, is is natural to choose one having minimal volume (minimal interval length for $d=1$). It can be shown that such a domain is the inside of a probability contour $p(\mathbf{x}) = \mathrm{Cst}$.

When $d=1$, the domain is an interval and the probability density takes the same value $\mathrm{Cst}$ on both end-points say $x_{\mathrm{left}}$ and $x_{\mathrm{right}}$. This is your horizontal line. If the distribution is strongly skewed, the two probabilities $\mathrm{Pr}[X < x_{\mathrm{left}} ]$ and $\mathrm{Pr}[X > x_{\mathrm{right}} ]$ can be very different, their sum being $\alpha$.

Finding $\mathrm{Cst}$ given $1-\alpha$ can be done using a numerical routine or simply a table.

The problem is a special case of the determination of a probabilty contour,
cf. Box and Tiao Bayesian Inference in Statistical Analysis.

Answer 2

What you are describing has a name: it's called the short estimator --or shortest $\alpha$ estimator for some $\alpha\in (0-1)$.

For a symmetric distribution, it will equal $IQR(\alpha/2):=x_{\lceil1-\alpha/2\rceil}-x_{\lceil\alpha/2\rceil}$ (see this answer) but from your question I gather you want the general --i.e. non symmetric-- case.

Even then, it's pretty easy to compute too, here is the algorithm -- in R to avoid some ambiguities:

#original data.
x<-rexp(100)
#looking for 
#narrowest strip 
#that contains 80% of the data
alpha<-0.8 

#algo
n<-length(x)
h<-floor(n*(1-alpha))
k<-ceiling(n*alpha)
y<-sort(x)
z<-rep(NA,h)
for(i in 1:h){
    z[i]<-y[i+k+1]-y[i]
}
iS<-which.min(z)
ShortStrip<-c(y[iS],y[iS+k+1])
diff(ShortStrip)
#test
sum(x<ShortStrip[2] & x>ShortStrip[1])