3
$\begingroup$

I'd like to determine the "representative" range for a series of data points. For example, I'd like to know the range where 98% (or any other given ratio) of points are located. I could have used the inter-quantile range (1%-99% in this case), but in case of seriously skewed distributions this gives "too much attention" to long tails of the distribution.

I'd better consider the following idea: imagine a histogram built for that series of points. Maximum of the histogram is considered to be 100%. Now draw the horisontal line at 1% mark and find the leftmost and rightmost intersections of the histogram curve with this line. Those intersections would give the left and right boundaries of the scale that contain most of the points.

Does this approach have a commonly defined name? Is this approach appropriate for determining the "proper" range of the data?

$\endgroup$
2
  • $\begingroup$ Is this a question about describing the data or about making inferences about their parent distribution? $\endgroup$
    – whuber
    Aug 23, 2012 at 13:41
  • $\begingroup$ It's about presenting the data. Although I'd also like to guess what's the distribution behind those series of sample data. $\endgroup$
    – mbaitoff
    Aug 24, 2012 at 3:17

2 Answers 2

2
$\begingroup$

Your intuition is good, yet it is clearer to work with a smooth probability distribution $p(x)$ rather than an histogram or an empirical distribution. I will assume for simplicity that the distribution is unimodal. The problem of finding intervals for a one-dimensional distribution may also be generalised to that of finding domains for the probability density $p(\mathbf{x})$ of a random vector $\mathbf{X}$ in dimension $d$.

Given a probability level $1-\alpha$ (e.g., $1-\alpha = 0.95$), many domains exist with this probability. Among these, is is natural to choose one having minimal volume (minimal interval length for $d=1$). It can be shown that such a domain is the inside of a probability contour $p(\mathbf{x}) = \mathrm{Cst}$.

When $d=1$, the domain is an interval and the probability density takes the same value $\mathrm{Cst}$ on both end-points say $x_{\mathrm{left}}$ and $x_{\mathrm{right}}$. This is your horizontal line. If the distribution is strongly skewed, the two probabilities $\mathrm{Pr}[X < x_{\mathrm{left}} ]$ and $\mathrm{Pr}[X > x_{\mathrm{right}} ]$ can be very different, their sum being $\alpha$.

Finding $\mathrm{Cst}$ given $1-\alpha$ can be done using a numerical routine or simply a table.

The problem is a special case of the determination of a probabilty contour,
cf. Box and Tiao Bayesian Inference in Statistical Analysis.

$\endgroup$
2
  • $\begingroup$ So, if I don't have the exact form of the probability distribution, I can binary-search the histogram table for such leftmost and rightmost bins where the number of population is just barely below the given threshold? $\endgroup$
    – mbaitoff
    Aug 23, 2012 at 9:30
  • $\begingroup$ Yes, at least if the distribution seems unimodal. You can also use a nonparametric estimation such as given by the 'density' R function in the stats package. $\endgroup$
    – Yves
    Aug 23, 2012 at 9:39
2
$\begingroup$

What you are describing has a name: it's called the short estimator --or shortest $\alpha$ estimator for some $\alpha\in (0-1)$.

For a symmetric distribution, it will equal $IQR(\alpha/2):=x_{\lceil1-\alpha/2\rceil}-x_{\lceil\alpha/2\rceil}$ (see this answer) but from your question I gather you want the general --i.e. non symmetric-- case.

Even then, it's pretty easy to compute too, here is the algorithm -- in R to avoid some ambiguities:

#original data.
x<-rexp(100)
#looking for 
#narrowest strip 
#that contains 80% of the data
alpha<-0.8 

#algo
n<-length(x)
h<-floor(n*(1-alpha))
k<-ceiling(n*alpha)
y<-sort(x)
z<-rep(NA,h)
for(i in 1:h){
    z[i]<-y[i+k+1]-y[i]
}
iS<-which.min(z)
ShortStrip<-c(y[iS],y[iS+k+1])
diff(ShortStrip)
#test
sum(x<ShortStrip[2] & x>ShortStrip[1])
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.