Headlines:
A keyword is shorth.
For an R implementation and links to a current project with publications, see Günther Sawitzki's page at http://www.statlab.uni-heidelberg.de/people/gs/
There is a Stata implementation, which can be installed by ssc inst shorth.
More slowly, and without trying to do justice to Sawitzki's work:
The order statistics of a sample of $n$ values of $x$ are defined by
$x_{(1)} \le x_{(2)} \le \cdots \le x_{(n-1)} \le x_{(n)}.$
Let $h = \lfloor n / 2\rfloor$. Then the shortest half of the data from rank $k$ to rank $k + h$ is identified to minimise $x_{(k + h)} - x_{(k)}$ over $k = 1, \cdots, n - h$. This interval we call the length of the shortest half.
The shorth was named by J.W. Tukey and introduced in the Princeton robustness study of estimators of location by Andrews, Bickel, Hampel, Huber, Rogers and Tukey (1972, p.26) as the mean of $x_{(k)}, \cdots, x_{(k + h)}$. It attracted attention for its unusual asymptotic properties (pp.50-52): on those, see also the later accounts of Shorack and Wellner (1986, pp.767-771) and Kim and Pollard (1990). Otherwise it quickly dropped out of sight for about a decade. Incidentally, Hampel (1997) shows that results available to the Princeton study on asymmetric situations, but not fully analysed at the time, put the shorth in better light than was then appreciated.
Interest revived in such ideas when Rousseeuw (1984), building on a suggestion by
Hampel (1975), pointed out that the midpoint of the shortest half $(x_{(k)} + x_{(k + h)}) / 2$ is the least median of squares (LMS) estimator of location for $x$. See Rousseeuw (1984) and Rousseeuw and Leroy (1987) for applications of LMS and related ideas to regression and other problems. Note that this LMS midpoint is also called the shorth in some recent literature (e.g. David and Nagaraja 2003, p.223; Maronna, Martin and Yohai 2006, p.48). Further, the shortest half itself is also sometimes called the shorth, as the title of Grübel (1988) indicates.
The length of the shortest half is a robust measure of scale or spread: see Rousseeuw
and Leroy (1988), Grübel (1988), Rousseeuw and Croux (1993) and Martin and Zamar (1993)
for further analysis and discussion.
The length of the shortest half in a Gaussian (normal) with mean 0 and standard
deviation 1 is 1.349 to 3 d.p. Thus to estimate standard deviation from the observed length, divide by this Gaussian length.
Some broad-brush comments follow on advantages and disadvantages of shortest half
ideas, from the standpoint of practical data analysts as much as mathematical or
theoretical statisticians. Whatever the project, it will always be wise to compare
shorth results with standard summary measures (including other means, notably geometric
and harmonic means) and to relate results to graphs of distributions. Moreover, if your
interest is in the existence or extent of bimodality or multimodality, it will be best
to look directly at suitably smoothed estimates of the density function.
Simplicity The idea of the shortest half is simple and easy to explain to students
and researchers who do not regard themselves as statistical specialists. It leads
directly to two measures of location and one of spread that are fairly intuitive. It is
also relatively amenable to hand calculation with primitive tools (pencil and paper,
calculators, spreadsheets).
Connections The similarities and differences between the length of the shortest
half, the interquartile range and the median absolute deviation from the median (MAD)
(or for that matter the probable error) are immediate. Thus, shortest half ideas are
linked to other statistical ideas that should already be familiar to many data analysts.
Graphic interpretation The shortest half can easily be related to standard displays
of distributions such as cumulative distribution and quantile plots, histograms and
stem-and-leaf plots.
Mode By averaging where the data are densest, the shorth and also the LMS midpoint
introduce a mode flavour to summary of location. When applied to distributions that
are approximately symmetric, the shorth will be close to the mean and median, but more
resistant than the mean to outliers in either tail and more efficient than the median for
distributions near Gaussian (normal) in shape. When applied to distributions that are
unimodal and asymmetric, the shorth and the LMS will typically be nearer the mode than
either the mean or the median. Note that the idea of estimating the mode as the
midpoint of the shortest interval that contains a fixed number of observations goes
back at least to Dalenius (1965). See also Robertson and Cryer (1974), Bickel (2002)
and Bickel and Frühwirth (2006) on other estimators of the mode. The half-sample mode
estimator of Bickel and Frühwirth is especially interesting as a recursive selection of
the shortest half. Stata users can download a Stata implementation by ssc inst hsmode.
Outlier identification A resistant standardisation such as (value - shorth) /
length may help in identifying outliers. For discussions of related ideas, see Carey et
al. (1997) and included references.
Generalise to shortest fraction The idea can be generalised to proportions other
than one-half.
At the same time, note that
Not useful for all distributions When applied to distributions that are
approximately J-shaped, the shorth will approximate the mean of the lower half of the
data and the LMS midpoint will be rather higher. When applied to distributions that are
approximately U-shaped, the shorth and the LMS midpoint will be within whichever half
of the distribution happens to have higher average density. Neither behaviour seems
especially interesting or useful, but equally there is little call for single mode-like
summaries for J-shaped or U-shaped distributions; for J shapes, the mode is, or should
be, the minimum and for U shapes, bimodality makes the idea of a single mode moot, if
not invalid.
Ties The shortest half may not be uniquely defined. Even with measured data,
rounding of reported values may frequently give rise to ties. What to do with two or
more shortest halves has been little discussed in the literature. Note that tied halves
may either overlap or be disjoint. Different implementations may tackle this in slightly different ways.
Rationale for window length Why half is taken to mean $1 + \lfloor n / 2\rfloor$ also does not appear to be discussed. Evidently we need a rule that yields a window length for both odd and even $n$; it is preferable that the rule be simple; and there is usually some slight arbitrariness in choosing a rule of this kind. It is also important that any rule behave reasonably for small $n$: even if a program is not deliberately invoked for very small sample sizes the procedure used should make sense for all possible
sizes. Note that, with this rule, given $n = 1$ the shorth is just the single sample
value, and given $n = 2$ the shorth is the average of the two sample values. A further
detail about this rule is that it always defines a slight majority, thus enforcing
democratic decisions about the data. However, there seems no strong reason not to use $\lceil n / 2\rceil$ as an even simpler rule, except that all authors on the shorth appear to have followed $1 + \lfloor n / 2\rfloor$.
Use with weighted data Identification of the shortest half would seem to extend
only rather messily to situations in which observations are associated with unequal
weights.
Length when most values identical When at least half of the values in a sample are
equal to some constant, the length of the shortest half is 0. So, for example, if most
values are 0 and some are larger, the length of the shortest half is not particularly
useful as a measure of scale or spread.
Andrews, D.F., P.J. Bickel, F.R. Hampel, P.J. Huber, W.H. Rogers and J.W. Tukey. 1972. Robust estimates of location: survey and advances. Princeton, NJ: Princeton University Press.
Bickel, D.R. 2002. Robust estimators of the mode and skewness of continuous data. Computational Statistics & Data Analysis 39: 153-163.
Bickel, D.R. and R. Frühwirth. 2006. On a fast, robust estimator of the mode: comparisons to other estimators with applications. Computational Statistics & Data Analysis 50: 3500-3530.
Carey, V.J., E.E. Walters, C.G. Wager and B.A. Rosner. 1997. Resistant and test-based outlier rejection: effects on Gaussian one- and two-sample inference. Technometrics 39: 320-330.
Christmann, A., U. Gather and G. Scholz. 1994. Some properties of the length of the shortest half. Statistica Neerlandica 48: 209-213.
Dalenius, T. 1965. The mode - A neglected statistical parameter. Journal, Royal
Statistical Society A 128: 110-117.
Grübel, R. 1988. The length of the shorth. Annals of Statistics 16: 619-628.
Hampel, F.R. 1975. Beyond location parameters: robust concepts and methods. Bulletin, International Statistical Institute 46: 375-382.
Hampel, F.R. 1997. Some additional notes on the "Princeton robustness year". In Brillinger, D.R., L.T. Fernholz and S. Morgenthaler (eds) The practice of data analysis: essays in honor of John W. Tukey. Princeton, NJ: Princeton University Press, 133-153.
Kim, J. and D. Pollard. 1990. Cube root asymptotics. Annals of Statistics 18: 191-219.
Maronna, R.A., R.D. Martin and V.J. Yohai. 2006. Robust statistics: theory and methods. Chichester: John Wiley.
Martin, R.D. and R.H. Zamar. 1993. Bias robust estimation of scale. Annals of Statistics 21: 991-1017.
Robertson, T. and J.D. Cryer. 1974. An iterative procedure for estimating the mode.
Journal, American Statistical Association 69: 1012-1016.
Rousseeuw, P.J. 1984. Least median of squares regression. Journal, American Statistical Association 79: 871-880.
Rousseeuw, P.J. and C. Croux. 1993. Alternatives to the median absolute deviation. Journal, American Statistical Association 88: 1273-1283.
Rousseeuw, P.J. and A.M. Leroy. 1987. Robust regression and outlier detection. New York: John Wiley.
Rousseeuw, P.J. and A.M. Leroy. 1988. A robust scale estimator based on the shortest half. Statistica Neerlandica 42: 103-116.
Shorack, G.R. and J.A. Wellner. 1986. Empirical processes with applications to statistics. New York: John Wiley.
