8
$\begingroup$

Is Shapiro Wilk test insensitive on the tails of sample distribution? I've read such statement on a paper but I cannot figure out why from its test statistic W. Could someone help me understand this?

$\endgroup$
2
  • 1
    $\begingroup$ Which paper did you read this in? $\endgroup$
    – Glen_b
    Dec 31, 2014 at 5:15
  • 2
    $\begingroup$ See my answer here to a similar question. It also has the references to papers with empirical research, comparisons to different alternatives including long-tailed distributions. $\endgroup$
    – Aksakal
    Dec 31, 2014 at 13:52

2 Answers 2

5
$\begingroup$

The situation is complicated, but the results tend to the opposite of this claim: for moderate dataset sizes $n$, the Shapiro-Wilk test is more sensitive in the tails than elsewhere.


Quantifying sensitivity

I take "sensitive" to mean the extent to which the results vary when values in the dataset are perturbed. (Another possible interpretation is that "sensitivity" is meant in terms of the power of the test to detect deviations from the tail behavior of a Normal distribution. However, since "sensitivity" and "power" are common, well-understood statistical terms with distinct meanings, this second interpretation does not seem appropriate.)

Generically, consider the test "results" (which usually would be taken as a p-value) to be some function $f$ of the ordered data $x$. Then we might want to define the sensitivity of $f$ to the $i^\text{th}$ element of $x$ to be

$$\frac{d}{dx_i} f(x_1,x_2,\ldots, x_n).$$

There are a few problems with this, however. First, $f$ might not be differentiable. Second, sensitivity to extremely small changes might be less relevant than sensitivity to larger changes. To cope with these complications we may (1) use directed finite differences to explore changes in $f$ when $x_i$ is separately increased and decreased and (2) obtain these differences for deviations which are appreciable compared to the spread of the data. To this end, given a deviation $\delta\ge 0$ let

$$s_\delta^{\pm i} f = \frac{f(x_1,\ldots,x_{i-1},x_i\pm\delta\sigma, x_{i+1},\ldots, x_n) - f(x_1,x_2,\ldots, x_n)}{\delta\sigma}$$

(where $\sigma$ is a standard measure of the spread of $x$, such as its standard deviation) and define the sensitivity of $f$ to be the vector of absolute difference quotients

$$(|s_{\delta/2}^i| + |s_{\delta/2}^{-i}|, i=1, 2, \ldots, n).$$

That is, each data value is displaced upwards and downwards by amounts $\delta/2$ times the overall spread. The sensitivity is the total absolute relative change, reflecting a net deviation of $\delta\sigma$ centered at the data.

Assessing sensitivity of distributional tests

Sensitivity can vary with the dataset. Should we be assessing it when the data conform to the null hypothesis or when they are far from the null? Both assessments can be informative. But for distributional tests we face the complication that the alternative often is not even parameterizable: although the null hypothesis might be that the data are sampled from a Normal distribution, the alternative would be that they are sampled from any distribution.

A thorough study would look at many alternatives and many sample sizes. Below I report on results for three sample sizes, $n=4, 12, 36$, which are typical of datasets where the Shapiro-Wilk test is used, and for the null (a Normal distribution), a short-tailed alternative (a Uniform distribution), a long-tailed alternative (an Exponential distribution), and a bimodal alternative (a Beta$(2,2)$ distribution). In each case I make the dataset look as much like its parent distribution as possible. This is accomplished by computing the quantiles of the distribution at $n$ probability plotting points (spaced according to Filliben's formulas, aka "Weibull plotting points").

For reference I have applied the same analysis to a variant of the Kolmogorov-Smirnov test. For this variant I first recenter the data, because (at least for the alternatives) the K-S test will not be a realistic comparison. With the recentered data, both tests often produce comparable p-values and those p-values range from $1$ down to $0.0003$, covering a useful range of possibilities.

Results

The sensitivities for $\delta=1$ are plotted on logarithmic axes against the data indexes (ranks). The results for the S-W test are shown in red with filled circles; those for the K-S test are in blue with filled triangles. (Sensitivities of zero are plotted at $10^{-12}$.)

Figure

The S-W test tends to be slightly more sensitive to data in the tails (i.e., where the ranks are close to $1$ or to $n$) than in the middle, except for very small datasets. The K-S test, by contrast, tends to be extremely sensitive to a small number of data in one or both tails, at least once the dataset size is sufficiently large. Clearly these tests are telling us different things about the shapes of the datasets.

By and large, the S-W test has substantially larger sensitivities than the K-S test. The reasons for this are complicated, but note especially that two distributional tests cannot be compared based on sensitivity alone: you should also consider the p-values at which these sensitivities are measured.

Code

The R code used to produce these results follows. It is structured to be easily modified to extend the study in any desired direction: different sample sizes, different dataset distributions, and different distributional tests.

filliben <- function(n) {
  a <- 2^(-1/n); c(1-a, (2:(n-1) - 0.3175)/(n + 0.365), a)
}
sensitivity <- function(x, f, delta=1, ...) {
  s <- delta * sd(x) / 2
  e <- function(i) {u <- rep(0, length(x)); u[i] <- s; u}
  f.x <- f(x)
  sapply(1:length(x), function(i) f(x + e(i)) - f.x) / abs(s)
}
sensitivity.abs <- function(x, f, delta, ...) {
  abs(sensitivity(x, f, delta/2, ...)) + abs(sensitivity(x, f, -delta/2, ...))
}

delta <- 1
beta <- function(q) qbeta(q, 1/2, 1/2) # A bimodal distribution
par(mfrow=c(3, 4))
for (n in c(4, 12, 36)) {
  x <- filliben(n)
  for (f.s in c("qnorm", "qunif", "qexp", "beta")) {
    # Perform the tests.
    y <- do.call(f.s, list(x))
    y <- (y - mean(y))
    cat(n, f.s, shapiro.test(y)$p.value, ks.test(y, "pnorm")$p.value, "\n")

    # Compute sensitivities.
    shapiro.s <- sensitivity.abs(y, function(x) shapiro.test(x)$p.value, delta)
    ks.s <- sensitivity.abs(y, function(x) ks.test(x, "pnorm")$p.value, delta)
    shapiro.s <- pmax(1e-12, shapiro.s) # Eliminate zeros for log plotting
    ks.s <- pmax(1e-12, ks.s)           # Eliminate zeros for log plotting

    # Plot results.
    plot(c(1,n), range(c(shapiro.s, ks.s)), type="n", log="y",
         main=f.s, xlab="Rank", ylab=paste0("Sensitivity, n=", n))
    points(shapiro.s, pch=16, col="Red")
    points(ks.s, pch=24, bg="Blue")
    lines(shapiro.s, col="#801010")
    lines(ks.s, col="#101080", lty=3)
  }
}
$\endgroup$
3
$\begingroup$

I don't think it's particularly insensitive; I'd say it's more sensitive there than the Lilliefors test, for example, and I'm having trouble thinking of another comparable goodness of fit test in wide use* that I think will be much more sensitive to the tails.

If we look at power comparisons of goodness of fit tests (on which there are numeorus papers), the Shapiro Wilk generally performs very well in a wide variety of situations, including some that I'd regard as relating to "sensitivity to tails".

* well, apart perhaps from the Anderson-Darling$^\dagger$ which might beat it on tail sensitivity in some cases.

Edit: I've been through a number of power comparison studies, including against symmetric heavy tailed alternatives and alternatives with small amounts of contamination by outliers (the two most obvious ways to look at 'sensitivity to tails') and the Shapiro-Wilk does extremely well, generally outperforming even the Anderson-Darling on this task (a task at which the A-D should be expected to excel).

$\dagger$ suitably adapted to the parameter estimation inherent in testing normality without specifying the parameters, of course - see the discussion in the book Goodness of fit techniques by D'Agostino and Stephens

[Do the authors say how this sensitivity was measured or what it was compared to? Does the paper give any justification or context for the claim at all?]

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.