3
$\begingroup$

I am trying to understand in basic terms what the Shapiro-Wilk test is doing, but the math in the original 1965 paper is beyond me. (For starters, why is the covariance matrix n x n? What do the $v_{ij}$'s represent?)

Nonetheless, it seems to me that with a basic understanding of linear regression, I should be able to obtain a clearer understanding of the meaning of the numerator of the $W$ statistic, which involves understanding the meaning of the $a$ coefficients.

I have been trying to find relevant posts that explain how the test works, such as this one: Good Text for Shapiro-Wilk Test; unfortunately most of what I'm finding are posts about interpretation of SW test results for specific samples.

In this answer to the above post, it is explained that the numerator is "based on the square of a linear function of the order statistics."  In what way?

From the original paper, we have:

$W = \frac{R^4\hat{\sigma}^2}{C^2S^2} = \frac{b^2}{S^2} = \frac{(a'y)^2}{S^2} = \left(\sum_{i=1}^na_iy_i \right)^2 \diagup \sum_{i=1}^n(y_i-\overline{y})^2$

It is explained that $b$ is "up to to the normalizing constant $C$, the best linear unbiased estimate of the slope of a linear regression of the ordered observations, $y_i$, on the expected values, $m_i$, of the standard normal order statistics." (p. 593)

I do not understand why the $a$ coefficients are needed to estimate the slope of this linear regression, and in general remain stumped by the meaning of the $W$ numerator in its various forms. Any light you can shed on this issue is appreciated.

$\endgroup$
  • $\begingroup$ What reference are you quoting at p. 593 of "it is explained" and what exactly is the "original paper"? The reason we need to know is that you appear to be using two different notational conventions, because there is no apparent connection between the $a_i$ in the "original paper" formula and the $m_i$ in your quotation. $\endgroup$ – whuber Sep 18 '19 at 17:56
  • $\begingroup$ Apologies for leaving out the link to the paper; I've now added it. $\endgroup$ – Jon Sep 18 '19 at 18:02
4
$\begingroup$

The Shapiro-Wilk test considers how closely a probability plot of the data adheres to an ideal diagonal line.

For instance, here are probability plots of four independent samples of size $n=5$ from a Normal$(10,2^2)$ distribution:

Figure

Recall that a probability plot is a scatterplot of the points $(m_i, x_{(i)})$ where $x_{(1)}\le x_{(2)}\le \cdots \le x_{(n)}$ are the order statistics of the sample and the $m_i$ are "plotting points" arranged so that ideally the points on the plot will fall along a diagonal line when the data are a random sample from some Normal distribution. The diagonal lines are the "best fits" to the points in a sense to be explained below.

In these plots, the plotting points $m_i$ are fixed numbers equal to the expected values of the order statistics in a standard Normal sample. The vertical coordinates $x_{(i)}$ are the random values from the samples. Thus, all deviations are measured vertically from the reference lines. The sizable deviations in these plots demonstrate that any given probability plot of truly Normal data still deviates from any given line.

It would be incorrect to construct these reference lines using ordinary least squares. The reason is that OLS assumes the vertical deviations are independent. These deviations obviously are not independent, because the data have been placed in order from left to right and bottom to top. The best least-squares fit has to account for these dependencies, as well as for the fact that the extreme points tend to deviate more than the central points.

To appreciate these dependencies, I created a thousand such samples, which produces an array of the $x_{(1)}$ values, an array of the $x_{(2)}$ values, and so on. We may study the dependencies among these arrays by means of a scatterplot matrix:

Figure of the scatterplot matrix

This table of scatterplots includes the reference lines $y=x$ to demonstrate that indeed these values are ordered: all the $x_{(j)}$ values are greater than all the $x_{(i)}$ values whenever $j \gt i,$ causing each scatterplot's points to fall on one side of this reference line. This creates visually obvious positive correlations among the order statistics. These correlations decrease as the difference between $i$ and $j$ increases: in the $x_{(1)}$ vs $x_{(5)}$ plots, for instance, the points exhibit almost no correlation.

The variance-covariance matrix $V$ of the order statistics expresses this dependency information in a quantitative way: for each $i$ and $j$, it gives the covariance of the random variables $X_{(i)}$ and $X_{(j)}$ representing the $i^\text{th}$ and $j^\text{th}$ smallest values in a sample of size $n.$ Notice that this matrix is a property of the Normal distribution, not of any particular sample of it.

Fitting suitable lines to the probability plots is carried out using Generalized Least Squares. Like all other least squares methods, this one is linear in the sense that it estimates the parameters of the fitted line as linear combinations of the data. Among other things, Shapiro and Wilk figured out what the coefficients of those linear combinations must be: these are proportional to their numbers $a_i.$ Their formula estimates the line's intercept as the mean of the order statistics,

$$\hat \mu = \bar x = \sum_{i=1}^n \frac{1}{n} x_{(i)},$$

where the coefficients all equal $1/n.$ In matrix notation, with $m=(m_1,\ldots, m_n)$ the array of plotting points, $V$ as the covariance matrix, and $x=(x_{(1)}, \ldots, x_{(n)})$ as the array of order statistics, it estimates the line's slope as

$$\hat \sigma = \left(\frac{1}{m^\prime V m} m^\prime V\right) x.$$

This slope estimates the standard deviation of the distribution when it is Normal. The conventional estimate of the SD, no matter what the distribution might be, is the usual sample standard deviation. The Shapiro-Wilk statistic is the (square of) the ratio of these estimators. The authors contend that "fairly extensive empirical sampling results" indicate this ratio is useful for detecting departures from Normality in the underlying distribution.

$\endgroup$
  • $\begingroup$ Very informative, thank you! $\endgroup$ – Jon Sep 19 '19 at 11:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.