I have two sets of data, and I want to test which is "more normal" (specifically residuals from two different models fitted to hourly and daily data - the daily data is the hourly data aggregated).

One appears "more normal" when plotted as a Q-Q plot. I've also performed a Anderson–Darling test and in both cases the p-value is < 0.05 but in one case > 0.01, the test statistic itself is lower for the "more normal" data.

My question is, is it valid to say that on the basis of a lower test statistic between two tests that one of the data is "more normal"? Particularly if neither meet some level of significance?

  • 7
    $\begingroup$ Why would it matter? $\endgroup$ – Tim Oct 22 '19 at 4:43
  • 2
    $\begingroup$ Please beware that p-values are heavily by sample size. If your samples are unequal, difference of size may be more important than the effects explained in the answers. $\endgroup$ – Pere Oct 22 '19 at 16:37
  • $\begingroup$ @Tim because I want to say "adding feature x resulting in the residuals being much closer that of a normal distribution" when tested on n independent data sets $\endgroup$ – David Waterworth Oct 23 '19 at 1:04
  • $\begingroup$ @DavidWaterworth why would you care about that? $\endgroup$ – Tim Oct 23 '19 at 4:37

If you want to quantify departure from normality, then a good measure is the Kolmogorov-Smirnov test statistic $D.$ Let's compare two samples of size $n = 5000.$

  • The sample x below it taken using an excellent algorithm in R that is known to sample from an essentially perfect normal population, $\mathsf{Norm}(\mu=1.5, \sigma=0.5).$

  • The sample y is based on sums of three standard uniform random variables. By the Central Limit theorem, we can guess that such a sum might be nearly normal, but the actual slightly non-normal population is known. It also has $E(Y) = 1.5, SD(Y) = 0.5.$


 x = rnorm(5000, 3/2, 1/2)
 mean(x); sd(x)
 [1] 1.492946
 [1] 0.5032069
    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 -0.4434  1.1552  1.4951  1.4929  1.8283  3.4453 

 ks.test(x, "pnorm", 3/2, 1/2)

         One-sample Kolmogorov-Smirnov test

 data:  x
 D = 0.013255, p-value = 0.3434
 alternative hypothesis: two-sided

 y = replicate(5000, sum(runif(3))) 
 mean(y); sd(y)
 [1] 1.503185
 [1] 0.500952
    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
 0.09379 1.15050 1.49884 1.50319 1.86148 2.90054 

A key non-normal feature of the Y-population is that it has no probability outside the interval $(0,3).$

ks.test(y, "pnorm", 3/2, 1/2)

        One-sample Kolmogorov-Smirnov test

data:  y
D = 0.018057, p-value = 0.07674
alternative hypothesis: two-sided

Histograms. Histograms of the two samples are shown below, along with densities of $\mathsf{Norm}(1.5, 0.5).$

enter image description here

ECDF plots. Empirical CDFs of the two samples are shown below, along with CDFs of $\mathsf{Norm}(1.5, 0.5).$

enter image description here

At the scale of these cumulative plots, it is difficult to see a difference between ECDFs and CDFs. However, there are slight discrepancies.

K-S test statistic. The Kolmogorov-Smirnov test statistic measures the maximum vertical absolute difference between ECDF and CDF in each case. For the $X_i$s, that absolute difference is $D \approx 0.013$ and for $Y_i$s, the absolute difference is a little larger $D \approx 0.018.$

A closer look. In order to show maximum absolute differences between ECDF and CDF more clearly, we show an ECDF plot of a sample of size $n = 5$ from the Y-population.

y1 = replicate(5, sum(runif(3)))
ks.test(y1, "pnorm", 1.5, .5)$stat  # '$'-notation shows test stat

plot(ecdf(y1), main="n=5: 'Nearly' Normal Population")
 curve(pnorm(x,1.5,.5), add=T, col="red")

enter image description here

The maximum vertical distance $D = 0.3369$ between the ECDF and CDF occurs at observation $0.7356.$

For two samples of the same size, the one with the smaller K-S normality test statistic $D$ could be said to be more nearly normal. However, there are other ways to measure differences between ECDFs and CDFs.

| cite | improve this answer | |

Let us begin with the assumption that you have data collected across time that is drawn from a normal distribution. If it is, then the frequency is irrelevant even if one level of frequency looks nicer than another. That is due to Donsker's Theorem.

As to

My question is, is it valid to say that on the basis of a lower test statistic between two tests that one of the data is "more normal"?

The answer is no, at least as you have constructed it. Your null hypothesis is that $x$ is drawn from a normal distribution in both cases. It is rejected. You cannot, at least in this manner, make statements about the differences in the samples. You did not perform a difference test such as $\mu_1-\mu_2$. Hypothesis tests are with regard to population parameters and not samples.

You have two choices on how to consider this, subject to the assumptions of the Anderson-Darling test and any instrumentation issues that may have existed in gathering the sample. You can either use the p-values as evidence against the null and reject that it is normal; or you can assume that the sample is an extreme case because the p-value only states that if the null is true, then the sample was unlikely. If the latter may hold, then you should perform another investigation.

By themselves, p-values are not informative as to whether your sample was bad but your hypothesis good and the case where the sample was good but your hypothesis bad.

The better question, regarding your residuals not being normal, is "so what?" Why would they be something else? What might be going on in your model?

| cite | improve this answer | |
  • $\begingroup$ Thanks, my assumption though is that at least the Anderson-Darling statistic is based (I believe) on the mean squared difference between the empirical and theoretical distribution. So given the same number of samples, and assume say we scale the data - then wouldn't a lower test statistic mean the "distance" to normality is lower irrespective of the p-value? $\endgroup$ – David Waterworth Oct 23 '19 at 0:56
  • $\begingroup$ Your assumption is tempting, but I would reiterate my response for three reasons. The first, if the models are the same, then it is the same data. One cannot be closer than the other. The second is that you may be suffering from the fallacy of aggregation. It could be your work is not scale-invariant, in which case there is an intervening function that you have missed. $\endgroup$ – Dave Harris Oct 23 '19 at 23:00
  • $\begingroup$ The third reason is subtle. Imagine you split a school in half, randomly assigning people to group A or B. You measure the height. You find $s_A^2>s_B^2$. What about tossing a coin in the air to assign membership made the measurements in B more accurate? What about aggregation made one look like it resembled the Gaussian more? $\endgroup$ – Dave Harris Oct 23 '19 at 23:02
  • $\begingroup$ Thanks for the information, I'll do some reading. I'm actually aggregating by time, i.e. I'm trying to compare a model fitted to data aggregated to daily level (energy consumption as a function of temperature) to a model fitted to 15 minute data. If I sum the residuals of the 15 minute model to daily, the MSE is lower and the residuals look more normal. My assumption is the day/night cycle makes the high frequency models aggregation more normal. $\endgroup$ – David Waterworth Oct 23 '19 at 23:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.