I’m testing the power of the Anderson Darling test to detect deviations from standard normal distributions. I’m calibrating the AD test statistic numerically using python and I noticed an unexpected behaviour (the rejection rate is lower than the significance level for incorrect distributions) when the number of observations is low (30). However, for higher number of observations (100), the test behave as expected (the rejection rate is higher than the significance level for incorrect distributions). Below you can find the process I'm following.

Any help is appreciated. Thanks in advance.

Process to calibrate the critical value:

  • Generate 100 (= n_obs) standard normal random variables {z}
  • Calculate the AD statistics at 1% significance level for the variables generated above.
  • Repeat the steps above 10^5 (=n_paths) times, sort the obtained list of ADs (ad_list) in ascending order and get the critical value at 99% (=conf_level) confidence level, i.e. ad_list[n_paths * conf_level]. The output of the calibration is ad_cval.

Process to calculate rejection rates:
Repeat the same process as above but instead of generating 100 standard normal random variables, generate 100 normal random variables with different standard deviation: sigma = 0.8 and 1.2. Obtain the number of rejections by calculating how many times: AD > ad_cval. Do the same thing using a Kolmogorov-Smirnov test for comparison.

The rejection rates when n_obs = 100 are as expected, meaning that when the distribution is not standard normal, the rejection rate is above the significance level = 1-conf_level = 1%. For example, for sigma = 0.8, 1 and 1.2, I see the following rejections rates: [ad = 3.092%, ks = 2.90%], [ad = 1.10%, ks = 1.06%] and [ad = 11.86%, ks = 4.18%], respectively. Note that when sigma = 1, the rejection rate is close to 1%.

Problems arise when the number of observations is low. For example, when n_obs = 30 and for sigma = 0.8, 1 and 1.2, I see the following rejections rates: [ad = 0.38%, ks = 0.93%], [ad = 0.97%, ks = 1.0%], [ad = 4.72%, ks = 2.20%], respectively.

A rejection rate < 1% for the AD test when sigma = 0.8 does not make sense, how is this possible? Also the KS test seems to struggle to detect deviations from standard normality when sigma = 0.8. Instead, for sigma = 1.2, the rejection rates are > 1% for both the AD and KS tests.

To calculate the AD statistic, I borrowed the code already implemented in python, I only slightly changed it to accommodate the fact that I want to test deviations from standard normal distributions, while the “anderson” function in python test normality (not just standard normality). Below is the code.

from scipy.stats import norm, kstest
import numpy as np

def ADtestCase0(x, dist='norm'):
    implemented for normal distribution only when mean and variance are both known (case 0)
    :param dist:
    if dist != 'norm':
        raise ValueError("Invalid distribution; dist must be 'norm'")

    N = len(x)

    w = np.sort(x)
    logcdf = norm.logcdf(w)
    logsf = norm.logsf(w)

    i = np.arange(1, N + 1)
    A2 = -N - np.sum((2*i - 1.0) / N * (logcdf + logsf[::-1]), axis=0)
    return A2

n_obs = 30
n_paths = 10**5
conf_level = 0.99

sigma_m = 0.8

ad_list = []
ad_list_m = []
ks_list = []
ks_list_m = []
for i in range(n_paths):
    r_norm = norm.rvs(loc=0, scale=1, size=n_obs)
    r_norm_m = norm.rvs(loc=0, scale=sigma_m, size=n_obs)

    ad = ADtestCase0(r_norm)

    ks = kstest(r_norm,'norm')[0]

    ad_m = ADtestCase0(r_norm_m)

    ks_m = kstest(r_norm_m, 'norm')[0]

ad_cval_99 = ad_list[int(n_paths * conf_level - 1)]
ks_cval_99 = ks_list[int(n_paths * conf_level - 1)]

ad_rej_99 = [i for i in ad_list_m if i > ad_cval_99]
ks_rej_99 = [i for i in ks_list_m if i > ks_cval_99]

normaliser = 100/n_paths
print('Rejection rates at 99% confidence level:')
ad_rej_rate = len(ad_rej_99)*normaliser
ks_rej_rate = len(ks_rej_99)*normaliser
print('AD rejection rate:', ad_rej_rate)
print('KS rejection rate:', ks_rej_rate)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.