2
$\begingroup$

So I'm trying to calculate the p value for a KS Test across some score data. I am using a variable called bad that is either 1 or 0 that splits the data into 2 samples effectively. I have run in SAS the 'PROC NPAR1WAY' procedure to calculate the p value and two sample ks statistic for reference. Unfortunately in SAS I am unable to view the math behind the procedure.

What I would like to know is how to calculate that p value manually so I can code it up in SQL or calculate it in Excel.

Any help is much appreciated. I have posted the picture of my SAS output for reference.SAS KS Test Output

$\endgroup$
  • 2
    $\begingroup$ You can see all the math you want behind the procedure by looking at the SAS documentation. In this case: support.sas.com/documentation/cdl/en/statug/63033/HTML/default/… Hint: SAS has multiple options on how to run this procedure, so you'll need more than just a screenshot of the test statistic and final $p$-value for any real hope at understanding how it's computed. You'll need source code too. $\endgroup$ – StatsStudent Jan 25 '19 at 1:06
  • 1
    $\begingroup$ The distribution of the KS test statistic is given by a theta function. The classical reference on these is Whittaker & Watson, A Course of Modern Analysis. The chapter on theta functions is immediately accessible to anyone who is well versed in Complex Analysis. Theta functions are ubiquitous--they appear as solutions of the Heat Equation, the basic equation governing diffusive phenomena--so a little searching ought to turn up good numerical algorithms. $\endgroup$ – whuber Jan 25 '19 at 14:07
2
$\begingroup$

You can find a closed-form matrix method to evaluate the Kolmogorov distribution in Marsaglia et al (2003) and an improved algorithm in Carvalho (2015). This method writes the CDF of the Kolmogorov distribution as an element from the power of a matrix with finite dimensions, so it allows exact computation. To facilitate the analysis, suppose we write the argument value of interest in the CDF as:

$$d = \frac{k-\theta}{n},$$

where $k$ is a positive integer and $0 \leqslant \theta < 1$ is a remainder. Let $m \equiv 2k+1$ and define $\mathbf{H}(\theta)$ to be an $m \times m$ matrix of the form:

$$\mathbf{H}(\theta) \equiv \begin{bmatrix} \tfrac{(1-\theta)}{1!} & 1 & 0 & 0 & 0 & \cdots & 0 & 0 \\ \tfrac{(1-\theta^2)}{2!} & \tfrac{1}{1!} & 1 & 0 & 0 & \cdots & 0 & 0 \\ \tfrac{(1-\theta^3)}{3!} & \tfrac{1}{2!} & \tfrac{1}{1!} & 1 & 0 & \cdots & 0 & 0 \\ \tfrac{(1-\theta^4)}{4!} & \tfrac{1}{3!} & \tfrac{1}{2!} & \tfrac{1}{1!} & 1 & \cdots & 0 & 0 \\ \tfrac{(1-\theta^5)}{5!} & \tfrac{1}{4!} & \tfrac{1}{3!} & \tfrac{1}{2!} & \tfrac{1}{1!} & \cdots & 0 & 0 \\ \vdots & \vdots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots \\ \tfrac{(1-\theta^{m-1})}{(m-1)!} & \tfrac{1}{(m-2)!} & \tfrac{1}{(m-3)!} & \tfrac{1}{(m-4)!} & \tfrac{1}{(m-5)!} & \cdots & \tfrac{1}{1!} & 1 \\ \tfrac{(1-\theta^{m})}{m!} & \tfrac{(1-\theta^{m-1})}{(m-1)!} & \tfrac{(1-\theta^{m-2})}{(m-2)!} & \tfrac{(1-\theta^{m-3})}{(m-3)!} & \tfrac{(1-\theta^{m-4})}{(m-4)!} & \cdots & \tfrac{(1-\theta^2)}{2!} & \tfrac{(1-\theta)}{1!} \\ \end{bmatrix}.$$

Then, for a Kolmogorov-Smirnov test with $n$ data points yielding the statistic $D_n$, the cumulative distribution of interest (under the null hypothesis) can be written in matrix form as:

$$F_n(d) \equiv \mathbb{P}(D_n \leqslant d) = \mathbb{P} \bigg( D_n \leqslant \frac{k-\theta}{n} \bigg) = \frac{n!}{n^n} \cdot [\mathbf{H}(\theta)^n]_{k,k}.$$

This formula allows us to compute the CDF of the Kolmogorov distribution by computing an element of a finite power of a finite matrix. It therefore gives an exact closed-form method that operates without any truncation or other approximation. The method is implemented in R in the kolmim package (documentation here).

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

There is a good answer post on the question of KS test in the following stackexchange post:How to determine which distribution fits my data best?, which you can read to understand the KS test and sampling.

But, note however, there are also mistakes in the answer posted, referred above, especially with respect to interpretation of p-value of the KS test, about which your question pertains (I am new to statistics and to this forum and hence could not provide response to that post also). I am giving below the salient points with regard to interpretation of p-value of KS statistic and its statistics and manual calculation procedures:

  1. The D statistic (one sample), calculated through the KS test, measures the maximum difference between empirical and theoretical distributions that you wish to compare. The statistic (D*sqrt(sample size)) is distributed as per Kolomogrov distribution which is well explained in the wikepedia page on ks test.

  2. Note that the KS test is different from the usual normal and t-test on sampling distributions, as the test statistics has a separate distribution as above (for large sample sizes). It is not normally distributed and interpretation of p-value and confidence intervals does not follow the same logic (in other words, it is not based on CLT theorem which states that standard error of sample is normally distributed, but similar to it, since the d-statistic approaches the given kolomogrov distribution as sample size increases).

  3. Hence, the cumulative p-value can be calculated using the kolomogrov distribution.

  4. A p-value of, say 80%, implies that only 20% of samples, taken from the theoretical distribution for given distribution statistics, would contain a difference in D statistic above the obtained value. Note that ideally we want the two distributions to be equal and hence D to be zero. As the distributions vary, the D-value increases. This p-value is not related to the usual confidence intervals (alphas) of normal or t.tests

  5. If you have obtained the D-statistic, you can manually obtain the p-value from the kolomogrov d-statistic tables (like we do for t-statistics). A table is available here:https://projecteuclid.org/download/pdf_1/euclid.aoms/1177730256. You have to use D*Sqrt(samplesize) and refer the p-value as (1-table value) - that is to emphasize the reverse distribution which you can understand intuitively - a difference of zero (D=0) would imply that p-value would be 1, implies 100% of samples would have D value zero or more.

  6. If you want to further understand this calculation, refer to the above post by coolserdsah which however would be misleading without following corrections:

    • After obtaining d-statistic through simulation from theoretical distribution, instead of doing logspline on the values, you can directly refer the results as the cumulative probability distribution of d-statistics, given by (s=d*sqrt(samplesize)) is distributed as per kolomogrov. Suppose if you want to refer to statistic applicable for 95% confidence level, refer to 0.95 quantile of the results of the simulation (the "stats" vector calculated in that post referred above). This is the p-value given for KS test statistics in R package. It cannot be any different in any other package also as it is based on the known stated distribution - refer wikipedia)
  7. For two sample test, the calculation slightly differs as explained in the wikipedia page, but calculation and interpretation of p-value are same.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

enter image description hereenter image description here

Hi All,

So using the background documentation I’ve written some VBA code to run the calculation. Couple of things: -I’ve excluded the ‘1 - ‘ at the start as I’ve flipped the inequality at the start -my ks statistic came from a sas calculation -my n values are the count of observations in each dataset -upper bound was 30370 as this was the square root of the largest value I was able to use in vba and given that we have to square this value anything greater would result in an overflow error

And the result is equal to the SAS calculation of the P Value to 8 decimal places.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.