This is a somewhat basic question, I guess. I have a sequence of random variables $X_1,\ldots,X_n$ that I believe to be i.i.d. uniform on $[0,1]$. Being i.i.d. uniform corresponds to my model predicting the correct distribution for $P(Y_k\mid Y_{1:k-1})$ for some time series $Y$. I can test that by, say, applying the Kolmogorov-Smirnov or the Anderson-Darling test.

However, there is a type of dependence that I want to check: all together they might have a uniform ecdf, but, $X_1,\ldots,X_{n/2}$ and $X_{n/2+1},\ldots,X_n$ separately might not. But if I apply the KS test to each half, the result will not be independent from the statistic generated by the KS test for the whole sequence.

Furthermore, if I apply the KS test to each window $X_k,\ldots,X_{k+m}$, and get a function $k\mapsto \text{p-value}$, $k=1,\ldots,n$, I can check visually whether there are intervals where KS says they are not uniform. But there are $n-m$ tests for each $k$, so they are very much not independent of each other, so this isn't very sound.

What goodness-of-fit test should I use here? How do I justify this procedure properly? Should I be doing this differently? Which textbooks/articles do I need to read?

  • 1
    $\begingroup$ This is a nice idea and it can readily be carried out by simulating the null distribution. But windows of such sequences can look remarkably different from the sequences themselves. Moreover, the KS test (as well as most distribution tests) is remarkably powerful. Thus you're almost guaranteed to find extremely low p-values for some windows somewhere in the sequence. I suspect you would get more out of choosing a small number of appropriate tests in advance instead of conducting such a large number of interdependent tests. $\endgroup$
    – whuber
    Commented Aug 13, 2014 at 18:23
  • $\begingroup$ @whuber Shouldn't the windows' p-values be uniformly distributed, so that I can adjust for the number of windows I have? I am not even sure how to tell if a given test would work for me, or what I should check about it. What if I look at something like the length of the longest run of windows with p-values $<0.5$ (assuming the window length is fixed at something reasonable). $\endgroup$
    – Kirill
    Commented Aug 13, 2014 at 21:59


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.