Suppose I have some large number n of draws from a strictly positive distribution that I believe to be a member of a particular parametric distributional family. I use the draws to estimate the parameters and fit the distribution in question. I then want to know if the if the data is consistent with my hypothesized distributional family against a generalized alternative.
I imagine that there are many tests that one might use to address this question, though I only know of one, the Kolmogorov–Smirnov test and a variant from Kuiper, both of which are basically scaled differences between the infimum and the supremum of the empirical CDF and the estimated CDF.
(I asked a related question six years ago, and got answers including the Shapiro–Francia test and the Lilliefors test. Both of these are tests that I have usually seen used as tests for normality, but both seem to me to have straightforward adaptations to tests against a more generalized parametric distribution. I believe the Shapiro–Francia test is likely to be uniformly more sensitive than the Lilliefors, for reasons discussed below. I don't know how it would compare with the test I suggest below.)
I think the Kolmogorov–Smirnov test is inadequately sensitive. For example, if the kurtosis is wrong, you might get long stretches where the empirical and estimated CDF differ, without ever getting a very large difference between the two CDFs. In particular, if an improbably low stretch is followed by an improbably high stretch, or vice versa, the latter may offset the former, so that the maximum and minimum remain within bounds to accept the hypothesized distributional family, even though the these improbable stretches should cumulate to a higher degree of improbability, rather than offsetting. In general, the CDF smooths out such irregularities in the pdf, or can, at least.
One non-parametric approach that has occurred to me is to order the sample, subtract the fitted values from the empirical pdf, and then count the longest string of observations of the same sign. (Note that it is possible that I am reinventing a well-known test). Under the null that the distributional family is correct, then this would be a sequence of independent Bernoulli trials with probability of success 0.5. According to this posting in the stackexchange math forum, the problem of estimating the probability that the longest run of successive values exceeds some number m was solved by de Moivere in the 18th century and the formula is given there. (And my sample is big enough that limiting distributions are probably relevant. See, e.g. The Longest Run of Heads: A Review by Amarioarei Alexandru). We can then pick m to correspond the level of improbability we wish to use to reject the null. This test also helps us learn where the empirical function differs from the estimated function.
However it seems to me that this test is still far less sensitive than it could and should be. There may be multiple runs of positive or negative values that are individually only mildly improbable but collectively astronomically improbable. For that matter, there could be too many short runs, if there is some high-frequency noise for which we have not accounted. (One might regard this as a reduction of a general misspecification test to a test of whether the distribution of run lengths in our sample is similar to that in a sequence of uncorrelated p = 0.5 Bernoulli trials).
So my questions are:
- I this a sensible test of the stated hypothesis against a general alternative?
- How might one use more of the information in the distribution of strings of runs to strengthen the test?
- Is this a test that is already well-known, and if so, under what name?
- If there is another non-parametric test of the same hypothesis which is known to be generally more powerful (possibly including the Shapiro–Francia test mentioned above), I would rather know what it is (and would accept it as an answer) than know the answers to questions 1 one through 3.