1
$\begingroup$

An additional reason that the chi-squared distribution is widely used is that it turns up as the large sample distribution of generalized likelihood ratio tests (LRT).[6] LRTs have several desirable properties; in particular, simple LRTs commonly provide the highest power to reject the null hypothesis (Neyman–Pearson lemma) and this leads also to optimality properties of generalised LRTs. However, the normal and chi-squared approximations are only valid asymptotically. For this reason, it is preferable to use the t distribution rather than the normal approximation or the chi-squared approximation for a small sample size. Similarly, in analyses of contingency tables, the chi-squared approximation will be poor for a small sample size, and it is preferable to use Fisher's exact test. Ramsey shows that the exact binomial test is always more powerful than the normal approximation.[7] :https://en.wikipedia.org/wiki/Chi-squared_distribution

By the central limit theorem, because the chi-squared distribution is the sum of $k$ independent random variables with finite mean and variance, it converges to a normal distribution for large $k$. For many practical purposes, for $k>50$ the distribution is sufficiently close to a normal distribution for the difference to be ignored. ${ }^{[13]}$ Specifically, if $X \sim \chi^{2}(k)$, then as $k$ tends to infinity, the distribution of $(X-k) / \sqrt{2 k}$ tends to a standard normal distribution. However, convergence is slow as the skewness is $\sqrt{8 / k}$ and the excess kurtosis is $12 / k$

Does there always exist a non-chi-squared test-statistic for the likelihood-ratio (neyman-pearson, karlin-rubin), score, and wald-tests?

In the nested-mixed models/gee, is there a better statistic than the chi-square from asymptotic theory?

Is there a method/strategy for identifying the (non-chi-squared) test-statistic in these tests? I want to avoid asymptotic distributions as I never have infinite data.

Does the uniformly most powerful property of the neyman-pearson and wilks-rubin property depend on avoiding the asymptotic chi-square distribution of the test-statistic in small sample sizes and use the correct distribution for the test-statistic or are these tests even more powerful when one chooses the correct distribution in small-samples?

$\endgroup$
2
  • $\begingroup$ Would you please elaborate on your sentiment that you "want to avoid asymptotic distributions as [you] never have infinite data"? $\endgroup$
    – Ben
    Jul 31, 2022 at 21:04
  • $\begingroup$ S.S. Wilks demonstrated that the likelihood-ratio test statistic under sample-size -> infinity, the distribution of the test-statistic, a random variable, is chi-square. In reality, I never have all the data and I don't think the test-statistic converges in distribution to a chi-square for small sample sizes, so using a chi-square for n=10 in a likelihood ratio-test is misleading inference. $\endgroup$
    – user318514
    Jul 31, 2022 at 21:09

1 Answer 1

2
$\begingroup$

While $n$ is never infinite, as a practical matter it is in many cases large enough to give usable results.

It seems kind of odd to worry overly much about the exactness and power of the small sample test when the distributional model assumption itself will (almost always) be an approximation; perhaps it would make more sense to worry about its robustness to plausible deviations from that model. That is, it's fine to be worried about whether the impact of sample size is large or small when the model is correct, but the larger problem may be that the model is not correct.

Algebraic manipulations

Nevertheless, with likelihood ratio statistics (or indeed the score statistic and so forth) we can sometimes obtain exact small sample distributions; this would be considered on a case-by-case basis but the more complicated the model the harder it typically becomes.

The first issue with the asymptotic distribution in small samples is accuracy of significance level. Naturally if you conduct tests based off the same statistic at different significance levels, their power curves differ. Control of significance levels may be important.

Whether you use $\Lambda$ or $-2\log \Lambda$ is not material (they're equivalent statistics), what matters is where you place your critical value for each; obtaining a critical value from the asymptotic chi-squared result is not going to be exact for finite samples (though it may be fine for some purpose).

We might expect that the Wald and score tests may tend to have less power in small samples (once we get the attained significance levels to be the same), though sometimes there will be little or no difference.

Simulation

You may be able to use simulation from the null model (plus all the relevant assumptions) to identify suitable rejection regions in many cases; this can be made as accurate as you require by simulating more.

Nonparametric tests based on the likelihood ratio (etc) statistic

There's another possibility with finite samples, which is to use such statistics as the basis of a resampling test. In many simple cases you may be able to perform a permutation test using such a statistic (yielding an 'exact' test - irrespective of the distribution - with good power when the model is correct); alternatively there's the possibility of bootstrapping in large samples. Various common techniques can improve the approximation of the significance level for the bootstrap in smaller samples.

$\endgroup$
11
  • $\begingroup$ using asymptotic distributions in small sample sizes is using a bad approx of a bad approx, and the approximation error is not ignorable. $\endgroup$
    – user318514
    Aug 3, 2022 at 14:34
  • $\begingroup$ so your anwser is to use resampling? $\endgroup$
    – user318514
    Aug 3, 2022 at 14:35
  • $\begingroup$ I'm not specifically saying "use it" nor "don't use it", since there may be other considerations that impact the choice. Rather that, since the concern seems to be "avoid exceeding a chosen type I error rate when samples are not large and you don't know the exact population distribution" that where they're viable, permutation tests will allow you to do just that, using almost any statistic you like (including likelihood ratio statistics, Wald statistics, score statistics, etc). ... ctd $\endgroup$
    – Glen_b
    Aug 4, 2022 at 2:13
  • $\begingroup$ ctd ... It does surprise me that when people are very focused on maintaining an upper limit on type I error rates that they don't regularly take advantage of this rather simple mechanism to achieve it. Such tests (and indeed resampling tests more generally) are not a panacaea, but they do seem to me to be vastly underused by people whose express concerns are closely aligned with what they allow you to achieve. $\endgroup$
    – Glen_b
    Aug 4, 2022 at 2:13
  • $\begingroup$ do you know a book/ paper that describes those methods? $\endgroup$
    – user318514
    Aug 4, 2022 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.