Why do different statistical tests differ?

Question

I heard from the grapevine that when somebody has statistics, some tests may indicate one type of significance, while others do not. The tests themselves may be inconsistent. My question is, what is the fundamental problem that causes the tests to break down?

My answer, which I am not sure if correct or not, is that it comes down to the problem of not knowing how many terms one needs to get "close" to the convergence. To use an example from analysis, we may know that a sequence of functions converges, but we may not know how many terms are necessary to get "close". In statistics, the sample points are used to estimate the mass function of the random variable. By various limit theorems (strong law of large numbers, central limit theorem, etc), these sample points need to converge to the mass function. However, as we doing inverse-probability theory, we do not know many terms are necessary to converge "close" enough. Sometimes 20,000 sample points seems like a lot, but we do know that there are sequences that converge painfully slow. Perhaps, this is why the tests are inconsistent?

Answer 1

[Beware using the term 'inconsistent' in this context, as inconsistency has a particular technical meaning when applied to hypothesis tests. I'll use it because you did, but with the clear stipulation that it's not taking its technical meaning in this discussion.]

Different test statistics - even when attempting to test quite similar hypotheses - respond to different aspects of the data.

That doesn't necessarily make them "inconsistent" with each other, since they're sensitive to different things and make different assumptions. It's like taking pictures through different filters ... they don't necessarily look the same.

That doesn't mean anything "broke down".

Answer 2

A classical test for a null hypothesis $H_0$ relies on a statistic $T$, the distribution of which under $H_0$ is known. However, the choice of the statistic depends on what you imagine to be likely for $H_1$, that is when $H_0$ is not true, or more generally on the general underlying model you assume for the data -- not only for the null, for all kinds of data you may observe.

Let's take the example of the association between a binary variable $Y = 0, 1$, and a variable with three different levels $X = 0, 1, 2$ -- this is a classical case in genetics when studying association between a disease and di-allelic genetic marker. The null hypothesis is independence of $Y$ and $X$. Using the logistic regression framework, there are different natural models that can be used to test this: $$\def\logit{\mathop{\text{logit}}}\def\P{\mathop{\mathbb P}} \logit \P(Y=1) = \alpha + \beta_1 \mathbf 1_{\{X = 1\}} + \beta_2 \mathbf 1_{\{X=2\}},$$ and $H_0$ is $\beta_1 = \beta_2= 0$ ; $$\logit \P(Y=1) = \alpha + \beta X$$ and $H_0$ is $\beta = 0$.

These two models lead to valid tests (you can compute a $p$-value which is uniformly distributed under $H_0$), but the outcome may be discordant for some observations. For the record, the score test derived under the second model is equivalent to a test on $2\times 3$ contingency table called the Armitage Trend Test; it is the default choice in genetics.

This is just an example: there are many other situations where you can propose different tests for the same null hypothesis. Another classical example would be parametric versus non-parametric tests, which most of the time means assuming normality or not.

Answer 3

First of all I wouldn't really call the tests 'inconsistent'. Anyway I think what you're pointing at is that some tests have low power, i.e. they will only detect extreme scenarios when the sample size is relatively small. This sure is one possible reason for 'inconsistent' results.

Another (closely related) reason is that often different tests assume slightly different underlying distributions/models. An nice example is given when testing whether there's an association in a 2x2 table. One then often uses either the Fisher exact test which assumes a hypergeometric distribution or a $\chi^2$ test which assumes a multinomial model. It's clear that 'extreme' cases in the hypergeometric model might not be considered extreme when you use a multinomial model. And thus do these tests often give slightly different results.