Aside from Durbin-Watson, what hypothesis tests can produce inconclusive results?

Question

The Durbin-Watson test statistic can lie in an inconclusive region, where it is not possible either to reject or fail to reject the null hypothesis (in this case, of zero autocorrelation).

What other statistical tests can produce "inconclusive" results?

Is there a general explanation (hand-waving is fine) for why this set of tests are unable to make a binary "reject"/"fail to reject" decision?

It would be a bonus if someone could mention the decision-theoretic implications as part of their answer to the latter query — does the presence of an additional category of (in)conclusion mean that we need to consider the costs of Type I and Type II errors in a more sophisticated way?

Answer 1

The Wikipedia article explains that the distribution of the test statistic under the null hypothesis depends on the design matrix—the particular configuration of predictor values used in the regression. Durbin & Watson calculated lower bounds for the test statistic under which the test for positive autocorrelation must reject, at given significance levels, for any design matrix, & upper bounds over which the test must fail to reject for any design matrix. The "inconclusive region" is merely the region where you'd have to calculate exact critical values, taking your design matrix into account, to get a definite answer.

An analogous situation would be having to perform a one-sample one-tailed t-test when you know just the t-statistic, & not the sample size^†: 1.645 & 6.31 (corresponding to infinite degrees of freedom & only one) would be the bounds for a test of size 0.05.

As far as decision theory goes—you've a new source of uncertainty to take into account besides sampling variation, but I don't see why it shouldn't be applied in the same fashion as with composite null hypotheses. You're in the same situation as someone with an unknown nuisance parameter, regardless of how you got there; so if you need to make a reject/retain decision while controlling Type I error over all possibilities, reject conservatively (i.e. when the Durbin–Watson statistic's under the lower bound, or the t-statistic's over 6.31).

† Or perhaps you've lost your tables; but can remember some critical values for a standard Gaussian, & the formula for the Cauchy quantile function.

Answer 2

Another example of a test with possibly inconclusive results is a binomial test for a proportion when only the proportion, not the sample size, is available. This is not completely unrealistic — we often see or hear poorly reported claims of the form "73% of people agree that ..." and so on, where the denominator is not available.

Suppose for example we only know the sample proportion rounded correct to the nearest whole percent, and we wish to test $H_0: \pi = 0.5$ against $H_1: \pi \neq 0.5$ at the $\alpha = 0.05$ level.

If our observed proportion was $p=5\%$ then the sample size for the observed proportion must have been at least 19, since $\frac{1}{19}$ is the fraction with the lowest denominator which would round to $5\%$. We do not know whether the observed number of successes was actually 1 out of 19, 1 out of 20, 1 out of 21, 1 out of 22, 2 out of 37, 2 out of 38, 3 out of 55, 5 out of 100 or 50 out of 1000... but whichever of these it is, the result would be significant at the $\alpha = 0.05$ level.

On the other hand, if we know the sample proportion was $p = 49\%$ then we do not know whether the observed number of successes was 49 out of 100 (which would not be significant at this level) or 4900 out of 10,000 (which just attains significance). So in this case the results are inconclusive.

Note that with rounded percentages, there is no "fail to reject" region: even $p=50\%$ is consistent with samples like 49,500 successes out of 100,000, which would result in rejection, as well as samples like 1 success out of 2 trials, which would result in failure to reject $H_0$.

Unlike the Durbin-Watson test I've never seen tabulated results for which percentages are significant; this situation is more subtle as there are not upper and lower bounds for the critical value. A result of $p=0\%$ would clearly be inconclusive, since zero successes in one trial would be insignificant yet no successes in a million trials would be highly significant. We have already seen that $p=50\%$ is inconclusive but that there are significant results e.g. $p=5\%$ in between. Moreover, the lack of a cut-off is not just because of the anomalous cases of $p=0\%$ and $p=100\%$. Playing around a little, the least significant sample corresponding to $p=16\%$ is 3 successes in a sample of 19, in which case $\Pr(X \leq 3) \approx 0.00221 < 0.025$ so would be significant; for $p=17\%$ we might have 1 success in 6 trials which is insignificant, $\Pr(X \leq 1) \approx 0.109 > 0.025$ so this case is inconclusive (since there are clearly other samples with $p=16\%$ which would be significant); for $p=18\%$ there may be 2 successes in 11 trials (insignificant, $\Pr(X \leq 2) \approx 0.0327 > 0.025$) so this case is also inconclusive; but for $p=19\%$ the least significant possible sample is 3 successes in 19 trials with $\Pr(X \leq 3) \approx 0.0106 < 0.025$ so this is significant again.

In fact $p=24\%$ is the highest rounded percentage below 50% to be unambiguously significant at the 5% level (its highest p-value would be for 4 successes in 17 trials and is just significant), while $p=13\%$ is the lowest non-zero result which is inconclusive (because it could correspond to 1 success in 8 trials). As can be seen from the examples above, what happens in between is more complicated! The graph below has red line at $\alpha=0.05$: points below the line are unambiguously significant but those above it are inconclusive. The pattern of the p-values is such that there are not going to be single lower and upper limits on the observed percentage for the results to be unambiguously significant.

R code

# need rounding function that rounds 5 up
round2 = function(x, n) {
  posneg = sign(x)
  z = abs(x)*10^n
  z = z + 0.5
  z = trunc(z)
  z = z/10^n
  z*posneg
}

# make a results data frame for various trials and successes
results <- data.frame(successes = rep(0:100, 100),
    trials = rep(1:100, each=101))
results <- subset(results, successes <= trials)
results$percentage <- round2(100*results$successes/results$trials, 0)
results$pvalue <- mapply(function(x,y) {
    binom.test(x, y, p=0.5, alternative="two.sided")$p.value}, results$successes, results$trials)

# make a data frame for rounded percentages and identify which are unambiguously sig at alpha=0.05
leastsig <- sapply(0:100, function(n){
    max(subset(results, percentage==n, select=pvalue))})
percentages <- data.frame(percentage=0:100, leastsig)
percentages$significant <- percentages$leastsig
subset(percentages, significant==TRUE)

# some interesting cases
subset(results, percentage==13) # inconclusive at alpha=0.05
subset(results, percentage==24) # unambiguously sig at alpha=0.05

# plot graph of greatest p-values, results below red line are unambiguously significant at alpha=0.05
plot(percentages$percentage, percentages$leastsig, panel.first = abline(v=seq(0,100,by=5), col='grey'),
    pch=19, col="blue", xlab="Rounded percentage", ylab="Least significant two-sided p-value", xaxt="n")
axis(1, at = seq(0, 100, by = 10))
abline(h=0.05, col="red")

(The rounding code is snipped from this StackOverflow question.)