Additional steps when fail to reject null hypothesis

My understanding of the hypothesis test below is:

$$H_0: \theta_0$$ = 0 ; $$H_A: \theta_0 \neq$$ 0

given $$\alpha$$ = .05 if my p-value is < .05 I reject the null in favor of the alternative then:

• Look at 95% CI to get an idea of the precision
• Determine if the effect size is meaningful

if p-value >.05 fail to reject $$H_0$$

My question is now what does one do specifically

Do you still look at the 95% CI?

I would assume a narrow interval that barely includes zero might suggest maybe the setup lacked power to detect an effect. However what about the opposite case where the interval is exceptionally wide?

What does the wide CI suggest and does it invalidate your failure to reject the null?

• Hi: The CI uses the same information as the hypothesis test so it does not invalidate or validate those results. In fact, one can go from one to the other ( in a 2 sided test ) by checking if the CI crosses zero. If it does, then this is the same as not rejecting the null. Finally, you calculate a CI in either case because it gives you some feel for precision. Apr 1, 2021 at 1:26
• You can look at what your statistical power is. Some statistics have well known weak power, like the ADF, so you have to know how to address these issues. Apr 1, 2021 at 1:35

Suppose you have $$n = 10$$ observations in vector x in R from a normal distribution and wish to test $$H_0: \mu = 0$$ against $$H_a: \mu \ne 0.$$ at the 5% level.

x
[1]  7.04  1.94 -2.42  3.85  3.58 -5.70 -4.86 -3.14  4.50  4.04

summary(x);  sd(x)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
-5.700  -2.960   2.760   0.883   3.993   7.040
[1] 4.491528

stripchart(x)


The sample mean is a little above $$0;$$ the stripchart shows six observations above $$0$$ and four below. But in view of the relatively large spread of the of the data, there seems to be inadequate evidence that $$\mu \ne 0.$$

A t test in R, gives P-value 0.5496, which is not below 5%. That is pretty much the end of the story. Your ten observations don't contain information for you to believe $$\mu$$ differs from $$0.$$

If you're P-hacking, you might try a Wilcoxon signed rank test, a median test, or some kind of permutation test. But the truth is that your ten observations are not enough to detect a population mean $$\mu$$ different from $$0,$$ a t test was an appropriate one, and you should accept its verdict.

t.test(x)

One Sample t-test

data:  x
t = 0.62168, df = 9, p-value = 0.5496
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
-2.330046  4.096046
sample estimates:
mean of x
0.883


The 95% confidence interval $$(-2.33, 4.10)$$ includes $$0.$$ A confidence interval might be considered an interval of potential values of $$\mu$$ that could not be rejected in view of the data at hand. So it will do nothing to change the verdict that $$H_0$$ cannot be rejected.

If you have reason--beyond the data at hand--to believe that $$\mu \ne 0,$$ and sufficient budget to continue investigation, then a reasonable "additional step" might be to perform an experiment with more observations.

If you have a specific amount $$\Delta$$ in mind by which you believe $$\mu$$ differs from $$0,$$ you might use a power and sample size computation to find a sample size $$n$$ that would give a good chance of rejecting $$H_0.$$ if such a difference is really present. Part of that computation requires you to guess at the population standard deviation $$\sigma.$$ The sample standard deviation $$S = 4.49$$ of your current data is an estimate of $$\sigma.$$ However, if $$\mu = 0$$ is really true, then more investigation is not going to be useful.