# Is there an expression that can be used to evaluate the threshold that characterizes a substantial difference between sample means for t-test?

I'm currently performing a paired t-test for the difference between sample means. I have an interest in determining if there is a way to obtain a scalar value for a threshold that quantifies whether the difference between the sample means is substantially greater than zero given a prescribed significance level. Does there exist such an expression or threshold? It would be nice to be able to report that given the results of the hypothesis test, a difference between sample means of "x" or greater was determined to be an indicator of a difference that was substantially greater than zero given a significance level of "y". This result would be used to support the conclusion from the p-value by adding practical significance to the statistical significance of comparing the p-value to the significance level.

• I'm not quite sure what you mean. Is it a confidence interval? A standardized effect size (like Cohen's d)? A confidence interval of an effect size? If it's statistically significant, the difference is unlikely to be zero - but whether it's substantial is a different question that probably requires domain knowledge. 1 lb difference in body weight might be statistically significant, but is probably not substantial. 10 lb difference in weight might be substantial, but not statistically significant. Commented Jun 4 at 18:06
• Hey Jeremy, thanks for the recommendations. How would I go about computing a confidence interval of an effect size? I'm working with paired data and would like to use the Cohen's d method to arrive at a result that provides practical significance behind the differences between means. I'm assuming that using a pooled standard deviation would be best in this case, although, it is worth noting that the paired data come from a time-series dataset (comparing means between two points in time). Commented Jun 4 at 20:15
• I've read that 0.2, 0.5, and 0.8 are small, moderate, and large effect indicators, but not sure that this is generalizable to cost data that I'm working with. Commented Jun 4 at 20:16
• The size of effect that is practically or scientifically "substantial" is ALWAYS context-dependent and so any attempt to provide a formulaic threshold will be harmful. Statistical analyses can yield statistical inferences, but only a thoughtful and informed analyst can extrapolate from statistical inferences to inferences about the real world. See this discussion of the topic: link.springer.com/chapter/10.1007/164_2019_286 Commented Jun 4 at 21:36