Can the null hypothesis of a t-test of "no significant difference" be used as a result?

Question

I am not so familiar with statistics, since I am a computer science student. I hope it is alright to ask this question here.

I am currently working towards the end of a thesis. I compared the User Experience of Android Apps and Progressive Web Apps (PWA- basically Websites which act like normal apps) and the research question was if PWA can keep up with android apps in terms of UX.

For measuring UX I conducted a user study and used an existing questionnaire (UEQ), which calculates different scales of UX.

To find out if a significant difference exists between those apps I performed t-tests for each scale (calculated value).

Every calculated p-value was way above the alpha-level of 0.05, so I reject the null hypothesis for every scale. At the beginning I thought everything was fine -> no statistical difference found -> PWA can keep up.

But the more I read about this topic I found out a high p-value does not indicate that the groups are equal or that there is no effect. Other sources say "no statistical significance" just means that the Null hypothesis cannot be rejected and "anything is possible".

Question: Are the results of the questionnaire of any use to answer my research question or do I have to say "anything is possible"?

In addition I conducted a qualitative study, which concluded that both types of apps were able to offer the same UX, but now I am doubting if I can put these result in relation to the results of the quantitative questionnaire.

Originally I thought you could use the null hypothesis as a result but I guess I was wrong. A number of research papers, which had a similar topic, just said "no statistical significance has been found, therefore we conclude that both types of Apps were able to ........", which seems to be logically but not mathematically correct.

Answer 1

If you want to provide evidence for equivalence (by contrast with evidence for difference), you can perform a t-test for equivalence using TOST. (In the below "$\theta$" is the difference between groups you are estimating a la $\bar{\mu}_{1} - \bar{\mu}_{2}$ or whatever.)

1. General 'negativist' null hypothesis: $H_{0}^{-}: |\theta| \ge \delta$, with $H_{\text{A}}^{-}:|\theta|< \delta$. This general null hypothesis requires two one-sided null hypotheses to actually test:
   
   $H_{01}^{-}:\theta \ge \delta$ with $H_{\text{A}1}^{-}:\theta < \delta$, and
   $H_{02}^{-}:\theta \le -\delta$ with $H_{\text{A}2}^{-}:\theta > -\delta$.

   If you reject $H_{01}^{-}$, then you conclude $\theta$ is less that $\delta$. If you reject $H_{02}^{-}$, then you conclude $\theta$ is greater than $-\delta$. If your reject both these then you conclude $-\delta < \theta < \delta$.

   What is $\delta?$ It is a priori (i.e. before you perform the test) the smallest difference in your measures (i.e. $\theta$) that you actually care about.

The above graph gives an imaginary distribution of $\theta$ to illustrate the rejection region (the green shaded region, which is both less that $\delta$, and greater than $-\delta$) for two one-sided tests for equivalence for a $\delta=.536$.

2. The t-test statistics are $t_{1} = \frac{\delta - \theta}{\sigma_{\theta}}$ and $t_{2} = \frac{\theta + \delta}{\sigma_{\theta}}$. Both these statistics have been constructed to obtain p-values in the right-tail of the distribution, therefore…

3. Obtain p-values as $p_{1} = P(T_{df} \ge t_{1})$, and $p_{2} = P(T_{df} \ge t_{2})$. ($p_{1}$ and $t_{1}$ are for $H_{01}^{-}$ and so forth.)

4. Decision to reject for $H_{0}^{-}$ is based on $p_{1} \le \alpha$ and $p_{2} \le \alpha$ (not $\alpha/2$).

5. Conclusions are: Reject $H_{0}^{-}$ (i.e. by rejecting both $H_{01}^{-}$ and $H_{02}^{-}$), then conclude "found evidence of equivalence within $\delta$ at the $\alpha$ level;" otherwise Not reject $H_{0}^{-}$, then conclude "failed to find evidence of equivalence within $\delta$ at the $\alpha$ level."

Bonus round: Relevance Tests

Combine inference from a test for equivalence (t-test of $H_{0}^{-}$) with a test for difference (t-test of $H_{0}^{+}$) to (1) incorporate statistical power directly into the conclusion, and (2) incorporate minimum relevant effect size directly into the conclusion:

• Reject $H_{0}^{+}$ and fail to reject $H_{0}^{-}$: conclude "found evidence of a relevant difference at least as big as $\delta$."
  

• Fail to reject $H_{0}^{+}$ and reject $H_{0}^{-}$: conclude "found evidence of equivalence within $\delta$."
  

• Reject $H_{0}^{+}$ and reject $H_{0}^{-}$: conclude "found evidence of a trivial difference (i.e. yes evidence of difference, but a priori you don't care about differences that small, and the test was 'overpowered')."
  

• Fail to reject $H_{0}^{+}$ and fail to reject $H_{0}^{-}$: conclude "Data were indeterminate (i.e. the power of the data given $\alpha$ and $\delta$ is too low to say anything)."

References

Hauck, W. W., & Anderson, S. (1984). A new statistical procedure for testing equivalence in two-group comparative bioavailability trials. Journal of Pharmacokinetics and Pharmacodynamics, 12(1), 83–91.

Schuirmann, D. A. (1987). A Comparison of the Two One-Sided Tests Procedure and the Power Approach for Assessing the Equivalence of Average Bioavailability. Journal of Pharmacokinetics and Biopharmaceutics, 15(6), 657–680.

Answer 2

Your research question is whether user experience differs between android apps and progressive web apps. You designed an experiment and collected data. Your results turned out not significant at the a priori chosen $\alpha$ level of $0.05$. Assuming the data collection and all statistical assumptions were met, the p-value gives you the probability of observing the data you observed (or more extreme; extreme as in even farther away from the null than suggested by your sample) given that there is no difference in the mean user experience betweeen android apps and progressive web apps at the population level.

That being said the p-value can generally be interpreted as the evidence against the null hypothesis. If the evidence against the null is small (or the p-value is large), suggests that progressive web apps don't seem to impact customer experience (at the population level) differently than the android apps. And that would be your result. You cannot make statements about the null hypothesis being true or that the data supports the null hypothesis.

A few good wording examples on how to interpret non-significant results can be found here: Interpretation of non-significant results as "trends"

Another good read I suggest is Moving to a World Beyond “p < 0.05”

As for your thesis, I would recommend you still graphically display your results and not simply call them off because they are not significant. It would also be good to provide the actual p-values and not simply $P<0.05$ (in case you haven't done this already).