0
$\begingroup$

If there are 2 results / elements of research who carry a p-value (based on a paired t-test), which tests some null-hypotheses, and one part of the research (some measurement which is 0.593 numerically) yields significance level of 1% whereas another part of the research (some measurement which is 0.571 numerically) yields a significance level of 5%, then how do I exactly interpret this please?

In other words: how to interpret a significance level of 1% versus one of 5%?

Thank you.

$\endgroup$
  • $\begingroup$ You wrote "p-value/significance level" as if they are the same thing. Please clarify and possibly edit your question: Were the p-values for the two tests 0.01 and 0.05 and are you therefore asking how to interpret a p-value of 0.01 vs. a p-value of 0.05? Or were the significance levels of the tests 0.01 and 0.05 and are you asking how to interpret a significance level of 0.01 vs. a significance level of 0.05? $\endgroup$ – elmo Feb 6 at 10:11
  • $\begingroup$ @ELM I clarified this now in the question. The p-values were 0,002 for the first data-point, and 0,019 for the second data-point. My question was about significance levels, but I can equally use an explanation of the p-values. :) $\endgroup$ – Vincent Mia Edie Verheyen Feb 6 at 18:05
1
$\begingroup$

The short answer is that you have very strong evidence from the first dataset that the two populations have indeed different mean responses and strong evidence from the second.

The way to combine results from multiple studies is by conducting a meta analysis. However in the case of just two studies it's irrelevant.

Now, different scientific fields use different $\alpha$'s. For instance in Physics, the results are considered significant at 5 or 6 sigma ie $\alpha$ = 0.00033. In other scientific fields, the most common $\alpha$ used is 5% and it's something that should be fixed beforehand. Otherwise you risk biasing your inference. Thus, in my opinion you should either fix it at 5% and say that both studies are significant at a significance level of 5% or fix it 1% and say that one was and they other one wasn't.

Finally, if the two studies are basically the same (measurements, cut-off points etc) with just different subjects, you can collect both raw datasets in one study and run your own paired t-test that would yield a finalised result and p-value. However, without knowing more details I can't recommend you to do that.

$\endgroup$
1
$\begingroup$

The paired-sample t-test is used to measure the probability that we observe a difference in sample-means between two data sets at least as large as the one we observed, given that they are both generated by a Normal distribution of unknown variance and equal mean to each other. That is, that data set 1 is generated by distribution $\mathcal{N}(\mu, \sigma_1^2)$, and set 2 is generated by distribution $\mathcal{N}(\mu, \sigma_2^2)$. Notice that the means are equal for the two. (The "standard" paired t-test assumes equal variance as well.) That is the null hypothesis.

Since it's possible for the sample means to be different if they're generated by distributions with the same mean, we want to calculate the probability that they're at least as different as they are, with the above assumptions. That is the $p$ value that we obtain. The $p$ value is $$P(\text{difference in sample means } \ge \Delta \mid \text{samples both Normally distributed with equal mean})$$ where $\Delta$ is the observed difference between sample means of the two sample sets.

The significance level is an arbitrary standard created by researchers, by which they determine whether a hypothesis is rejected, or failed to be rejected. The significance level is set for the experiment before the $p$ value is calculated. Then, if the measured $p$ value is at or below the significance level, the researchers declare that the null hypothesis is rejected. If the $p$ value is above the pre-set significance level, then they declare that they failed to reject the null hypothesis.

In the case of the paired t-test, if the $p$ value is $\le$ the significance level, then the researchers declare that they have rejected the hypothesis that the samples are both generated by Normal distributions with equal mean. Otherwise, they failed to reject that hypothesis, and they're still basically agnostic as to the question of whether or not the generating distributions are Normal with equal means.

Keep in mind that, while the $p$ value is a genuine statistical quantity (albeit one whose definition is based on very stringent theoretical assumptions), the significance level is not. The significance level is a judgement call made by researchers, used as a way to interpret the resulting $p$ value. For that reason, if you read two experiments with the same data and same $p$ value, but different significance levels, the only difference between the two studies was the researchers' interpretation of the result, not the result itself.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.