Interpreting p value results in paper

Question

Sorry about the ambiguous title. I have many independent variables, each with their own hypothesis. Some of them have significant p-values and some don't. How am I supposed to write this in my paper?

For example, if my hypotheses were:

• IV1 has an influence on DV.
• IV2 has an influence on DV.
• IV3 has an influence on DV.
• IV4 has an influence on DV.

and while IV1, IV2, and IV3 had a p-value of < 0.01, IV4 had a p-value of > 0.05.

From this, I understand that IV1, IV2, IV3 all have a positive or negative impact on my DV (depending on the coefficient). But what should I write about IV4? That I tried it out, but the result wasn't significant enough to prove anything? Or that it did not show any relationship in my data? Or ?

For my data, the former (IV4's significance wasn't considerable) makes more sense, but I haven't read any research making such connection in a similar case...

Does anyone know what to do?

Answer 1

First of all remember that it all depends on the significance level that you choose in your tests (very often we choose 5% or 1%, here it seems like you are using 1%/10 and 5%/10). Remember also that you have to choose 1 significance level only for all your tests otherwise they will not be comparable. So your question should be reformulated as "given my significance level of x%, what if all the variables except for V4 have p-values lower than x%, while V4 doesn't?". In trivial "non-statistical" words a two-tailed test like yours works like this: you start from a null hypothesis that v1,v2,v3,v4 has a coefficient value equal to 0 (and, as such, a non-significant relationship with the DV) and, based on the chosen significance level, you search for evidence in the data that may allow you to reject the null hypothesis. If you find such evidence, then you can reject the null hypothesis, otherwise you cannot reject the null.

Then on your paper you might write that, while at x% significance level we can reject the null hypothesis that v1,v2,v3 do not have a significant impact on the DV, instead there is no statistical evidence that we can reject the hypothesis that v4 has no significant impact. That, translated into less statistical language, means that, at, that chosen significance level x%, we have statistical evidence that v1,v2,v3 have a significant impact on DV, but we cannot conclude that v4 does the same (i.e. simply we cannot say v4 influences the DV, because we have not enough evidence to reject our base hypothesis that is the null hypothesis that the true value of v4 coefficient is 0 in a two-tailed test).

Answer 2

The p-value is defined as the probability of observing a test statistic (i.e., your t-value, or correlation coefficient, or whatever your test was) as extreme (or more extreme) as the one observed, if the null hypothesis is true.

To put this definition into more concrete terms, let's say your observed p-value for IV1 on your DV was 0.012. You can interpret this p-value as such: If IV1 truly has no effect in the population (i.e., if the null hypothesis is true), then the probability of observing the results you have observed (or more extreme results) is 0.012. Therefore, if the null hypothesis is true, your observed results are pretty surprising. Researchers tend to therefore state such a low p-value provides some evidence against the null.

However (and coming back to your specific question), the consequence of this definition is that if you have a non-significant p-value (greater than 0.05 in your case), then all you can do is fail to reject the null hypothesis of no difference. All you can conclude is that your experiment failed to reject the claim of no difference. You could write something like: "IV4 had no significant effect on the DV; therefore, we failed to reject the null hypothesis that IV4 has no effect on the DV"

Be careful not to be lured into trying to "accept the null hypothesis" when your p-value is non-significant; you cannot do this with a p-value.

Answer 3

Great question! Here are some thoughts on how to write/organize the results. Hope this helps!

First, based on decision theory, choosing and justifying a threshold (before conducting your analyses) is important in order to avoid p-hacking, and explicitly writing/justifying that as part of your hypothesis testing procedure will make your results/analysis more robust. To follow up on thresholds, you also have to make sure if you're doing multiple comparisons and whether adjustments are necessary (they may change what is "significant" in your results).

Second, to answer your question directly on IV4, negative findings and negative results ARE IMPORTANT. There is an unfortunate bias towards publishing/reporting positive results only, but negative results are equally important as they yield information on treatments we believed had effects or relationships on the outcome. Something like,

"prior literature suggested IV4 and DV had a strong relationship but our studies could not corroborate that, potentially due to X,Y, or Z.. Future studies will require a closer examination to confirm or dispute the assumed relationship b/w IV4 and DV. In addition, we found that IV1-3 were related to DV, confirming prior studies... "

Here are a couple of papers illustrating why publishing negative results keeps science honest and pushes it forward:

https://www.nature.com/articles/d41586-017-07325-2

Weintraub, P. G. (2016). The importance of publishing negative results. Journal of Insect Science, 16(1).

They describe the importance and also give strategies on how to publish negative results. Hope this helps.