Can p values be used to show impact of treatment

Question

Does it make sense to use the differences in p value to show a tendency or the 'importance' of the effect of a treatment. for example, I have treated a contaminated soil and I test the treatments against one control. I can see the high molecular weight contaminants are more affected by the treatment and that the p values (testing treatment against control) for these contaminants are lower than the ones of the low molecular weight ones, which if im right would mean that the significance is higher.

This seems to correlate with the graph, but could this difference in significance seen through the p values be used in itself to show the difference in the impact of the treatment, I mean in the discussion of your results.

Thank you in advance.

Answer 1

Let me paraphrase your question: "I statistically tested a hypothesis and obtained some p-values. Can I use these p-values to evaluate a different hypothesis?" I don't think that is a good idea. You are asking two different scientific questions. Consequently, you'll want to use two different tests to evaluate the two different hypotheses.

The first question is: "Does the treatment affect any contaminants" Here, you're comparing the different treatments against the control. Most likely, you have calculated a slope of some sort and shown that it is different from zero. This is where you got your p-values. However, these p-values do not address the second question (see below), and I suggest you do not use them to answer the second question.

The second question is: "Is contaminant A (high mw) affected more strongly than contaminant B (low mw)?". For this, I suggest you test whether the effect of treatment on the contaminants significantly differs. Note that you should use a fair normalization, for example use %reduction, rather than reduction in mass such that the effects you measure (e.g. slopes) can be compared at all. Using the estimated slopes and their uncertainties will allow you to perform a test (e.g. t-test) for non-zero difference.

Answer 2

The usual way would be to measure the treatment effect as the difference of both effects: $\beta_{Treated}$ - $\beta_{control}\equiv \theta$. In your case, you need to calculate $\theta$ and then test its significance.

The difference in p-values may eventually give you some elements for discussion (or rather speculation ?) but the discussion will become less rigorous as the hypothesis testing itself suffices. An straight measure of the treatment effect differential is given by the statistic behind $\theta$ or by $\hat\theta$ itself e.g. if the hypothesis testing is based on an statistic that folows a t-student distribution $T_n\sim t_q$ then it's easy to argue that the statistic proportional to the estimated difference $\hat \theta$.

It seems that you have treated two soils ? and that you want to asses the RELATIVE effectivity by comparing both tests on each soil? If that's the case a test : $H_0: \theta_1 -\theta_2 \neq 0 $ or $\hat\theta_1-\hat\theta_2$ would provide you with an objective measure and hypothesis testing of the relative performance.

Answer 3

I think one of the key reasons why you usually can't do what you are suggesting is sample size vs variability. A p-value tells you whether or not the available data can tell you that you have detected a difference. It does not directly tell you the magnitude of the difference. You can have correctly selected the hypothesis test and have a situation where the null hypothesis should be rejected but your sample size is too small to detect the actual effect size. This could mean that you need a very large sample because of large random error or that the chosen sample size turned out to be too small. When you try to compare p-values from different studies the differences in sample size muddies the comparison. Of course I agree with Procrastinator's comments as well.