I am studying a treatment that degrades device quality and performed an independent sample test on two batches that had different initial quality. I analyzed the data with a two variable linear regression, the details are in my previous question here: How to combine 2 unpaired t tests that test the same effect

My specific questions in bold at the bottom of the post after this background information.

For easy reference the output of the regression is here:

enter image description here

I am interested in isolating and quantifying the effect of the treatment variable. Because the estimate for the treatment status effect magnitude is ~25% of the intercept this effect is important for us to consider.

However since the standard error of the treatment effect is 50% of the effect size there is a question of if this is a "real" effect.

Looking at the data below (1 standard deviation error bars), it is obvious that for either of the initial quality batches we can draw a horizontal line that passes within the error bar range for both the treated and untreated devices. This is used as justification that my conclusion is not reliable.

enter image description here

In presenting my data I used the P-value of the treatment variable as measure of the statistical significance of my finding. My exact words were the following:

“although the uncertainty appears large compared to the size of the effect, the observation of excess loss from the [treatment] is statistically significant with a p-value of 0.06. This indicates that if the [treatment] actually [does] not contribute excess loss then we would observe loss of this magnitude or greater only 6% of the time.”

This statement has been strongly criticized as incorrect and a common misunderstanding of the P-value. My understanding is that the technical definition of the P-value is: “Under the assumption of the null hypothesis, we would expect to see an effect this extreme or stronger P of the time”

I believe in my application the null hypothesis would be: The treatment has no effect.

My questions are the following:

1) If my definition of the P-value is correct and I am applying the null hypothesis correctly I do not understand what is wrong with the statement that I made. What is my error and how should I amend my statement or analysis?

2) What additional or alternative analysis could / should I do to make my statement about the significance of the effect more complete, precise, constrained, or specific?

  • 2
    $\begingroup$ (1) Can you explain at all why your statement "has been strongly criticized as incorrect"? (2) You haven't quite explained your plot, but it looks like means & standard deviations estimated separately for each of the four groups (on a reciprocal loss scale - why?). Usually you'd plot standard errors or confidence intervals for the mean. Other posts here deal with what conclusions you might or might not draw from overlap or non-overlap, but note that with large sample sizes you could have very precise estimates of each mean & one-std-deviation error bars that still overlap. $\endgroup$ – Scortchi - Reinstate Monica Jul 5 '16 at 14:43
  • $\begingroup$ I don't completely understand the criticism of my statement, which is why I'm posting here. :). I was told that this discription of the p-value is a very common misconception and that the p-value "says no such thing". I was also directed to this article on the misuse of p-values: nature.com/news/… I'm imagining that part of the criticism is the way I verbalized my understanding of the null hypothesis as "If the treatment actually does not contribute to excess loss..." $\endgroup$ – Canaryyellow Jul 6 '16 at 3:56
  • $\begingroup$ In response to (2). I edited the plot a little to make it easier for this community to parse. In my field The Quality factor = 1/Loss is what is typically reported. However since loss is quantity that adds in the normal way: Loss_total = Loss_(due to initial quality) + Loss_(due to treatment) I was working with the loss when I did the regression. $\endgroup$ – Canaryyellow Jul 6 '16 at 4:03

Assuming that there was random assignment of treatments, your analysis seems appropriate and your way of interpreting the p-value seems right to me. Of course, I could understand anyone that feels that they would want to see more evidence and that p=0.06 is not very compelling (even if the significance level was specified to be e.g. 0.1 beforehand - not quite sure what you specified, presumably something > 0.06) - as long as they are consistent and will not consider a p-value just below 0.05 as much different from p=0.06. Admitting to some uncertainty about whether the observed effect is definitely real may be appropriate. I can only speculate and may be totally wrong about this (since I lack details), but perhaps the reviewer got an impression that you equate "just about statistical signficant" = "my observed finding definitely reflects a proven scientifc truth" and due to such an impression pushes back very hard?

It looks to me like - the hopefully pre-specified, otherwise I could see concerns about using the p-value the way you do - inclusion of the (pre-treatment) quality in the model seems logical and appears to have some explanatory value - and importantly the treatment effect looks very similar in each category. It should be no surprise that splitting the data into sub-categories increases the uncertainty about the treatment estimate, if it is only estimated within sub-category. However, as I understand it, the assumption is that the treatment effect is constant across categories and the overall treatment effect is what is of interest so that the precision of the estimate in each sub-category should not be of primary concern.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.