# How to ascribe and defend statistical significance in linear regression. P-value interpretation and alternatives

I am studying a treatment that degrades device quality and performed an independent sample test on two batches that had different initial quality. I analyzed the data with a two variable linear regression, the details are in my previous question here: How to combine 2 unpaired t tests that test the same effect

My specific questions in bold at the bottom of the post after this background information.

For easy reference the output of the regression is here:

I am interested in isolating and quantifying the effect of the treatment variable. Because the estimate for the treatment status effect magnitude is ~25% of the intercept this effect is important for us to consider.

However since the standard error of the treatment effect is 50% of the effect size there is a question of if this is a "real" effect.

Looking at the data below (1 standard deviation error bars), it is obvious that for either of the initial quality batches we can draw a horizontal line that passes within the error bar range for both the treated and untreated devices. This is used as justification that my conclusion is not reliable.

In presenting my data I used the P-value of the treatment variable as measure of the statistical significance of my finding. My exact words were the following:

“although the uncertainty appears large compared to the size of the effect, the observation of excess loss from the [treatment] is statistically significant with a p-value of 0.06. This indicates that if the [treatment] actually [does] not contribute excess loss then we would observe loss of this magnitude or greater only 6% of the time.”

This statement has been strongly criticized as incorrect and a common misunderstanding of the P-value. My understanding is that the technical definition of the P-value is: “Under the assumption of the null hypothesis, we would expect to see an effect this extreme or stronger P of the time”

I believe in my application the null hypothesis would be: The treatment has no effect.

My questions are the following:

1) If my definition of the P-value is correct and I am applying the null hypothesis correctly I do not understand what is wrong with the statement that I made. What is my error and how should I amend my statement or analysis?

2) What additional or alternative analysis could / should I do to make my statement about the significance of the effect more complete, precise, constrained, or specific?

• (1) Can you explain at all why your statement "has been strongly criticized as incorrect"? (2) You haven't quite explained your plot, but it looks like means & standard deviations estimated separately for each of the four groups (on a reciprocal loss scale - why?). Usually you'd plot standard errors or confidence intervals for the mean. Other posts here deal with what conclusions you might or might not draw from overlap or non-overlap, but note that with large sample sizes you could have very precise estimates of each mean & one-std-deviation error bars that still overlap. Commented Jul 5, 2016 at 14:43
• I don't completely understand the criticism of my statement, which is why I'm posting here. :). I was told that this discription of the p-value is a very common misconception and that the p-value "says no such thing". I was also directed to this article on the misuse of p-values: nature.com/news/… I'm imagining that part of the criticism is the way I verbalized my understanding of the null hypothesis as "If the treatment actually does not contribute to excess loss..." Commented Jul 6, 2016 at 3:56
• In response to (2). I edited the plot a little to make it easier for this community to parse. In my field The Quality factor = 1/Loss is what is typically reported. However since loss is quantity that adds in the normal way: Loss_total = Loss_(due to initial quality) + Loss_(due to treatment) I was working with the loss when I did the regression. Commented Jul 6, 2016 at 4:03