I am studying a treatment that degrades device quality and performed an independent sample test on two batches that had different initial quality. I analyzed the data with a two variable linear regression, the details are in my previous question here: How to combine 2 unpaired t tests that test the same effect
My specific questions in bold at the bottom of the post after this background information.
For easy reference the output of the regression is here:
I am interested in isolating and quantifying the effect of the treatment variable. Because the estimate for the treatment status effect magnitude is ~25% of the intercept this effect is important for us to consider.
However since the standard error of the treatment effect is 50% of the effect size there is a question of if this is a "real" effect.
Looking at the data below (1 standard deviation error bars), it is obvious that for either of the initial quality batches we can draw a horizontal line that passes within the error bar range for both the treated and untreated devices. This is used as justification that my conclusion is not reliable.
In presenting my data I used the P-value of the treatment variable as measure of the statistical significance of my finding. My exact words were the following:
“although the uncertainty appears large compared to the size of the effect, the observation of excess loss from the [treatment] is statistically significant with a p-value of 0.06. This indicates that if the [treatment] actually [does] not contribute excess loss then we would observe loss of this magnitude or greater only 6% of the time.”
This statement has been strongly criticized as incorrect and a common misunderstanding of the P-value. My understanding is that the technical definition of the P-value is: “Under the assumption of the null hypothesis, we would expect to see an effect this extreme or stronger P of the time”
I believe in my application the null hypothesis would be: The treatment has no effect.
My questions are the following:
1) If my definition of the P-value is correct and I am applying the null hypothesis correctly I do not understand what is wrong with the statement that I made. What is my error and how should I amend my statement or analysis?
2) What additional or alternative analysis could / should I do to make my statement about the significance of the effect more complete, precise, constrained, or specific?