How to ascribe and defend statistical significance in linear regression. P-value interpretation and alternatives I  am studying a treatment that degrades device quality and performed an independent sample test on two batches that had different initial quality.  I analyzed the data with a two variable linear regression, the details are in my previous question here:  How to combine 2 unpaired t tests that test the same effect 
My specific questions in bold at the bottom of the post after this background information.
For easy reference the output of the regression is here:

I am interested in isolating and quantifying the effect of the treatment variable.  Because the estimate for the treatment status effect magnitude is ~25% of the intercept this effect is important for us to consider.
However since the standard error of the treatment effect is 50% of the effect size there is a question of if this is a "real" effect.
Looking at the data below (1 standard deviation error bars), it is obvious that for either of the initial quality batches we can draw a horizontal line that passes within the error bar range for both the treated and untreated devices.  This is used as justification that my conclusion is not reliable.

In presenting my data I used the P-value of the treatment variable as measure of the statistical significance of my finding.  My exact words were the following:
“although the uncertainty appears large compared to the size of the effect, the observation of excess loss from the [treatment] is statistically significant with a p-value of 0.06. This indicates that if the [treatment] actually [does] not contribute excess loss then we would observe loss of this magnitude or greater only 6% of the time.”
This statement has been strongly criticized as incorrect and a common misunderstanding of the P-value. My understanding is that the technical definition of the P-value is:  “Under the assumption of the null hypothesis, we would expect to see an effect this extreme or stronger P of the time”
I believe in my application the null hypothesis would be:  The treatment has no effect.
My questions are the following:
1)  If my definition of the P-value is correct and I am applying the null hypothesis correctly I do not understand what is wrong with the statement that I made.  What is my error and how should I amend my statement or analysis?
2)  What additional or alternative analysis could / should I do to make my statement about the significance of the effect more complete, precise, constrained, or specific?
 A: Assuming that there was random assignment of treatments, your analysis seems appropriate and your way of interpreting the p-value seems right to me. Of course, I could understand anyone that feels that they would want to see more evidence and that p=0.06 is not very compelling (even if the significance level was specified to be e.g. 0.1 beforehand - not quite sure what you specified, presumably something > 0.06) - as long as they are consistent and will not consider a p-value just below 0.05 as much different from p=0.06. Admitting to some uncertainty about whether the observed effect is definitely real may be appropriate. I can only speculate and may be totally wrong about this (since I lack details), but perhaps the reviewer got an impression that you equate "just about statistical signficant" = "my observed finding definitely reflects a proven scientifc truth" and due to such an impression pushes back very hard?
It looks to me like - the hopefully pre-specified, otherwise I could see concerns about using the p-value the way you do - inclusion of the (pre-treatment) quality in the model seems logical and appears to have some explanatory value - and importantly the treatment effect looks very similar in each category. It should be no surprise that splitting the data into sub-categories increases the uncertainty about the treatment estimate, if it is only estimated within sub-category. However, as I understand it, the assumption is that the treatment effect is constant across categories and the overall treatment effect is what is of interest so that the precision of the estimate in each sub-category should not be of primary concern.
