Qualifier: I am not a statistician. I have 4 data sets and want to determine whether the 4 regression coefficients obtained are statistically significantly different. I compared each pair of coefficients separately (6 hypotheses) with t-tests and found no significant difference (t values ~1 for 6 d.f.). Does this support my null hypothesis (there is no significant difference in the coefficients) or do I need to make a correction because I am comparing multiple hypotheses. That is, if I made a correction would it only make the level of significance even lower?

  • 1
    $\begingroup$ If the form of the regression is the same for all 4 data sets, you could have more power to detect differences in regression coefficients among data sets by analyzing all your data together in a single regression, including an indicator variable for the data set, and seeing if there is a significant interaction between the data-set variable and the variable(s) whose regression coefficient(s) you are examining. Editing your question to provide more details about your study might help. $\endgroup$ – EdM Jan 4 '17 at 15:53
  • $\begingroup$ Another advantage of the suggestion made by @EdM is that you get a single, more powerful test. $\endgroup$ – mdewey Jan 4 '17 at 16:23

Answering briefly on this.

Using a P-value in Null Hypothesis Significance Testing (NHST) does not allow you to draw conclusions about your null hypothesis, just that you cannot reject it at the observed data. This is a common misconception. See e.g. Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241-301.

Instead, you could calculate confidence intervals to get an idea about the population estimate, see for example Abserson 2002 - or use Bayesian statistics, but that is another chapter.

You are right that your P-values will be even higher when correction for multiple comparisons is performed.

  • $\begingroup$ I think you mean a high P-value. If the P-value is sufficiently low common practice is to to reject the null hypothesis. $\endgroup$ – Michael R. Chernick Jan 4 '17 at 20:22
  • $\begingroup$ Thanks for the addition, I was looking for a better way to say this. Incorporated an edit! $\endgroup$ – David Jan 4 '17 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.