0
$\begingroup$

Qualifier: I am not a statistician. I have 4 data sets and want to determine whether the 4 regression coefficients obtained are statistically significantly different. I compared each pair of coefficients separately (6 hypotheses) with t-tests and found no significant difference (t values ~1 for 6 d.f.). Does this support my null hypothesis (there is no significant difference in the coefficients) or do I need to make a correction because I am comparing multiple hypotheses. That is, if I made a correction would it only make the level of significance even lower?

$\endgroup$
2
  • 1
    $\begingroup$ If the form of the regression is the same for all 4 data sets, you could have more power to detect differences in regression coefficients among data sets by analyzing all your data together in a single regression, including an indicator variable for the data set, and seeing if there is a significant interaction between the data-set variable and the variable(s) whose regression coefficient(s) you are examining. Editing your question to provide more details about your study might help. $\endgroup$
    – EdM
    Jan 4, 2017 at 15:53
  • $\begingroup$ Another advantage of the suggestion made by @EdM is that you get a single, more powerful test. $\endgroup$
    – mdewey
    Jan 4, 2017 at 16:23

1 Answer 1

1
$\begingroup$

Answering briefly on this.

Using a P-value in Null Hypothesis Significance Testing (NHST) does not allow you to draw conclusions about your null hypothesis, just that you cannot reject it at the observed data. This is a common misconception. See e.g. Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old and continuing controversy. Psychological Methods, 5, 241-301.

Instead, you could calculate confidence intervals to get an idea about the population estimate, see for example Abserson 2002 - or use Bayesian statistics, but that is another chapter.

You are right that your P-values will be even higher when correction for multiple comparisons is performed.

$\endgroup$
2
  • $\begingroup$ I think you mean a high P-value. If the P-value is sufficiently low common practice is to to reject the null hypothesis. $\endgroup$ Jan 4, 2017 at 20:22
  • $\begingroup$ Thanks for the addition, I was looking for a better way to say this. Incorporated an edit! $\endgroup$
    – David
    Jan 4, 2017 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.