1
$\begingroup$

I was asked to do a correction of p-values for my analyses, but I'm not sure what p-values I am to correct, and presuming I'm using Bonferroni, what is the number that I am dividing the .05 level with. I have done three sets of multiple regression analyses, each with seven different time points for a total of 21 models. The method was backwards regression, independent variables remaining in the model at p < .05. The sets are comprised of the same group of independent variables entered, but the dependent variables differ (3 total x 7 time points). Due to the nature of backwards regression, obviously the final predictors in the model differ. Just to make the case clearer, by seven time points I mean the same variables have been measured at seven time points (i.e. A1, B1 to X1; A2, B2 to X2 etc).

So, what I really want to know is what is the correct way to go about this. Do I need to adjust the model ANOVA's p-values printed by SPSS, or the individual regression coefficients? Why this baffles me is the fact that in essence, a given time point's (potential) predictors are used for three different analyses. Then again, there are 7 actual regression analyses done for each set. This is of importance as some of the individual regression coefficients are close to .05 and if corrected, won't remain significant, whereas all the models even if corrected with what I think is the right level of alpha here, will remain significant.

Thank you in advance.

$\endgroup$
  • 1
    $\begingroup$ The issue is what are the hypotheses that you want to test simultaneously. The n you need would be the number of hypotheses. The p-value you are looking for is the probability of simultaneously rejecting all null hypotheses when the null hypotheses are true. Bonferroni is only one of several ways to approximate this p-value. In fact Bonferroni is very conservative in that it underestimates this p-value and is some cases by a large margin. You also could look at bootstrap and permutation methods (resampling approaches) as well as a method due to Tukey. $\endgroup$ – Michael Chernick Jan 5 '18 at 17:58
1
$\begingroup$

I can see three related issues:

A) What was your research question? Was there a main outcome and a clearly formulated hypothesis before you gathered data? The number of outcomes and predictors suggests not. If you are performing an exploratory analysis, say so in your report, refrain from making any strong conclusions, forget about p-value corrections, select the best outcome & model, and do a second experiment to replicate.

B) More importantly: It is commonly accepted nowadays that stepwise (forward/backward) regression is bad practice (e.g. see this, this, and especially this post). It will almost certainly lead you to false conclusions and your reviewers/referees should not let you pass. If you want to do efficient model selection, there are other techniques to do so (e.g. LASSO). If you want to do inference (i.e. make claims about your independent variables), go for the full model justified by the theory and stick with it (assuming your sample size is adequate for the number of predictors involved). If you insisted on the stepwise approach, you would not only have to correct for the obvious numbers of multiple comparisons (number of outcomes (3) * timepoints (7) = 21) , but also for all of the multiple comparisons hidden in your backwards regression (i.e. all possible combinations of your predictors, lets call this figure X). To do so in a Bonferroni fashion, just multiply your p-values obtained in SPSS with 3*7*X ... before comparing with your threshold good luck with that.

C) As far as I can see from your description you have a longitudinal research design, where you obtain repeated observations from the same experimental units. You then perform separate regressions (construct separate statistical models) for each time point. You can greatly reduce the number of multiple comparisons in your analysis by performing a single mixed model analysis for each outcome. However, this is not trivial and you may need help irl to do so. Additionally, I can recommend:

West, Brady T., Kathleen B. Welch, and Andrzej T. Galecki. Linear mixed models: a practical guide using statistical software. CRC Press, 2014.

and

Gelman, Andrew, and Jennifer Hill. Data analysis using regression and multilevelhierarchical models. Vol. 1. New York, NY, USA: Cambridge University Press, 2007.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.