1
$\begingroup$

I have a large data set (>1000 obs) and i'm performing regressions tests, both linear and logistic, on a series of clinical outcomes.

In this test I verify the effect of interactions between two cat variables by creating a dummy interaction variable with the two vars joined in one (with R is done using the interaction() fuction). In the equation are present also other vars for adjustment.

After every test I perform a post-hoc to verify if there is difference between every level of this interaction variable. The test is performed via glht() of the multcomp package in R. I believe, (i'm not 100% sure) that the post-hoc is performed using Tukey methodology.

Of course applying the post-hoc procedure I loose significance. Often appears that results that were significant in the regression test become not significant. It's ok. I always think that p<0.05 it's a very shallow threshold, especially when you have large numbers like I have. Some times it even happens that some relevant effects between levels that were not show in the regression become visibile in the post hoc and this is a good thing.

My question is therefore, when you perform a post hoc of a regression analysis, using Tukey, and with these large numbers, it's still easier to make a type 1 error and accept no real results, or a type 2 error, being too hard on data and loosing interesting results?

Or it's impossible to know? (probably this is the correct answer)

Thanks!

Improve this question
$\endgroup$
3
  • $\begingroup$ Which Tukey post-hoc btw? $\endgroup$
    – russellpierce
    Jun 13, 2014 at 16:42
  • $\begingroup$ good question! i don't know. I used glht() with mcp(variable = 'Tukey') and then summary() to show the results. I would be grateful if someone would tell me whether I really used Tukey or not. $\endgroup$
    – Bakaburg
    Jun 13, 2014 at 16:45
  • $\begingroup$ or which kind of post test i really used, btw? $\endgroup$
    – Bakaburg
    Jun 14, 2014 at 10:18

1 Answer 1

Reset to default
1
$\begingroup$

Impossible to know because you don't know the base rate at which H0 is actually true or false. That being said...

You are already 'protected' against type I error when you restrict yourself to doing posthoc analyses only on elements of models that were statistically significant overall. In addition, posthoc correction procedures tend towards draconian criteria for evidence. Therfore, I'd tend to think that you'd be drifting towards increasing your risk of notable Type II error. So, I'd suggest that (if your peers will allow it) you look at the magnitude of the effects from your overall analysis and not fiddle around with posthocs.

Improve this answer
$\endgroup$
3
  • $\begingroup$ I did post hoc on the interaction variable against itself, testing every possible couple combination of the levels of the variable, whether they were significant or not in the normal (against baseline) regression test. Sorry I think I didn't understand you last suggestion. $\endgroup$
    – Bakaburg
    Jun 13, 2014 at 16:48
  • $\begingroup$ could you elaborate? $\endgroup$
    – Bakaburg
    Jun 14, 2014 at 10:18
  • $\begingroup$ Ah, well then you don't have the protection I assumed you had. So, go back to my initial answer "impossible to know". $\endgroup$
    – russellpierce
    Jun 15, 2014 at 13:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.