I have a large data set (>1000 obs) and i'm performing regressions tests, both linear and logistic, on a series of clinical outcomes.
In this test I verify the effect of interactions between two cat variables by creating a dummy interaction variable with the two vars joined in one (with R is done using the interaction() fuction). In the equation are present also other vars for adjustment.
After every test I perform a post-hoc to verify if there is difference between every level of this interaction variable. The test is performed via glht() of the multcomp package in R. I believe, (i'm not 100% sure) that the post-hoc is performed using Tukey methodology.
Of course applying the post-hoc procedure I loose significance. Often appears that results that were significant in the regression test become not significant. It's ok. I always think that p<0.05 it's a very shallow threshold, especially when you have large numbers like I have. Some times it even happens that some relevant effects between levels that were not show in the regression become visibile in the post hoc and this is a good thing.
My question is therefore, when you perform a post hoc of a regression analysis, using Tukey, and with these large numbers, it's still easier to make a type 1 error and accept no real results, or a type 2 error, being too hard on data and loosing interesting results?
Or it's impossible to know? (probably this is the correct answer)