I'm trying to show that in an equation of the form var1 ~ var2 + var3 + ... + varX, var2 has a has a statistically significant effect. However, there are many ways of defining var1, var3... varX (var 2 is a logical variable), say I got var1_v1, var1_v2, etc. I ran regressions over all the permutations of var1_1, var3_1, ..., varX_1, var1_2, var3_1, ... I'm interested in the regression coefficients of var1 ~ var2 + var3 + ... + varX, not specifically thoses of var1_1 ~ var2 + var3_1 + ... + varX_1, as I feel the general result is more relevant than the individual variables. I'm looking for literature on similar processes.
The main way I have to suggest that the ensemble of variables is collectively significant seems to be using the p-values < 0.1 (for var2), which happen on about 40% of the regressions. Can I just use Bernoulli distribution to say that it is a few standard deviation for the expected type I error, so there is something likely significant? Would there be a purely bayesian way of doing so without relying on the p-values?
Otherwise, the lowest BIC are showing me a p-value < 0.1 for var2. Could I just take an average of the BIC on the p<0.1 versus p>0.1?
Is there anything specific about running multiple regressions to get show that however you define variables, a link is there? Or do I have to break down the analysis to which variables seems the most relevant?
