Is it valid to iterate over every permutation of a regression specification and compute an "average significance?" I had an idea, and was wondering if it's ever done and, if so, how to do it in an appropriate manner. 
Let's say I run an OLS model and the results come back significant.  There is discussion as to whether the positive results hold up given a different set of control variables.  However, it's theoretically unclear which control variables are actually relevant, and we don't just want to throw in the kitchen sink.
Let's say it's computationally feasible to run every single model (with every possible permutation of control variables), and we collect the coefficient and standard error of the coefficient of interest in each.  We can report the proportion of models where this coefficient is significant (correcting for multiple tests). Is there some kind of omnibus procedure that can be done showing that the test statistic is significant in a significant number of alternative specifications?  Is this ever done?  
 A: One similar practice is called model averaging. You fit many models, make many predictions, and average the results, weighting each by the posterior probability that the model is correct. There's a nice introduction here.
https://www2.stat.duke.edu/courses/Spring05/sta244/Handouts/press.pdf
I would advise you not to do this with coefficients, because it's not how BMA was meant to be used. The issue is that the same coefficient can have a different interpretation and target parameter in different models. Because of this, averaging two coefficients often doesn't make conceptual sense. Katharine Bannar explains via example: in a model for brain weight based on body size and gestation time, gestation time will have little association with brain weight once body size is already accounted for. If body size is left out of the model, the gestation coefficient would increase dramatically. There is more explanation here.
https://esajournals.onlinelibrary.wiley.com/doi/full/10.1002/eap.1419
Unfortunately, it sounds like you need to pick a set of control variables even if the theory about them is unclear. You could think about:

*

*What's the penalty for including variables that are not relevant? Often, this worsens variance but not bias.

*What's the penalty for including variables that are related, but causally downstream instead of upstream? This is a fraught topic but there is more and more lit now to turn to on causal interpretation of stuff you control for in a regression. One nice place to start is this paper. It outlines conditions when you should or shouldn't adjust for a given feature, including real examples of increased and decreased bias.

