Moving away from stepwise type of model building I do a lot of studies in which we have a disease/outcome and then we collect a lot of information on the patients such as age, gender, BMI, comorbidities, lifestyle factors etc. and then we run a model to see if something sticks. These studies are not to be considered as something you would create new guidelines for, but merely to give future researchers ideas.
So far we've done a lot of stepwise-type of model building in which we do a univariate regression for each independent variable and then include those with alpha <= 0.2 in a multiple regression model and finally report those with alpha <= 0.05 in the multiple model.
I've never quite understood this approach as I would normally use multiple regression to select variables of interest that may influence each other, regardless of their presenting alpha. To me it sounds like something you tell a graduate student to give them something to model their data with, without knowing any statistics to make sure they don't overfit their model.
My question is, I'm considering stopping with the stepwise-type of approach and starting to report studies more in the way of selecting which variables I included in a multiple regression model subjectively based on clinical reasoning.
Would this not strengthen the statistical part of the study?
 A: The approach that is currently used is wrong for many reasons, see Algorithms for automatic model selection for a nice summary. As for alternatives, there are two (three) main ones, depending on your goal:


*

*As you suggested, use domain knowledge to filter the data and then run one model, in which case inference holds and can be reported for all variables. Note: you really should use only domain knowledge and not use the data itself to filter the data, as that would just be another form of overfitting. Meaning you should not use summaries and plots to filter the data (some would call it data leakage). Note2: this won't necessarily "strenghten" the statistical model (it may end up being a worse model), but it will make it correct for interpretation.


Alternatively use a train/test set approach, do all the filtering and processing on the train set and then test your model (once) on the test set, and report these results.


*Use more modern approaches which incorporate regularization, such as LASSO. Here you can include all (or most) variables and hope the model will filter all the redundant variables for you. The downside is that you won't get inference results, but the model will select "important" variables.

A: Considering you're comfortable with the idea of making subjective assumptions, and it's not a model that will determine if someone lives or dies, I would suggest a Bayesian model might be a good option.
This would allow you to state your assumptions explicitly and visualize them in a graph, so it can be audited and interpreted in a fairly straightforward way, and still yields quantitative outputs.
