Can a slightly overfitted model be useful for exploratory (i.e. hypotheses generating) modelling?

Question

Let's say you have a set of potential explanatory variables (e.g. p = 8) that you think are important to explain your response variable ($Y$) but your sample is too small to include them all in the same model (e.g. n = 50) but you do it anyway, would this overfitted model be worth something to generate new testable hypotheses?
For instance, if the model's summary shows that $X_{2}$ and $X_{5}$ have a large (positive or negative) effect on $Y$, would it be appropriate to conclude something like: "the results suggest that these two variables may be influencial, so we recommend testing the effects of $X_{2}$ and $X_{5}$ on $Y$ in a controlled experiment"?

I know I could use a penalization method instead but I still would like answers to these questions.

Answer 1

Essentially, you suggest to (1) do variable selection from an under-powered study and (2) validate the selected variables with a follow-up experiment (presumably, with sufficient power).

Step (1) has the usual pitfalls of variable selection from a small dataset: the most significant coefficients are most likely to be over-estimates of the true effect sizes. This is related to the concepts of type S (sign) and type M (magnitude) errors [1].

Fortunately, if that's the case, step (2) will produce unbiased estimates of the effect sizes of "influential" variables because regression to the mean is a powerful phenomenon.

So you will do good science by externally validating the variable selection process. And in this sense your proposal for small-data driven hypothesis generation is "appropriate", which is not the same as "efficient".

[1] A. Gelman and J. Carlin. Beyond power calculations: Assessing type S (sign) and type M (magnitude) errors. Perspectives on Psychological Science, 9(6):641–651, 2014.