Hypothesis Testing on coefficients in two subsets of data after Stepwise Regression

Question

Is it a reasonable approach to run a hypothesis test to test whether the coefficients of a variable in two regressions on two different subsets of the same population are different if you have used stepwise regression to build the model in each subset? So assuming the variable you want to test is included in the regression for each of the subsets but other independent variables are included for one and not for the other, can you still test for a significant difference in the variable you want to examine?

If so, what approach would be optimal? I was thinking about using an interaction term between the variable which defines the two subset variables (in this case gender) and the variable I want to test for differences in the coefficient in (in this case years of education) however I'm not sure exactly how to go about this and which regression I'd use the interaction term in (i.e whether it would be the male or female regression).

Answer 1

No.

Stepwise model building is not reasonable, but is cargo cult science. The $p$-values produced from stepwise regression model building do not have the typical meaning of "probability of observing the estimate/test statistic assuming the null hypothesis is true," but rather "probability of observing the estimate/test statistic based on a series of unstated conditionals that are almost certainly predicated on some number of variables not included in the presented model."

Use stepwise regression if you want to appear to be performing meaningful statistical analysis, while providing results (estimates, "noise" variables, missing "real" variables, $p$-values, $R^{2}$, etc.) that are very likely to be biased.

Some relevant citations
Babyak, M. A. (2004). What you see may not be what you get: A brief, nontechnical introduction to overfitting in regression-type models. Psychosomatic Medicine, 66:411–421.

Flom, P. L. and Cassell, D. L. (2007). Stopping stepwise: Why stepwise and similar selection methods are bad, and what you should use.

Henderson, D. A. and Denison, D. R. (1989). Stepwise regression in social and psychological research. Psychological Reports, 64:251–257.

Huberty, C. J. (1989). Problems with stepwise methods—better alternatives. Advances in Social Science Methodology, 1:43–70.

Hurvich, C. M. and Tsai, C.-L. (1990). The impact of model selection on inference in linear regression. The American Statistician, 44(3):214–217.

McIntyre, S. H., Montgomery, D. B., Srinivasan, V., and Weitz, B. A. (1983). Evaluating the statistical significance of models developed by stepwise regression. Journal of Marketing Research, 20(1):1–11.

Pope, P. T. and Webster, J. T. (1972). The use of an $F$-statistic in stepwise regression procedures. Technometrics, 14(2):327–340.

Rencher, A. C. and Pun, F. C. (1980). Inflation of $R^{2}$ in best subset regression. Technometrics, 22(1):49–53.

Romano, J. P. and Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica, 73(4):1237–1282.

Sribney, B., Harrell, F., and Conroy, R. (2011). Problems with stepwise regression.

Steyerberg, E. W., Eijkemans, M. J., and Habbema, J. D. F. (1999). Stepwise selection in small data sets: a simulation study of bias in logistic regression analysis. Journal of Clinical Epidemiology, 52(10):935–942.

Thompson, B. (1995). Stepwise regression and stepwise discriminant analysis need not apply here: A guidelines editorial. Educational and Psychological Measurement, 55(4):525–534.

Whittingham, M., Stephens, P., Bradbury, R., and Freckleton, R. (2006). Why do we still use stepwise modelling in ecology and behaviour? Journal of Animal Ecology, 75(5):1182–1189.

Wilkinson, L. (1979). Tests of significance in stepwise regression. Psychological Bulletin, 86(1):168–174.