May a not-significant regression coefficient be used for the estimation of yhat and the calculation of residuals?

Question

I am working on a thesis where I have panel data with company information (such as industry, earnings, etc). I am trying to replicate the methodology of a paper whereby I conduct a cross-sectional OLS regression to then estimate yhat for each section. The difference between Y and yhat (basically the residuals) is going to be the variable I need for further analysis. The regression looks as follows: $Y = b_1*A + b_2*B + e$

Now in the paper, they suggest to do the cross-sectional analysis splitting by industry, year and accounting standard. However, if I do this the resulting subsamples are extremely small (sometimes less than 10 observations). I decided not to split by industry in order to reduce the number of subsamples. The subsamples are still fairly small, some only have 10 or 12 observations. Consequently, if I run the regression, the coefficients are not significant (p>0.05). My question is: since I am merely using the coefficients to estimate the residuals, can use them even if they are not significant? Or does that make every analysis that follows (based on those residuals) illegitimate?

Answer 1

Using insignificant coefficients in prediction

Hypothesis testing and prediction are separate objectives. Significance is only meaningful for the former. While significant coefficients might be better predictors than non-significant predictors, this does not have to be the case at all.

I advice you to split your dataset into a train and test set. You then fit the regression model on the train set and evaluate the performance on the test set. The model with the lowest test error is then the 'winner'. You may want to read up on prediction and model selection. $p$-values are not suitable for this purpose.

Introduction and Elements of Statistical Learning provide excellent explanations of predictive techniques and are available for free in PDF:

http://www-bcf.usc.edu/~gareth/ISL/

https://web.stanford.edu/~hastie/ElemStatLearn/

Other remarks

Rather than splitting by industry, year and accounting standard, you can also include those variables in the regression analysis. If you believe the unique combinations of industry, year and accounting standard should have separate estimates, include interaction terms with A and B.

In R, simply use the * operator in the formula:

lm(y ~ A * industry * year * accounting standard + B * industry * year * accounting standard)