I am working with a data set of ~1200 rows and 60 variables, and I'm trying to build a multiple linear regression model. I do this by separating 10% of the dataset to be used for validation and I use the rest of the data to train the linear model.
I have found that I am able to build very strong regression models, with an adjusted coefficient of determination (R-square) around .998 and <3% of the predicted values trained on the validation set deviating from the true values by more than accepted threshold (also 3%). I also have all the variables selected (~20 variables) as significant at the 5% level, including first-order interaction variables.
However, I have found that by randomly selecting a different 10% of the data as validation, the regression model I trained previously does not consistently have its explanatory variables remain significant, and the proportion of predicted values within the accepted threshold varies, sometimes improving but usually decreasing. The coefficients of the variables also vary. However, the R-square remains extremely high.
How can I be sure to build a stable model? Or rather, what could I reasonably consider a stable model? I'm confused by the apparent disparity between how the R-square and the actual predicted values explain the strength of the model. I'm also concerned that variables previously considered to be significant at the 5% level often fail to remain so. Can anyone give any tips on how to approach model building so that the proportion of the data saved for validation doesn't drastically alter explanatory variables selected?