Statistical methods are also used in the social science and in my major, Political Science, researchers often use regression and related techniques. They run regressions for the collected data and discuss whether the key variable (the variable they are interested in their research) is statistically significant.
However, I learned from a book written by statisticians that regression is often used for prediction: estimate model parameters by given data and use it for future prediction. To have better predictions, one must avoid overfitting.
I know one can increase data fitting, for example by adding quadratic terms, which might returns statistically significant results. And since social scientists (political scientists) doesn't put so much emphasis on prediction, I think there is a possibility that statistically significant result occurs just because of overfitting.
Does this mean that background theory is important to select which variables to be put in the model?