1
$\begingroup$

Statistical methods are also used in the social science and in my major, Political Science, researchers often use regression and related techniques. They run regressions for the collected data and discuss whether the key variable (the variable they are interested in their research) is statistically significant.

However, I learned from a book written by statisticians that regression is often used for prediction: estimate model parameters by given data and use it for future prediction. To have better predictions, one must avoid overfitting.

I know one can increase data fitting, for example by adding quadratic terms, which might returns statistically significant results. And since social scientists (political scientists) doesn't put so much emphasis on prediction, I think there is a possibility that statistically significant result occurs just because of overfitting.

Does this mean that background theory is important to select which variables to be put in the model?

$\endgroup$
  • 1
    $\begingroup$ I suspect most analyses are in a way trying to predict, even if it is not clearly state. It is a serious issue discussed e.g. by Gelman (e.g. stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf), if people overfit models and/or repeatedly modify their model until p<0.05 (and thus, publishable). Claims that such an extreme result/correlation would have only occurred by chance 5% of the time are then usually untrue and it is quite likely something is a chance finding. $\endgroup$ – Björn Nov 7 '15 at 6:59
  • $\begingroup$ Thank you for your comment, @Björn. I can agree that in statisticians' perspective, most analyses do more or less prediction. However, I think political scientists have been interested in the effect of a certain variable on another variable, such as "does foreign aid have positive / negative effect on democratization". I wonder how, as a political science student, we can use statistical techniques that fundamentally seeks prediction to answer our questions. $\endgroup$ – user2978524 Nov 7 '15 at 7:11
  • $\begingroup$ Another paper you might find interesting that discusses this issue, for the study of war, is "The perils of policy by p-value". To some extent your question seems rhetorical as a theory produces falsifiable predictions, so yeah the background theory is important in variable selection. Unfortunately, there seems to be a heavier focus on statistical inference rather than prediction in the social sciences leading to overfitting and use of garbage can or kitchen sink models. $\endgroup$ – HorseOfTheYear Nov 7 '15 at 15:02
  • $\begingroup$ Thank you, @BlankUsername. Yes, I totally agree that social scientists are more interested in statistical inference than prediction. That's why I wonder why regression is heavily used at least in the past 10 years. I'm still not sure how one can interpret statistically significant coefficients as "average causal effect", since it only says one unit increase of the independent variable is related to increase/decrease of dependent variable. I also understand there is a new trend of using design-based approaches to realize experimental setting (natural experiment) and get precise causal effect. $\endgroup$ – user2978524 Nov 8 '15 at 0:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.