3
$\begingroup$

I have a dataset of about 7,900 rows, which includes 28 independent variables. This is a naturally occurring data that I am trying to explore which of these independent variables have significant effect on the dependent variable. I.e., I do not have any specific theory in mind that explains relationships that I can test. I am trying to figure out which independent variables and interactions I should enter the regression model?

I have already read this thread and this thread, but considering the fact that my case is exploratory rather than predicting or testing a specific hypothesis, I do not think the approaches suggested in the following threads can help me.

One approach that came to my mind was to write a loop that enters different combinations of the independent variables and checks the AIC or BIC of the model. At the end it chooses the model with the lowest AIC or BIC. I think the issue with this idea might be overfitting the data.

I'll appreciate it if you help me with answering this question or giving me more references to learn about this.

Improve this question
$\endgroup$
2
  • 3
    $\begingroup$ If you don't know anything about the relationship, you can't really make an assumption of linearity. Thus, I would use a model that doesn't make such an assumption. I'm partial to Generalized Additive models (GAM), which can shrink some smoothers to zero (i.e., remove them). But there are other methods from machine learning that could be useful (regression trees, neural networks, ...). However, it would be better if you sat down and thought about what mechanisms could have created your data. It's very rare that you know nothing about the processes behind your data. $\endgroup$
    – Roland
    Commented Dec 20, 2016 at 6:50
  • 2
    $\begingroup$ If you want to do exploratory data analysis,then take that serious and don't start with estimating models, but make lots and lots and lots of graphs, and stare at them till you know what to do. It is a very meditative exercise. $\endgroup$
    – Maarten Buis
    Commented Dec 20, 2016 at 8:49

1 Answer 1

Reset to default
1
$\begingroup$

You are right, one of the ways to choose consists in picking models with the lowest AIC/BIC statistics. As you pointed out, this approach has a bias towards overfitting the model, including more variables than the true specification in general.

A R-Project package that might help you is: https://cran.r-project.org/web/packages/glmulti/glmulti.pdf

Further details of this package in: https://www.jstatsoft.org/article/view/v034i12/v34i12.pdf

Improve this answer
$\endgroup$
1
  • $\begingroup$ Actually AIC and BIC have specific penalty functions to mitigate the overfitting problem. Using R-square and maximizing it lead to an overfitted model. I am having a hard time figuring out what you want to do with a model you pick for "exploratory" analysis if not to predict Y. A list of explanatory variables can be misleading because of the potential multicollinearity problem. Maybe an exploratory phase could eliminate some of the variables that are unimportant but not necessarily all of them $\endgroup$
    – Michael R. Chernick
    Commented Dec 20, 2016 at 2:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.