2
$\begingroup$

I have data on about 8000 persons and I am trying to find independent predictors of a health outcome variable (yvar). The predictor variables are age, gender, height, city and 3 other predictor variables (xvar1, xvar2, xvar3). Some are continuous while others are categorical. The categorical variables are kept as such and not converted to numbers (e.g. 'M' and 'F' are levels in gender). The outcome variable (yvar) is continuous.

If I use following code in R (applying all interactions):

lm(yvar~age+gender+heigth+city+xvar1+xvar2+xvar3)

I get 5 of these 7 to be with $p<0.05$ (many are much less than 0.05) and overall $R^2$ of 0.11

On using following code:

lm(yvar~age*gender*heigth*city*xvar1*xvar2*xvar3)

I get $R^2$ of 0.18 but NONE of the predictors has $p<0.05$

What do I conclude from this? Should I or should I not use interactions? What is the best way to analyze such data?

Also, should I use one of above formats or following format:

lm(yvar~(age+gender+heigth+city+xvar1+xvar2+xvar3)*(age+gender+heigth+city+xvar1+xvar2+xvar3) )

These produce only 2-way interactions and not all combination interactions as in second format.

Improve this question
$\endgroup$
2
  • $\begingroup$ Terminology tip: The word "parameter" used to mean just index or measure or property or variable clashes with its established statistical use as meaning a unknown constant you are estimating. $\endgroup$
    – Nick Cox
    Commented Dec 9, 2014 at 12:40
  • $\begingroup$ I changed from 'parameter' to 'outcome variable' in the question. $\endgroup$
    – rnso
    Commented Dec 9, 2014 at 17:35

2 Answers 2

Reset to default
8
$\begingroup$

The approach you are using is devoid of input from subject-matter knowledge. This is usually a recipe for trouble. Using $P$-values to guide model specification is fraught with statistical problems. And you may be using the wrong statistical tests as the basis for your concern, i.e., you may be trying to define a main effect in the presence of an interaction effect. The appropriate tests to use are combined tests of main effects + interaction effects. But more importantly having 7th-order interactions in your model is a huge overkill resulting in estimation of far too many parameters, making everything unstable. It would be better to stick to second-order interactions (in R (a + b + c)^2) but this approach is still a bit dangerous and you have not included all the original variables as main effects, which will give you a false impression of importance of interactions.

Try to think of a statistical analysis not as a fishing expedition but rather through reasoned model specification.

Improve this answer
$\endgroup$
1
  • 2
    $\begingroup$ I would add that this kind of high-order interactions would be impossible to interpret and would have a high potential for overfitting. $\endgroup$
    – Tim
    Commented Dec 9, 2014 at 12:42
2
$\begingroup$

If I understand you correctly, you have a number of explanatory variables but not much clue which of them could be relevant and which not. Here are a few ways to proceed:

  1. Try out all possible subsets of variables and pick the one that gives a regression with the smallest Bayesian information criterion (BIC) value. See e.g here for relevant R functions. In you want to allow for interactions, too, then either define new regressors by multiplying the existing ones in a pairwise fashion or look for an existing function to do that for you.
    Using BIC will help strike the right balance between possible overfitting and underfitting. If you intend to use your model for forecasting rather than explanation, use Akaike information criterion (AIC) instead of BIC.

  2. Forward or backward stepwise selection: start from a small model and add regressors one by one based on their relevance (broadly speaking) or start from a general model and remove regressors one by one, again based on their relevance.

  3. Shrinkage methods (LASSO, ridge regression, elastic net, principal components regression, partial least squares): if you want to reduce the mean squared error of your model and do not care exclusively about the unbiasedness of your estimates, you might want to allow for some bias to gain a decrease in variance. This makes sense if you intend to forecast but not so much if your study is explanatory.

See Hastie et. al "The Elements of Statistical Learning" chapter 3 subsections 3.3-3.7 for a more detailed overview.

To address your concern about p-values: they are not sacred and many say they are given too much importance. Sometimes (when there is a lot of data) even irrelevant variables become statistically significant, but the magnitudes of their coefficients are small and substantively negligible. Sometimes two or more variables are jointly significant but not so individually. Thus you have to interpret them carefully and not just mechanically.

Improve this answer
$\endgroup$
16
  • 1
    $\begingroup$ Note that some of these are more controversial than others. #2 is widely considered a very bad idea, for reasons well explained by Frank Harrell, a contributor here, in his book Regression modeling strategies (New York: Springer, 2001). I'd want to add 0. Consist subject-matter experts if you lack subject-matter expertise to consider what kinds of relationships are likely to be of interest or importance. $\endgroup$
    – Nick Cox
    Commented Dec 9, 2014 at 17:43
  • 1
    $\begingroup$ #1 is also a very bad idea. $\endgroup$
    – Frank Harrell
    Commented Dec 9, 2014 at 21:05
  • 1
    $\begingroup$ No. It is important to fully understand the assumptions made and the performance characteristics of any method for model development. $\endgroup$
    – Frank Harrell
    Commented Dec 10, 2014 at 11:52
  • 1
    $\begingroup$ This has been studied extensively and shown to fail. See for example my RMS course notes. Among the problems: overfitting/poor validation, ruined standard errors, bias, instability, ruined confidence interval coverage. $\endgroup$
    – Frank Harrell
    Commented Oct 14, 2020 at 12:16
  • 2
    $\begingroup$ It's a tough problem. Throughout my career I've never found exploratory interaction searching to be close to reliable. So it remains necessary to narrow potential interactions to include in the model. Ask the question "could we explain this interaction were it to be found important?" $\endgroup$
    – Frank Harrell
    Commented Oct 21, 2020 at 14:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.