2
$\begingroup$

I have data on about 8000 persons and I am trying to find independent predictors of a health outcome variable (yvar). The predictor variables are age, gender, height, city and 3 other predictor variables (xvar1, xvar2, xvar3). Some are continuous while others are categorical. The categorical variables are kept as such and not converted to numbers (e.g. 'M' and 'F' are levels in gender). The outcome variable (yvar) is continuous.

If I use following code in R (applying all interactions):

lm(yvar~age+gender+heigth+city+xvar1+xvar2+xvar3)

I get 5 of these 7 to be with $p<0.05$ (many are much less than 0.05) and overall $R^2$ of 0.11

On using following code:

lm(yvar~age*gender*heigth*city*xvar1*xvar2*xvar3)

I get $R^2$ of 0.18 but NONE of the predictors has $p<0.05$

What do I conclude from this? Should I or should I not use interactions? What is the best way to analyze such data?

Also, should I use one of above formats or following format:

lm(yvar~(age+gender+heigth+city+xvar1+xvar2+xvar3)*(age+gender+heigth+city+xvar1+xvar2+xvar3) )

These produce only 2-way interactions and not all combination interactions as in second format.

$\endgroup$
  • $\begingroup$ Terminology tip: The word "parameter" used to mean just index or measure or property or variable clashes with its established statistical use as meaning a unknown constant you are estimating. $\endgroup$ – Nick Cox Dec 9 '14 at 12:40
  • $\begingroup$ I changed from 'parameter' to 'outcome variable' in the question. $\endgroup$ – rnso Dec 9 '14 at 17:35
8
$\begingroup$

The approach you are using is devoid of input from subject-matter knowledge. This is usually a recipe for trouble. Using $P$-values to guide model specification is fraught with statistical problems. And you may be using the wrong statistical tests as the basis for your concern, i.e., you may be trying to define a main effect in the presence of an interaction effect. The appropriate tests to use are combined tests of main effects + interaction effects. But more importantly having 7th-order interactions in your model is a huge overkill resulting in estimation of far too many parameters, making everything unstable. It would be better to stick to second-order interactions (in R (a + b + c)^2) but this approach is still a bit dangerous and you have not included all the original variables as main effects, which will give you a false impression of importance of interactions.

Try to think of a statistical analysis not as a fishing expedition but rather through reasoned model specification.

$\endgroup$
  • 2
    $\begingroup$ I would add that this kind of high-order interactions would be impossible to interpret and would have a high potential for overfitting. $\endgroup$ – Tim Dec 9 '14 at 12:42
2
$\begingroup$

If I understand you correctly, you have a number of explanatory variables but not much clue which of them could be relevant and which not. Here are a few ways to proceed:

  1. Try out all possible subsets of variables and pick the one that gives a regression with the smallest Bayesian information criterion (BIC) value. See e.g here for relevant R functions. In you want to allow for interactions, too, then either define new regressors by multiplying the existing ones in a pairwise fashion or look for an existing function to do that for you.
    Using BIC will help strike the right balance between possible overfitting and underfitting. If you intend to use your model for forecasting rather than explanation, use Akaike information criterion (AIC) instead of BIC.

  2. Forward or backward stepwise selection: start from a small model and add regressors one by one based on their relevance (broadly speaking) or start from a general model and remove regressors one by one, again based on their relevance.

  3. Shrinkage methods (LASSO, ridge regression, elastic net, principal components regression, partial least squares): if you want to reduce the mean squared error of your model and do not care exclusively about the unbiasedness of your estimates, you might want to allow for some bias to gain a decrease in variance. This makes sense if you intend to forecast but not so much if your study is explanatory.

See Hastie et. al "The Elements of Statistical Learning" chapter 3 subsections 3.3-3.7 for a more detailed overview.

To address your concern about p-values: they are not sacred and many say they are given too much importance. Sometimes (when there is a lot of data) even irrelevant variables become statistically significant, but the magnitudes of their coefficients are small and substantively negligible. Sometimes two or more variables are jointly significant but not so individually. Thus you have to interpret them carefully and not just mechanically.

$\endgroup$
  • 1
    $\begingroup$ Note that some of these are more controversial than others. #2 is widely considered a very bad idea, for reasons well explained by Frank Harrell, a contributor here, in his book Regression modeling strategies (New York: Springer, 2001). I'd want to add 0. Consist subject-matter experts if you lack subject-matter expertise to consider what kinds of relationships are likely to be of interest or importance. $\endgroup$ – Nick Cox Dec 9 '14 at 17:43
  • 1
    $\begingroup$ #1 is also a very bad idea. $\endgroup$ – Frank Harrell Dec 9 '14 at 21:05
  • $\begingroup$ I tried and found bestglm very easy to use. Simple command "bestglm(mydata)" uses default options and gives me the best model. Is it fair to use in most situations? $\endgroup$ – rnso Dec 10 '14 at 11:41
  • 1
    $\begingroup$ No. It is important to fully understand the assumptions made and the performance characteristics of any method for model development. $\endgroup$ – Frank Harrell Dec 10 '14 at 11:52
  • $\begingroup$ I checked and found that in most studies interactions are not used at all. Only different independent variables are put in multiple regression using '+' (and not '*') and significant ones are reported. Is this safe to follow when interactions are not obvious? $\endgroup$ – rnso Dec 10 '14 at 12:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.