3
$\begingroup$

I have a set of data containing 4 predictors (environmental conditions and animal size) and one predicted variable (animal growth rate). I want to fit a regression model to this data. I have two objectives:

  1. Test the importance and interactions between predictors. I am most interested in interactions between size and other predictors, as I have some clue that animal should change its reactions to environmental factors as it grows. But other interactions may be interesting as well and I am not sure if I can (or should) just drop some interactions from the model. I want to test it using t-test for coefficients (calculated automatically in R with the model fit).

  2. Select the best model for future predictions. To do this I want to use AIC or BIC criterion to all possible submodels, as the number of combinations is not so big.

The question is:

Should I perform t-test after or before model selection? The two mentioned objectives are more or less independent as the first one is just theoretical insight and the second is more practical. This suggests that I should perform t-test on full model and model selection afterwards. But I am not sure.

$\endgroup$
1
$\begingroup$

Your 1st objective has to do with insights/interpretation and the 2nd one has to do with predictability/accuracy. Usually those two are not going together, especially when you have many variables. In practice you have to create simple models for interpretation (might be general and simplistic though) and more complex models for predictions.

In your case I would suggest to perform various simple regression models first. 1) one predictor variable at a time will show you how important they are (individually) and if they have a positive or negative effect. 2) Two predictors plus interaction at a time will enable you to investigate interactions of interest.

Then you use all your variables (plus interactions; all or selected ones) and focus on prediction.

I'd encourage you to try some regression trees as well and compare findings. Tree models are great in spotting interactions (automatically). They won't give you coefficients to interpret, but they will give you a kind of variable grouping in terms of predicting your dependent variable.

$\endgroup$
  • $\begingroup$ Thank you for the answer! However, the idea of checking 1-2 variable models is counterintuitive for me. For example: I expect the food availability to be the main predictor of the animal growth rate, but population density may have a slight effect too. If I check the influence of density alone, I have to add the variability caused by different food conditions to the random noise, which would probably hide any effect of density. But this "noise" was added on purpose in experimental design and if I had used constant food conditions, then the effect of density might have been visible. $\endgroup$ – Marta Cz-C Aug 27 '15 at 16:25
  • $\begingroup$ Agree about the one variable models, but when I say 2 variables I mean checking main effects and then interaction. So, something like y ~ x1 + x2 and then y ~ x1 + x2 + x1*x2. Of course the variables you pick must also make sense to you. You can interpret coefficients using p values as you mention above. A regression tree might show you a specific range/group of food condition where the density has bigger influence vs. a group of food condition that the density has no effect. $\endgroup$ – AntoniosK Aug 27 '15 at 17:02
1
$\begingroup$

Testing variable significance using the $t$-test can be performed and interpreted the regular way if the model specification is given beforehand. If you have specified your model (or someone else has done it for you) before you even looked at the data, the $t$-statistics will have the distribution they are supposed to have, and significance testing will be fine.

Meanwhile, if the model specification is determined from the data first and significance testing is done subsequently, this will change the way $t$-statistics should be intrepreted. If you first look at the data and select the regressors based on their sample values, the significance tests to be performed later could be intrepreted the regular way but conditional on the model you have selected. However, you are normally not interested in variable significance conditional on the model but rather in unconditional variable significance. Thus in this case you are in trouble.

In practice, it may be difficult to select a model before you even look at the data; that also implies you will often not be able to really trust the $t$-statistics (unless you honestly selected the model first and only then estimated it using the data).

Regarding forecasting, it has little to do with significance testing and can be treated as a separate question. In a sense, you can do it independently from significance testing. Using AIC on all possible submodels may be OK if the number of possible submodels is not too large. Combining forecasts from different models could also be an option, see e.g. Burnham & Anderson "Model Selection and Multimodel Inference".

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.