Non-significance and model specification

I have a rather complicated model that I am testing on a panel data set, containing 3 categorical variables and 3 continuous ones, where I want to specifically test for the interaction effects between the categorical and continuous variables. So far, I have used, in Stata, fixed effect and random effect regression, both with and without the -robust- option. Unfortunately, I have been getting no or very few significant coefficients as results. The question I have is whether tinkering with the regression type, e.g. using -xtmixed- or other regression models, will

1. has a good chance of achieving significant results. More generally, does using different, I guess more "complex", models, often result in a change of significance of effects?
2. is there a decent statistical justification for this? My knowledge of the topic is rather less comprehensive, and the changes to the model I have undertaken so far feel more like random tinkering than anything substantial. I'd be happy if by chance significant effects show up, but I feel like I would have a hard time justifying the whats and whys.

Thanks so much for your consideration!

All the best, Daniel

• Significance-hunting in this way is not a suitable way of finding things out and is not statistically justified. – Glen_b Jul 11 '13 at 15:04

Continuing on @Maarten's point:

You should select models not because they give you the results you want, but because they are appropriate to what you want to do and the data you have.

This does not mean you cannot do data exploration (as long as you are honest when you report it, and realize that your results may not generalize and that p values from models that took a lot of tinkering will be too low, etc). But that tinkering should be guided by something other than "there must be something here!"

For example, if fixed effects are appropriate then it is hard to see how random effects could also be appropriate. (Although maybe there is a way).

Do some plotting, in fact, do a lot of plotting.

• Thanks a lot, Peter! Would you have a rough outline/method of what kinds of plotting would be helpful to recognize important information? – Daniel Jul 11 '13 at 14:52
• Plot the dependent variable against each IV using scatter or box plots for continuous and categorical IVs respectively . Look at pairs of IVs using either lattice plots in lattice or faceting in ggplot. For the model, try the effect plots in the effects package. – Peter Flom Jul 11 '13 at 17:06

I realize you do not want to hear this, but hunting around for significant effects is unlikely to help you. In all likelihood your dataset just does not contain enough information to find those interaction effects. No statistical method can "invent" information that is not present in your data, and that is a good (but sometimes frustrating) thing.

• As always, very much to the point. Thanks a lot, Maarten! – Daniel Jul 11 '13 at 14:49

It depends on the multicollinearity of the data if your significant model experiences vast changes in the significance of its parameters.

The random tinkering pretty much depends not only on the new values, but also on the changes.

For example: Does the empirical distribution of your residuals of your regression change a lot? If so, your model is probably not specified correctly.
Or, how do the significance values for the model (for example F-Test) and parameters of your regression change? Is one stable and the other one not? That might be a problem in multicollinerity.

In general model specification is not exact, and including/excluding variables depends on the case. There are also issues related to successively adding and removing arguments to the regression.
In general, however, I'd say that when it comes to picking your regression methodology, so the kind of model you want to use, you should definitly have a reason for doing so.