Non-significance and model specification I have a rather complicated model that I am testing on a panel data set, containing 3 categorical variables and 3 continuous ones, where I want to specifically test for the interaction effects between the categorical and continuous variables. So far, I have used, in Stata, fixed effect and random effect regression, both with and without the -robust- option. Unfortunately, I have been getting no or very few significant coefficients as results. The question I have is whether tinkering with the regression type, e.g. using -xtmixed- or other regression models, will


*

*has a good chance of achieving significant results. More generally, does using different, I guess more "complex", models, often result in a change of significance of effects?

*is there a decent statistical justification for this? My knowledge of the topic is rather less comprehensive, and the changes to the model I have undertaken so far feel more like random tinkering than anything substantial. I'd be happy if by chance significant effects show up, but I feel like I would have a hard time justifying the whats and whys.


Thanks so much for your consideration!
All the best, Daniel
 A: Continuing on @Maarten's point:
You should select models not because they give you the results you want, but because they are appropriate to what you want to do and the data you have.
This does not mean you cannot do data exploration (as long as you are honest when you report it, and realize that your results may not generalize and that p values from models that took a lot of tinkering will be too low, etc). But that tinkering should be guided by something other than "there must be something here!" 
For example, if fixed effects are appropriate then it is hard to see how random effects could also be appropriate. (Although maybe there is a way). 
Do some plotting, in fact, do a lot of plotting. 
A: I realize you do not want to hear this, but hunting around for significant effects is unlikely to help you. In all likelihood your dataset just does not contain enough information to find those interaction effects. No statistical method can "invent" information that is not present in your data, and that is a good (but sometimes frustrating) thing.
A: It depends on the multicollinearity of the data if your significant model experiences vast changes in the significance of its parameters.
The random tinkering pretty much depends not only on the new values, but also on the changes.
For example: Does the empirical distribution of your residuals of your regression  change a lot? If so, your model is probably not specified correctly.
Or, how do the significance values for the model (for example F-Test) and parameters of your regression change? Is one stable and the other one not? That might be a problem in multicollinerity.
In general model specification is not exact, and including/excluding variables depends on the case. There are also issues related to successively adding and removing arguments to the regression.
In general, however, I'd say that when it comes to picking your regression methodology, so the kind of model you want to use, you should definitly have a reason for doing so.
