What to do when the predictors do not accord with common sense/literature, but the model is fine/best according to log likelihood and LRT?

Question

I would try to clarify the problem and then ask the questions.

---

The problem (variable names are masked due to confidentiality):

I ran a binary logistic regression, in which there were 5 independent variables (IVs): A, B, C, D, and E. A and B are my concern. C is also my concern and I would talk about it later. They have two factors each (A1 and A2, B1 and B2). When I ran the estimation, some significant findings appeared. Nevertheless, the direction of the coefficients (beta values) for A and B was the opposite of what I expected! According to the most of the literature (not all of it), I expected A and B to have a positive beta, while they both had a negative coefficient.

First I checked the models for about three full days. There was no mistake in them, and the directions did not change whatever changes I made to the models (except that in none of those changes, I attempted to drop the interactions). The log likelihoods as well showed that I am in a good direction.

Then I decided to put my subjective view against the strange results aside and trust the results of the regression analysis. Then I passed to discuss those strange results and tried to justify the controversial findings. While discussing, I came to this point that "those two variables are heavily interconnected. Firstly, they had a significant interaction. Secondly, the distribution of the predictor A was heavily affected on the B, and according to the literature, A and B could have opposite effects; it could be important in my sample which was not balanced in terms of B. This imbalance could confound the effect of A as well.

So I though maybe these are causing some problems. Then since C was as well strange, I thought maybe the whole model is being affected in a bad way by problems such as multicollinearity. I asked myself "what will happen if I isolate only A and B in the model?" If the interactions between A, B, and C are some sources of bias, can reducing the number of IVs lead to different results? The answer was yes: when I excluded all the other IVs from the model and left only A, B, and A*B, one of the coefficients became favorable and more in line with the literature and common sense. Thus I might tell that some errors do exist in my model which disrupt the main model (such as multicollinearity maybe).

The I decided to examine every strange predictor, in isolation. When I excluded the interactions and left only the five IVs, the results seemed Much more consistent. Apparently, the problem begins when some specific interactions (but not all of them) are added to the model. After adding them, the directions of betas for A and C get reversed. It is a little annoying since by adding those specific interactions, the log likelihood reduces considerably (from about -75 to -48), so I cannot easily ignore those interactions.

---

Questions (the main ones are 3 and 4, but an answer to the rest is as well much appreciated):

1. When the model acts strangely, but LRT and log likelihood tell that it is fine, which one should we choose? The subjective common sense, or the objective statistical measures?

2. Do you think is there a "problem" in my case, in the first place? Maybe everything is fine. If you wished, I can provide the raw data too.

3. What would you do in my case? At least three choices can be made: A. Dropping the interactions. B. Not dropping them and reporting the strange model. C. Reporting both the models with and without interactions, and also models of limited numbers of IVs (for example only A and B), and then try to subjectively discuss that "it is the interactions that cause the main large model strange".

4. I am going to do the latter (3.C). But that would be so messy and not so good looking. I wonder if there is an elegant, objective way of finding the source of error in the main model (well if there is any errors, of course), so instead of subjective discussions, I can substantiate my claims on some objective statistical measures. For example, is there a way to highlight the problematic interactions according to some statistics?

5. Do you have any other valuable suggestions or ideas?

Answer 1

My favorite professor in grad school used to say "If you're not surprised, you haven't learned anything". You are surprised. Now, what can you learn?

One thing that could be causing the problem is collinearity. Have you checked this?

Next, if that isn't it, I would look at graphs. You don't mention these; but, especially with interactions, I find graphs invaluable (actually, graphs are always invaluable). If you are using R, I think the effect plots in the car package are very nice. But also look at a scatterplot matrix of your 5 IVs; look at parallel boxplots of each IV with the DV. Perhaps look at trellis graphs (in R there is the lattice package, for example).

Once you've checked all that out, if your results are still surprising, then things to think about are: 1) How is your population different from other populations? 2) How is your sample different from other samples? Were all of them random samples? Were the samples different in some way? 3) How is your model different from other models? Were the same terms entered? Were they operationalized in the same way? 4) How different were the effect sizes? 5) Where exactly were the effect sizes different? Were the predicted values of your model very different from the other models? If so, at what levels of which IVs were they different?

To me, this is the most fun part of statistical analysis: Something makes you say "Gee, that's funny" and you are off to the races.

Answer 2

In addition to the answer by @PeterFlom, interpreting a single coefficient in a complex model can be difficult at best and is often impossible or at least confusing rather than enlightening. Remember the textbook definition of a single coefficient is the change in your predicted response given a change in the variable of interest while holding everything else constant. But can you hold an interaction term constant while changing the main effect associated with it? And think of increasing a person's height while keeping their weight exactly the same.

Looking at the predictions and plots can show you what is really going on (sometimes a negative main effect is offsetting a large interaction effect and does not actually decrease the prediction). Peter gave some good examples of plots and R tools for creating them. An additional one that I suggest (though I may be biased on this one) is to use the Predict.Plot function in the TeachingDemos package for R. This will show the relationship between one continuous predictor and the predicted response for a given set of values for the other predictors. You can then change one of the other predictors and add that line as well to see how the predictions change and better visualize interactions. The TkPredict function in the same package lets you interactively change the values of the other predictors to see the effect dynamically.

Answer 3

First let me express my huge gratitudes to both of you. You helped me unlock this dilemma, and learn something while enjoying solving the problem efficiently. Both of your comments clarified the multicollinearity problem in my case. Especially when Greg explained it as the height*weight example. I started to construct as many as possible different models, by adding one by one each variable. I don't mean stepwise regression, but different regressions where I manually added new variables.

Then carefully looked when the coefficients start to become strange. At those points I checked the correlation matrix and observed huge multicollinearities between the inputted interaction and both sides of that interaction (sometimes only one variable). Then I removed that interaction from the code and ran the code again, until facing another strange behavior. This way I could pick 5 or 6 interactions being badly correlated to the variables involved. By removing them, I now have a very very shining and desirable model which is just ready to be published!

Most of those interactions are unimportant, but a couple of them should be assessed further. I think i should run a second model in which those interactions are assessed in isolation, just to see if the interaction is significant or not.

So, thanks to you, every thing is now fully consistent.