7
$\begingroup$

When deconstructing my mixed effects model, I found a three-way significant interaction. I calculated my p-value by using maximum likelihood ratio tests allowing for a comparison of the fit of the two models (the model with all predictors minus the model with all predictors but the predictor of interest - in this case, the three-way interaction). When I conduct follow-up comparisons of the three-way interaction, do I need to correct the alpha level of significance with Bonferroni correction?

Thanks for all input!

EDIT: (merged from answer --mbq)

I want to look at the significance of the three-way interaction and then...I wanted to look at any other significant interactions within that first three-way interaction.

I use the same dataset...the model is a crossed random effects of participants and items (the data is comprised of repeated observations (response times) with Valence (positive and negative) and age as a between subjects factor as well as two continuous predictor variables (attachment dimensions). Thus my model is Valence x Age x Attachment anxiety x Attachment avoidance. I found that Valence x Age x Attachment avoidance is significant. However, I want to examine this interaction further. I did this by examining the same model but just for young adults vs old adults separately. Thus, I found with older adults a significant interaction of Valence and Attachment avoidance. However, when I calculated the p-value (as described above) of this two-way interaction, can I take the p-value as is or do I need to correct with Bonferroni? And if so, how? I hope this is clearer?

Thank you!

Basically I want to examine the 'direction' of my three-way interaction and test whether or not the differences within that interaction is significant.

$\endgroup$
2
  • $\begingroup$ What do you mean by 'follow-up comparisons'? Looking at the same three-way interaction in other data sets? $\endgroup$
    – onestop
    Jan 26, 2011 at 19:03
  • $\begingroup$ I have merged your account, but you should rather register it -- this way it won't get lost. You can do it here: stats.stackexchange.com/users/login $\endgroup$
    – user88
    Jan 26, 2011 at 20:59

1 Answer 1

1
$\begingroup$

It sounds like you basically have a problem of model choice. I think this is best treated as a decision problem. You want to act as if the final model you select is the true model, so that you can make conclusions about your data.

So in decision theory, you need to specify a loss function, which says how you are going to rank each model, and a set of alternative models which you are going to decide between. See here and here for a decision theoretical approach to hypothesis testing in inference. And here is one which uses a decision theory approach to choose a model.

It sounds like you want to use the p-value as your loss function (because that's how you want to compare the models). So if this is your criterion, then you pick the model with the smallest p-value.

But the criterion needs to apply to something which the models have in common, an "obvious" choice based on a statistic which measures how well the model fits the data.

One example is the sum of squared errors for predicting a new set of observations which were not included in the model fitting (based on the idea that a "good" model should reproduce the data it is supposed to be describing). So, what you can do is, for each model:

1) randomly split your data into two parts, a "model part" big enough for your model, and a "test" part to check predictions (which particular partition should not matter if the model is a good model). The "model" set is usually larger than the "test" set (at least 10 times larger, depending on how much data you have)

2) Fit the model to the "model data", and then use it to predict the "test" data.

3) Calculate the sum of squared error for prediction in the "test" data.

4) repeat 1-3 as many times as you feel necessary for your data (just in case you did a "bad" or "unlucky" partition), and take the average of the sum of squared error value in step 3).

It does seem as though you have already defined a class of alternative models that you are willing to consider.

Just a side note: Any procedure that you use to select the model, should go into step 1, including "automatic" model selection procedures. This way you properly account for the "multiple comparisons" that the automatic procedure does. Unfortunately, you need to have an alternative (maybe one is "foward selection" one is "forward stepwise" one is "backward selection", etc.). To "keep things fair" you could keep the same set of partitions for all models.

$\endgroup$
5
  • $\begingroup$ Thank you for your comments but I am not sure whether or not I understand. Perhaps the best thing to ask is how to determine the significance of my interactions in my mixed model other than just looking at the t-statistic. $\endgroup$
    – Joanna
    Jan 27, 2011 at 9:35
  • $\begingroup$ @ joanna - apologies for not tying the end back to the significance side of things. Once the procedure is finished, you have 1 model, and will act as if this model is true. Hence any interactions remaining in this model can be declared "significant" compared to those which don't make it into model. Because the procedure is for the whole model, this automatically adjusts for the multiple comparisons you want to do. And you can state that the relationships in this model are the ones which are able to reproduce the data as best as you can out of the alternatives you had. $\endgroup$ Jan 28, 2011 at 3:05
  • $\begingroup$ Thanks again for your input. However, I feel like I'm still not being very clear. I thought the initial model would be held true and thus probing the significant interactions of the fixed effects in the model, I could calculate the significance of the multiple comparisons stemming from that significant interaction. If I understand correctly, is what you suggest something along the lines of the Monte Carlo Markov Chain sampling? I apologize for being so dense...MEM is not my strong suit! $\endgroup$
    – Joanna
    Jan 28, 2011 at 17:39
  • $\begingroup$ @joanna - you are considering different models, therefore you must be uncertain about what the true model is. If you believe the initial model is already true, then by definition, everything in the initial model is significant. I don't think this is what you believe though. I think that the initial model specifies a class of models (i.e. all variables and their interactions), one of which you believe to be true (or at the very least, you are prepared to act as if the true model is in the class you specified)...see next comment for more... $\endgroup$ Jan 29, 2011 at 13:16
  • $\begingroup$ ...cont'd... Thus you are partially uncertain about the true model. The procedure above gives you a coherent way to choose a model from this class (p-values give you a quick way). And yes it is an Monte Carlo algorithm, but it is not a markov chain, because each cycle is independent of the other (more like a bootstrap than MCMC). Note it need not be a sampling based method, you could also do a "jacknife" based evaluation (split data into $G$ groups, and predict the $gth$ group based on the model fit to remaining $G-1$ groups) $\endgroup$ Jan 29, 2011 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.