Does a p-value of 1 for an interaction where the plot has crossing lines signal something is wrong in a 3-way ANOVA?

Question

I have run a 3-way ANOVA and the interaction between two of the factors looks significant when plotted (i.e. the two lines cross). However the p-value is 1.

Should I be worried about this? In particular:

1. Is a p-value of 1 in a 3-way ANOVA weird?
2. If the lines cross in a plot of a 2-way interaction, does this mean that there is definitely an interaction between the two factors or can the lines cross without there being an interaction?

The data comes from a yoked study on two different datasets and in the other dataset, there was a similar looking plot and the interaction was highly significant. If I were to interpret these results as the data is trending in that direction, but there is no significant interaction, would that be a not-completely false conclusion?

What I'm worried about is someone saying either 'oh well the p-value is 1 and that always means x, y, z' or 'oh well the lines might cross, but crossing lines isn't always a sign of an interaction, it can just be a result of x, y, z'. The p-value of 1 feels weird to me, as does the crossing lines without a significant p-value. Should it, or do p-values of 1 occur often and crossing lines without significant p-values occur often?

This is the graph:

Answer 1

I'll begin briefly with the headline news before addressing other parts of your question. There is nothing necessarily wrong with your model. A p value of approximately 1 suggests that there is almost certainly no significant three-way interaction in your model.

That being said, there are a few components of your question worth unpacking. First, to borrow from your example, an interaction model is not testing whether two lines cross. It is testing whether two lines are different. There are several ways in which two lines may not cross but still differ in their slopes. This is all to say that two lines crossing over is not an appropriate criteria to use when evaluating the presence of an interaction. However, if there is an interaction, plotting the effects is useful for interpretation.

The second aspect of your example that deserves consideration is what it is the lines represent. Whether testing a continuous or categorical interaction, the lines plotted invariably represent mean differences or associations or change. While knowing these sorts of average estimates is useful, inferential statistics require that we also think about and model the error associated with our sample-based statistics. Though we may be able to calculate the average differences for subgroups in our analyses for instance, we also have to admit to some degree of uncertainty in those estimates when we want to make inferences about populations. What is missing from your plots in a sense then are confidence intervals. For an example from some of my own work see below:

The inclusion of error bars provides some sense of the degree to which we are uncertain about our estimates. It also lets us know where significant differences do and do not exist (which again is only really useful when there is a significant interaction that requires probing).

Finally, one last aspect of your question worth noting is that three-way interactions are particularly hard to detect and often most analyses that attempt such analyses are wildly under-powered. I would be sure that your analysis has sufficient statistical power before attempting the inclusion of a three-way interaction.

Answer 2

If two lines are not exactly parallel, they will inevitably cross at some point. Without loss of generality let's simplify your question. Say that you were using one sample $t$-test and wanted to test $H_0 : \bar X = 0$. Obviously, you could say that if $\bar X \ne 0$, then the hypothesis is false, but we use hypothesis tests to account for the uncertainty of your results. $p$-values tell you how "likely" would such result be if the $H_0$ was true. So basically what your test tells you is "you could, as well, observe such result if there wouldn't be any interaction".