Why is my manipulation check not correlated with my outcome even though the manipulation produce a significant effect? I have a stats conundrum
I have an MSc thesis in which two groups were manipulated to either include others in their group, or manipulated to exclude others (c1 inclusive and c2 exclusive).
My dependent variable is how much income tax the participant would be willing to pay for the other group, with the prediction from the literature that when primed inclusively, people would pay a higher amount of tax for them (we pay for our own).
A one-way ANOVA did find a significant difference between the groups.
Also, there is a significant difference (in the same direction) between groups in how much they identify with the group they need to pay for (superordinate identity, which is supposed to reflect the manipulation).
At this point I got excited. However, there is no significant correlation between superordinate identity and my DV (how much income tax willing to pay). I’m really not sure what this means? Does this mean my manipulation isn't effective?
I took measures for other things which may be important to the DV. Many of these were strongly significantly correlated with the DV. For example, attitude towards paying taxes (makes sense), attitudes towards the other group (makes sense), how much of a threat the other group is (results didn't make much sense in context). However, there was no significant difference in these measurements between the two groups, showing that the manipulation had no significant impact on these factors.
I attempted a multiple regression to see which predictors best predicted the DV and these other strongly correlated predictors were included in the model, but not my predictor of interest (superordinate identity).
Now I really don’t know what this means. 


*

*How is there only a significant difference between groups on the DV and the scale which reflects the manipulation (superordinate identity) but no significant differences between groups on the other measurements?

*Additionally, why is my superordinate identity scale (reflecting the manipulation)  finding a significant difference between groups, but is not correlated with the DV or included in a MR model to predict the DV? 


I spent a restless night with it going round in my head, over and over, but still cannot understand what has happened. I'm petrified that my whole project might be a waste of time.
 A: Perhaps other will offer a more detailed explanation on what exactly is going on but here are some practical tips/ideas:


*

*Finding a difference on the manipulation check and on the variable of interest is already very encouraging. It is certainly good enough for many accomplished researchers and prestigious publications so don't worry about the project being a waste of time because of that.

*This could be an example of the “ecological fallacy”. When you try to correlate the manipulation check (superordinate identity) and the taxes your participants would be willing to spend, you are looking at individual-level data, which can have another pattern than group-level data.

*Don't obsess over statistical significance or put too much emphasis on the significance threshold. For example, correlation can be high or low, not only present or absent, and it is meaningful to interpret that kind of differences. A related idea is that “the difference between significant and not significant is not itself significant”. Concretely this means that the p-value of the F-test is something like .03 and the correlation is not significantly different from 0 with p = .08, both results are in fact very similar.

*As a consequence of the previous points, you should in any case create plots and interpret them (possibly post them here as well if you wish) rather than pour over lists of p-values.

*The Pearson correlation coefficient reflects linear relationships. It's possible than the relationship between a quantitative measure of the manipulation and the outcome has another shape, explaining the null result, even if the ANOVA suggests the manipulation does produce some sort of effect.

*The absence of a significant difference should not be interpreted as definitive evidence that there was no effect (as you appear to do in several places). If the sample size is small it often means that you just don't know precisely the sign or the magnitude of the correlation, but not necessarily that it is very close to 0.

