no effect for t-tests but for correlations - in search for a reason

Question

For my undergraduate dissertation I conducted a survey. The questionnaire applied starts with a prime, which is followed by a product that has in one scenario a hig degree of innovativeness and in the other a rather low. This is then followed by some scales.

When I started working with the data, two things became salient. The data is not normally distributed (which I thought to be neglegible with a sample-size of N = 180). Further, the manipulation check for the prime was just barely significant with p = .045 and d = 0.26. (the manipulation check for the manipulation of the products degree of innovativeness seemed to work out well with p < .001 and d = 1.40).

Following this, I conducted t-tests which in none of the cases led to a significant difference in the sample means. Applying the u-test didn't change anything either. But when I looked at the correlations, I could find, for all my hypotheses, significant correlations with mid-level or strong effect-sizes.

So far I could rule out that the non normal-distributed data could be the reason. But arguing for the weak prime being the reason doesn't make sense either, as otherwise in case of the innovation-manipulation the t-tests should have led at least to so some effects.

Can anyone help finding an explanation for the case stated above.

---

My apologies for being to unspecific @Martijn Weterings. I am afraid, but I can't post my experiment visually, as it is still in progress and has not yet been handed in for my graduation. Maybe I can try to explain the experiment and variables again.

Experiment:

The experiment starts with a visual prime of which two variations exist (happy vs. unhappy). It is followed by a product, of which also two variations exist (high degree of innovativeness vs. low degree of innovativeness). After this the proband has to complete some scales.

Sample:

The sample contains of 180 completed questionnaires.

Scales used:

-Degree of Happiness (later referred to as DoH)

-Perceived Degree of Innovation (later referred to as DoI)

-Perceived Uncertainty (later referred to as PU)

-Willingness to Buy (later referred to as WtB)

-Personal Relevance (later referred to as PR)

After having conducted a factor analysis as well as a reliability analysis I aggregated each of the scales above into a single variable (DoH, DoI, WtB and PR). Additionally I used two dummyvariables - one for the prime (lets call it PRIME) with 0 = unhappy, 1 = happy, and one for the products degree of innovativeness (lets call it INNOV with 0 = low innovativeness and 1 = highly innovative).

Manipulation Checks:

Then, I conducted two manpulation checks (i.e. t-tests). One for the prime by using the PRIME as the independent variable and DoH as the dependent.

And an additional manipulation check for checking, if the pobands perceived the product I manipulated indeed as low or highly-innovative in the respective cases. Here I used INNOV as the independent variable and DoI as the dependent.

The first manipulation check provided a difference in the sample means of 0.30 (i.e. 5.02 vs 5.32) with p = .045 and d = 0.26. The second manipulation check had a difference of 1.78 (i.e. 4.35 vs 6.13) with p < .001 and d = 1.40.

Hypotheses 1 a,b

My first hypotheses were that, holding the prime constant, people who received a highly innovative product will (a) perceive a higher degree of uncertainty and (b) perceive a lower willingness to buy the product in comparison to those who receive the low-innovative product.

Results:

absolutely none

Hypotheses 2a,b,c

People who received the happy prime (PRIME = 1), holding the degree of innovativeness constant, will (a) perceive less uncertainty in their buying decision, (b) have a higher willingness to buy the product and (c) experience the product to be of more personal relevance.

Results:

absolutely none.

Post Hoc:

But when looking at the correlations between DoI, DoH and those for the respective dependent variables PU, WtB, PR they show ALL (significant) correlations of mid-level to high effect-sizes of which all but one are in line with my hypotheses.

My attempt to explan was:

As my variables were not normally distributed, I checked my hypotheses with u-tests instead of t-tests. They also provided no significant differences in the respective sample means. Then I though, that maybe in case of the t-Tests using PRIME, the reason may be that the prime was just slightly significant and therefore the means of the respective groups didn't differ. But then I don't understand, why the t-tests using INNOV provide no results at all, as the manipulation check was highly significant with a Cohens d of 1.40!

Answer 1

Your new information (which you should have better incorporated into your question rather than making it an answer) makes things a bit more clear and I believe I can provide an answer which I hope relates to your case.

It would be nice if you could verify whether this is the core of the question and then clean up your question&answer combination.

What I understand is the following:

Data:

• You have 2 independent dummy variables PRIME and INNOV

• You have 5 dependent numerical variables (scores from a questionnaire) DoH, DoI, PU, WtB, PR

Tests:

• You do a t-test

  Note that this is still unclear how you do this exactly. You suddenly mention non-parametric testing as well (which has much less power and is a different thing). You also you mention hypothesis 1 and 2 with 'none results', but that is unclear. Where are these t-tests you mention in your question? How did you do them? What did you do to test the hypothesis?

  I currently assume that you did a t-test by splitting a variable in two groups according to the dummy variable.

• You perform a correlation between

  • Case 1, between: DoH or DoI and the other three PU, WtB and PR
  • But not Case 2, between: PRIME and INNOV and the other three PU, WtB and PR

---

Case 1:

p-values t-test and correlation are not equal

Since you do the grouping for t-tests based on on variable (PRIME or INNOV) but for the correlation you use another (DoH or DoI) you may get non-significant t-test but significant correlation (while manipulation check would be positive, ie DoH correlates with PRIME) in the following way:

Note that the following three are true:

• DoH correlates with PRIME (your significant manipulation check)
• PU does not correlate (or t-test) with PRIME (your t-test)
• but still DoH and PU are significantly correlated.

Case 2:

p-values t-test and correlation are equal

Note that PU and PRIME are not significantly correlated. P-value for correlation (Point-biserial correlation coefficient) and p-value for t-test are here the same thing (mathematically speaking).

Answer 2

Correlations are not measuring the same thing as T-tests and their p-values are not telling you the same story.

TTest

A Ttest measures within group variance vs between group variance.

A p value from a T-test is evaluating the probability that the means of two groups are different, to be precise that if you were to repeat the experiment with a random sampling of the same population using the same methodology the probability that you observing a difference in the means equal or greater than that observed when in fact the underlying difference is due to chance.

Correlation

A correlation measures shared variance vs total variance.

A correlation is evaluating the probability that variation of two sets of values are associated with each other. The p value here is the probability in a repeated experiment of seeing an association as stronger or stronger if the underlying trend was in fact due to random chance.

Correlations can be statistically significant but practically useless, for example a sample size of 600 is enough to get an R2 of 0.01 to be significant but each variable only describes 1% of the variance in the other.