For my undergraduate dissertation I conducted a survey. The questionnaire applied starts with a prime, which is followed by a product that has in one scenario a hig degree of innovativeness and in the other a rather low. This is then followed by some scales.
When I started working with the data, two things became salient. The data is not normally distributed (which I thought to be neglegible with a sample-size of N = 180). Further, the manipulation check for the prime was just barely significant with p = .045 and d = 0.26. (the manipulation check for the manipulation of the products degree of innovativeness seemed to work out well with p < .001 and d = 1.40).
Following this, I conducted t-tests which in none of the cases led to a significant difference in the sample means. Applying the u-test didn't change anything either. But when I looked at the correlations, I could find, for all my hypotheses, significant correlations with mid-level or strong effect-sizes.
So far I could rule out that the non normal-distributed data could be the reason. But arguing for the weak prime being the reason doesn't make sense either, as otherwise in case of the innovation-manipulation the t-tests should have led at least to so some effects.
Can anyone help finding an explanation for the case stated above.
My apologies for being to unspecific @Martijn Weterings. I am afraid, but I can't post my experiment visually, as it is still in progress and has not yet been handed in for my graduation. Maybe I can try to explain the experiment and variables again.
The experiment starts with a visual prime of which two variations exist (happy vs. unhappy). It is followed by a product, of which also two variations exist (high degree of innovativeness vs. low degree of innovativeness). After this the proband has to complete some scales.
The sample contains of 180 completed questionnaires.
-Degree of Happiness (later referred to as DoH)
-Perceived Degree of Innovation (later referred to as DoI)
-Perceived Uncertainty (later referred to as PU)
-Willingness to Buy (later referred to as WtB)
-Personal Relevance (later referred to as PR)
After having conducted a factor analysis as well as a reliability analysis I aggregated each of the scales above into a single variable (DoH, DoI, WtB and PR). Additionally I used two dummyvariables - one for the prime (lets call it PRIME) with 0 = unhappy, 1 = happy, and one for the products degree of innovativeness (lets call it INNOV with 0 = low innovativeness and 1 = highly innovative).
Then, I conducted two manpulation checks (i.e. t-tests). One for the prime by using the PRIME as the independent variable and DoH as the dependent.
And an additional manipulation check for checking, if the pobands perceived the product I manipulated indeed as low or highly-innovative in the respective cases. Here I used INNOV as the independent variable and DoI as the dependent.
The first manipulation check provided a difference in the sample means of 0.30 (i.e. 5.02 vs 5.32) with p = .045 and d = 0.26. The second manipulation check had a difference of 1.78 (i.e. 4.35 vs 6.13) with p < .001 and d = 1.40.
Hypotheses 1 a,b
My first hypotheses were that, holding the prime constant, people who received a highly innovative product will (a) perceive a higher degree of uncertainty and (b) perceive a lower willingness to buy the product in comparison to those who receive the low-innovative product.
People who received the happy prime (PRIME = 1), holding the degree of innovativeness constant, will (a) perceive less uncertainty in their buying decision, (b) have a higher willingness to buy the product and (c) experience the product to be of more personal relevance.
But when looking at the correlations between DoI, DoH and those for the respective dependent variables PU, WtB, PR they show ALL (significant) correlations of mid-level to high effect-sizes of which all but one are in line with my hypotheses.
My attempt to explan was:
As my variables were not normally distributed, I checked my hypotheses with u-tests instead of t-tests. They also provided no significant differences in the respective sample means. Then I though, that maybe in case of the t-Tests using PRIME, the reason may be that the prime was just slightly significant and therefore the means of the respective groups didn't differ. But then I don't understand, why the t-tests using INNOV provide no results at all, as the manipulation check was highly significant with a Cohens d of 1.40!