3
$\begingroup$

For my undergraduate dissertation I conducted a survey. The questionnaire applied starts with a prime, which is followed by a product that has in one scenario a hig degree of innovativeness and in the other a rather low. This is then followed by some scales.

When I started working with the data, two things became salient. The data is not normally distributed (which I thought to be neglegible with a sample-size of N = 180). Further, the manipulation check for the prime was just barely significant with p = .045 and d = 0.26. (the manipulation check for the manipulation of the products degree of innovativeness seemed to work out well with p < .001 and d = 1.40).

Following this, I conducted t-tests which in none of the cases led to a significant difference in the sample means. Applying the u-test didn't change anything either. But when I looked at the correlations, I could find, for all my hypotheses, significant correlations with mid-level or strong effect-sizes.

So far I could rule out that the non normal-distributed data could be the reason. But arguing for the weak prime being the reason doesn't make sense either, as otherwise in case of the innovation-manipulation the t-tests should have led at least to so some effects.

Can anyone help finding an explanation for the case stated above.


My apologies for being to unspecific @Martijn Weterings. I am afraid, but I can't post my experiment visually, as it is still in progress and has not yet been handed in for my graduation. Maybe I can try to explain the experiment and variables again.

Experiment:

The experiment starts with a visual prime of which two variations exist (happy vs. unhappy). It is followed by a product, of which also two variations exist (high degree of innovativeness vs. low degree of innovativeness). After this the proband has to complete some scales.

Sample:

The sample contains of 180 completed questionnaires.

Scales used:

-Degree of Happiness (later referred to as DoH)

-Perceived Degree of Innovation (later referred to as DoI)

-Perceived Uncertainty (later referred to as PU)

-Willingness to Buy (later referred to as WtB)

-Personal Relevance (later referred to as PR)

After having conducted a factor analysis as well as a reliability analysis I aggregated each of the scales above into a single variable (DoH, DoI, WtB and PR). Additionally I used two dummyvariables - one for the prime (lets call it PRIME) with 0 = unhappy, 1 = happy, and one for the products degree of innovativeness (lets call it INNOV with 0 = low innovativeness and 1 = highly innovative).

Manipulation Checks:

Then, I conducted two manpulation checks (i.e. t-tests). One for the prime by using the PRIME as the independent variable and DoH as the dependent.

And an additional manipulation check for checking, if the pobands perceived the product I manipulated indeed as low or highly-innovative in the respective cases. Here I used INNOV as the independent variable and DoI as the dependent.

The first manipulation check provided a difference in the sample means of 0.30 (i.e. 5.02 vs 5.32) with p = .045 and d = 0.26. The second manipulation check had a difference of 1.78 (i.e. 4.35 vs 6.13) with p < .001 and d = 1.40.

Hypotheses 1 a,b

My first hypotheses were that, holding the prime constant, people who received a highly innovative product will (a) perceive a higher degree of uncertainty and (b) perceive a lower willingness to buy the product in comparison to those who receive the low-innovative product.

Results:

absolutely none

Hypotheses 2a,b,c

People who received the happy prime (PRIME = 1), holding the degree of innovativeness constant, will (a) perceive less uncertainty in their buying decision, (b) have a higher willingness to buy the product and (c) experience the product to be of more personal relevance.

Results:

absolutely none.

Post Hoc:

But when looking at the correlations between DoI, DoH and those for the respective dependent variables PU, WtB, PR they show ALL (significant) correlations of mid-level to high effect-sizes of which all but one are in line with my hypotheses.

My attempt to explan was:

As my variables were not normally distributed, I checked my hypotheses with u-tests instead of t-tests. They also provided no significant differences in the respective sample means. Then I though, that maybe in case of the t-Tests using PRIME, the reason may be that the prime was just slightly significant and therefore the means of the respective groups didn't differ. But then I don't understand, why the t-tests using INNOV provide no results at all, as the manipulation check was highly significant with a Cohens d of 1.40!

$\endgroup$
  • $\begingroup$ Correlation and difference of means are two different things. Two variables with the same mean can correlate, significantly, in all sorts of ways. $\endgroup$ – Sextus Empiricus May 2 '18 at 12:06
  • 1
    $\begingroup$ You may get better answers when you explain your data a bit more and when you explain the t-test and tests for correlations that you did. They are a bit ambiguous. You probably did a correlation between two (groups) of your dependent variables instead of between one of your dependent variables and one of your independent variables. $\endgroup$ – Sextus Empiricus May 2 '18 at 13:23
  • $\begingroup$ The independent variables were scales for (a) perceived uncertainty (b) willingness to buy and (c) personal relevance. For hypotheses 1, I used for the t-tests the dummy-variable for the prime as dependent variable. In case of hypotheses 2, I used the, also dummy-coded, variable for the innovation-manipulation. When using the correlations, I did so by looking at the correlations of the respective scales (i.e. all the independent-variable-scales mentioned above and for the dependent variables instead of the dummy-variables I also took the scales). $\endgroup$ – KSaki May 2 '18 at 13:49
  • $\begingroup$ indicate your data; goals and key hypothesis. What is the bssid and structure of problem.Be brief and remove irrelevant issue's that you state in the back ground of your problem. $\endgroup$ – Subhash C. Davar May 2 '18 at 15:26
  • $\begingroup$ @KSaki as davar says write your comment out more clearly in your question (Start with 1. description of all variables, then 2. describe the test. Currently you suddenly introduce new, unclear, variables in the description of the tests). Could you also please add a graphical representation of what you are doing (edit your question) by using just two of the variables on which you perform both a t-test and correlation (a single correlation should always be representable as a scatterplot). $\endgroup$ – Sextus Empiricus May 2 '18 at 19:58
1
$\begingroup$

Your new information (which you should have better incorporated into your question rather than making it an answer) makes things a bit more clear and I believe I can provide an answer which I hope relates to your case.

It would be nice if you could verify whether this is the core of the question and then clean up your question&answer combination.

What I understand is the following:

Data:

  • You have 2 independent dummy variables PRIME and INNOV

  • You have 5 dependent numerical variables (scores from a questionnaire) DoH, DoI, PU, WtB, PR

Tests:

  • You do a t-test

    Note that this is still unclear how you do this exactly. You suddenly mention non-parametric testing as well (which has much less power and is a different thing). You also you mention hypothesis 1 and 2 with 'none results', but that is unclear. Where are these t-tests you mention in your question? How did you do them? What did you do to test the hypothesis?

    I currently assume that you did a t-test by splitting a variable in two groups according to the dummy variable.

  • You perform a correlation between

    • Case 1, between: DoH or DoI and the other three PU, WtB and PR
    • But not Case 2, between: PRIME and INNOV and the other three PU, WtB and PR

Case 1:

p-values t-test and correlation are not equal

Since you do the grouping for t-tests based on on variable (PRIME or INNOV) but for the correlation you use another (DoH or DoI) you may get non-significant t-test but significant correlation (while manipulation check would be positive, ie DoH correlates with PRIME) in the following way:

example for significant correlation while no significant t-test

Note that the following three are true:

  • DoH correlates with PRIME (your significant manipulation check)
  • PU does not correlate (or t-test) with PRIME (your t-test)
  • but still DoH and PU are significantly correlated.

Case 2:

p-values t-test and correlation are equal

Note that PU and PRIME are not significantly correlated. P-value for correlation (Point-biserial correlation coefficient) and p-value for t-test are here the same thing (mathematically speaking).

correlating with PRIME

$\endgroup$
  • $\begingroup$ First of all thank you very much for the detailed answer. I have to admit, Iam getting more and more confused. I can arrange my explanations a bit, so that things may become a little more clear. But maybe I first explain the thought process behind looking at the scales correlations. I did that to gen an insight, if e.g. people who rate the scales for perceived degree of innovativeness with high assessments are also likely to give high assessments for the scales of perceived unvertainty etc. Is that unusual to do and does that have no explanatory power? $\endgroup$ – KSaki May 3 '18 at 18:14
0
$\begingroup$

Correlations are not measuring the same thing as T-tests and their p-values are not telling you the same story.

TTest

A Ttest measures within group variance vs between group variance.

A p value from a T-test is evaluating the probability that the means of two groups are different, to be precise that if you were to repeat the experiment with a random sampling of the same population using the same methodology the probability that you observing a difference in the means equal or greater than that observed when in fact the underlying difference is due to chance.

Correlation

A correlation measures shared variance vs total variance.

A correlation is evaluating the probability that variation of two sets of values are associated with each other. The p value here is the probability in a repeated experiment of seeing an association as stronger or stronger if the underlying trend was in fact due to random chance.

Correlations can be statistically significant but practically useless, for example a sample size of 600 is enough to get an R2 of 0.01 to be significant but each variable only describes 1% of the variance in the other.

$\endgroup$
  • $\begingroup$ no problem, if it was helpful please upvote otherwise please let me know what is still needed. $\endgroup$ – ReneBt May 2 '18 at 12:02
  • $\begingroup$ Sure. But is there a typical explanation for this finding? I would like to bring it up as part of an post-hoc analysis. So far I could show the correlations and by doing so justify some further research. But I should at least give some indication for why my tests deliver no results, but all correlations do. Besides that - although the correlations' p-values being of less meaning, don't at least the r-values possess some explanatory power? As I stated above, their effect-sizes range all between mid-level and high. $\endgroup$ – KSaki May 2 '18 at 12:10
  • $\begingroup$ BTW, can't upvote before having 15 reputations, but will put you in my list :) $\endgroup$ – KSaki May 2 '18 at 12:11
  • $\begingroup$ If you correlate a dummy variable with some numerical variable (en.wikipedia.org/wiki/Point-biserial_correlation_coefficient) then I guess you could say that the p-value for a non-zero correlation is actually the same as the p-value for the t-test. (in which case correlation measures the same thing as t-test) $\endgroup$ – Sextus Empiricus May 2 '18 at 20:17
  • $\begingroup$ I just did that. In this case the p-values are indeed the same as those of the non-significant t-tests. $\endgroup$ – KSaki May 2 '18 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.