2
$\begingroup$

My study is about comparing the scores a group of students (sample, N=14) in two different conditions. They play the same game in a tablet and using a computer. They were administered with a questionnaire to evaluate the dependent variable (experience), which is made up of five different components. I found out that the mean value of each component is slightly greater in the tablet game. However, the p-value was greater than the standard alpha level of 0.05 in each case. which leads us to fail to reject the null hypothesis. The null hypothesis, Ho: Difference of the Means is zero. Alternative Hypothesis H1: Difference of the Means is not equal to zero.

How do we interpret this result? Can we say that they had the same experience in both Tablet and Computer environment, since the p-value was not significant. But how do we interpret the fact that the Mean Value was slightly greater for each component in the Tablet vs. Computer Game. For instance, component A: (µTablet:3.64 and µComp: 3.54); component B (µTablet:3.95 and µComp: 3.23); component C (µTablet:3.73 and µComp: 3.71); component D (µTablet:3.80 and µComp: 3.63); component E (µTablet:3.75 and µComp: 3.50).

Is it that N=14 (is not normally distributed), and is too small of a sample?

Kindly advise.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ Regarding "is too small of a sample"... that's what pilot studies and power calculations are for - to help you find a suitable sample size. In some situations, 14 might be plenty big enough to pick up the size of difference that might be interesting to you; in others it might fall well short. One thing that's not clear from your description is whether you considered the potential for order effects (e.g. that people tended to prefer whatever they played first). $\endgroup$
    – Glen_b
    Commented Jan 12, 2015 at 1:38
  • 1
    $\begingroup$ Yes, order effects were taken into consideration $\endgroup$
    – user39531
    Commented Jan 12, 2015 at 14:21

1 Answer 1

Reset to default
2
$\begingroup$

You have found the problem with small sample studies! You do not know whether there is truly no difference in the means, and the differences you observed were just due to random variation, or whether if you had obtained more data you would have gotten a significant p-value.

The interpretation to your study results is simply that the study did not reject the null hypothesis. You have not proven that the conditions are the same, you just did not prove they were different.

In the future, consider your sample size and also the possibility of false positive results. You said you did hypothesis tests on 5 different components. You need far more data to support the amount of hypothesis testing you performed. If you corrected for false positive rate using a standard Bonferroni correction for 5 tests, your p-value would have to be less than 0.01 for you to claim you can reject the null hypothesis.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Can you please clarify about which conditions were not proven to be same or different. Is it that the sample size is too small to arrive at a conclusion? How should I best report the findings, as it is? The dependent variable "experience" is composed of five different components that I individually ran the paired t-tests, and for each I obtained the p-value not to be significant. I am not sure how to proceed in order to correct for :false positive: Do you want me to run a test for significance for the Global Average Mean value of all the 5 components. Please advise. Thanks so much $\endgroup$
    – user39531
    Commented Jan 12, 2015 at 1:17
  • $\begingroup$ Are you suggesting that there are 5 different measurements that make up your dependent variable? If so, you may want to do multivariate paired T-test, called the Multivariate Paired Hotelling's T-Square. See: onlinecourses.science.psu.edu/stat505/node/116. This should also help guard against multiple comparison problems. $\endgroup$
    – StatsStudent
    Commented Jan 12, 2015 at 9:01
  • $\begingroup$ @user39531 You can only report that the study failed to show any significant differences between the interventions (tablet or computer) on any of the 5 component scores. I think that's all you can do now for this study. In terms of whether a larger study would have found a significant difference, now that you have the means and variability for these 5 scores, you could perform a power calculation to see how many subjects would be needed to find a significant difference at alpha=0.01. Keep in mind that there is a difference between "statistical difference" and "meaningful difference." $\endgroup$
    – Bosley
    Commented Jan 12, 2015 at 12:29
  • $\begingroup$ Hello User39531: Can I also add the following: the two platforms were not statistically different, but on the practical significance basis, we can say that users preferred the tablet as the µ was relatively larger in each case. $\endgroup$
    – user39531
    Commented Jan 12, 2015 at 14:18
  • $\begingroup$ Hello StatsStudent: That is new to me, and great resource. Any tips how to conduct the multivariate paired t-test in SPSS. I believe that method minimizes errors..thanks so much $\endgroup$
    – user39531
    Commented Jan 12, 2015 at 14:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.