I am trying to perform a visual analysis of significance on these stats. Other information provided is the Standardized motor skills test score [M = 100, SD = 15], not sure if this is relevant. What I can see is that the difference between mean scores is 0.18 which seems such a small number considering the standard is M = 100. So my guess would be that this is not significant however it is just a gut feeling, is there anything else here that I am missing. I specifically need to give an anticipated answer based on visual inspection.

  • $\begingroup$ Please read our tag wiki and change your question to suit - what are your thoughts on it? What do you understand? Where do your specific difficulties occur? $\endgroup$ – Glen_b -Reinstate Monica Apr 19 '15 at 5:14
  • $\begingroup$ Why does your title say "t-test", your tags say "z-test" yet your body text actually ask for neither (but rather, a visual assessment)? Could the three be made more consistent? $\endgroup$ – Glen_b -Reinstate Monica Apr 19 '15 at 5:30

Some initial discussion to hopefully orient your thinking:

Your sense that "0.18 is small" is that there's probably not much that's of practical importance in that size of difference.

However, statistical significance and practical importance are quite different. In a rough sense, statistical significance is "Is the sample inconsistent with the population described in the hull hypothesis?".

In small samples, important differences may not be statistically significant (there's so much uncertainty in our estimates that we can't tell even big differences aren't just caused by random noise). In very large samples, quite trivial differences may well be statistically significant.

An appropriate sense of statistical significance can be obtained by seeing if the difference in sample means is much more than some estimate of the standard error of the difference in means, at least for statistics that have close-to-normal sampling distributions, as here.

In this case rough lower and upper bounds on the standard error will let you put corresponding upper or lower bounds on the z-score, allowing you to say either "not significant" or "significant" (respectively). Equivalently, you're trying to get a sense of how many standard-deviations-of-the-difference-in-means it is from zero.

Often you only need a rough idea to make a decision. Sometimes you need to be a little more sophisticated.

As a rough first approximation, the standard error of the difference will be bigger than the larger of the two standard errors, and smaller than the sum of them.

You mention visual assessment: one common visual display is to plot a mean $\pm$ a standard error for each group (or some multiple of it). Since the standard error of the difference is smaller than the sum of the standard errors, if the two mean$\pm$se's overlap, they could be less than 1 standard error apart; and if two mean$\pm$2se's don't overlap they must be at least two standard errors apart. If the sample size isn't small ($n_1+n_2>7$), that would suggest a significant difference. However, you can do that from the table without drawing a display.

enter image description here

Since the two means are each less than the larger of the two standard errors apart, they must be less than one standard error of the difference apart; visually, this shows as a big overlap in those red intervals.

  • $\begingroup$ Apologies, the course topic is t-test but this particular section is a visual assessment only. Yes it is a course related problem, apologies again, this is my first post and I was unaware of all of the rules. Thank you for your very eloquent response. $\endgroup$ – Tracy Apr 20 '15 at 4:56
  • $\begingroup$ Added a display. $\endgroup$ – Glen_b -Reinstate Monica Apr 25 '15 at 3:18

Cohen's d parameter is used to assess the effect size for 2 group comparisons. Simply put, it is mean difference / pooled standard deviation (http://en.wikipedia.org/wiki/Effect_size#Cohen.27s_d). Cohen's d values of 0.2, 0.5, and 0.8 represent small, medium, and large effect sizes respectively (http://www.statmethods.net/stats/power.html). Considering this you can see that for your data Cohen's d will be very small (it is 0.013) and hence the effect size is very small.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.