2
$\begingroup$

I have two sets of data and need to compare mean times for a variety of situations (i.e. time of day, day of the week, different areas, etc...). Some of the situations have small samples sizes (n=8) and need to be compared against a larger sample size (n=260). Neither data set is normally distributed (both are usually positively skewed). The larger data set has a standard deviation that is approximately 3 times the size of the mean. The p-value for the T-test is 0.000003. The p-value for the Wilcoxon is 0.61. It seems to me that given the nature of the data Wilcoxon may be my best bet, but the p-value for the T-test seems more believable.

Am I approaching this correctly?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
3
$\begingroup$

Even with the skewness, it seems rather surprising that the p-value for the t-test is quite so low when that for the Wilcoxon is high.

It looks like your response variable is time. Have you considered comparing speed (or 'rate') instead of time? This will be proportional to the inverse of time, but if there's some natural rate variable that would be proportional to it you should probably use that.

Another reasonably common thing to do is to model the log of time instead of time. (It will make no difference at all to the Wilcoxon - if you've done it right, the answers will be the same on time, speed or log(time).)

[How do 1/time and log(time) behave? Are they skewed?]

--

Now to an even more important issue:

If you are comparing over many different independent variables, the fact that some may have an effect will impact your inference for all of the other variables.

You're in a situation where univariate tests (either t or Wilcoxon) don't seem to make a lot of sense.

You should be doing your comparisons adjusted for the other important variables. The usual way to do that is to fit some model that accounts for all of those variables, so that the means can be compared with the effects of other variables accounted for.

If you're going to do that, you might consider a GLM rather than an ordinary linear model - this would allow you to model the time directly, perhaps as an exponential or gamma random variable (which are right skewed).

Improve this answer
$\endgroup$
1
  • $\begingroup$ Glen_b raises a good point that I did not consider. The true analysis may require treating the data as distinctly multivariate. Although, for poking around in a preliminary way, univariate testing can still give some guidance if you keep in mind that such testing lowers the power to detect weak effects. $\endgroup$
    – Doctorambient
    Commented Mar 13, 2013 at 1:36
1
$\begingroup$

Assuming that your data support the analyses that you've started...

The t-test really does depend on the normality assumption; when that fails the p-values it gives may be in error. This failure is often in the direction of a smaller p-value that is really supported by the data. So I would be suspicious of the tiny p you got from the t-test, especially given the much larger Wilcoxon number (I will assume that you set that up correctly.) I've seen the situation you describe in my own consulting work. What you must consider for yourself is how severe is the skew? If it is a lot, the t must be abandoned in this case I suspect.

Failing normality, the t-test works well for data that is unimodal (your description suggests this may be true) and symmetric about its center. Clearly this last fails as you said that the data you have is skewed. If the skew is a lot, as I said, the t-test is not really justified.

Also, as an aside, given your vastly different sample sizes, the classical Student t-test should be replaced with the SWS t-test (called Welch's t-test, Welch–Satterthwaite test, etc.) This test uses a different degrees of freedom calculation to deal with unequal sample sizes and (possibly) unequal variances. There are statisticians who believe that it should always be used in place of the classical Student t. (I'm one of them; sorry I don't have the reference handy!)

From your description the Wilcoxon test is a good option, but as it is fully non-parametric, you may be losing a lot of power to detect differences that are actually significant. Another option would be to go the resampling/bootstrapping route. Or to use approximate permutation testing, assuming the groups are independent (you did not say, but the Wilcoxon also assumes this). But you will have to do some research to figure out how to implement these tests in your software. If you want to follow this path, and you are using R, a good start is at this post. I think it might help you.

However, if the data is truly skewed, and you must pick Wilcoxon or t, I think you have to live with the Wilcoxon.

Improve this answer
$\endgroup$
0
$\begingroup$

The t-test is obviously a risky choice.

However, in your example, the Wilcoxon also has problems. The Wilcoxon test assumes that the shapes of the distribution in both groups are the same, but there is a location shift. This is clearly not the case in your example.

If you can transform the data so the shapes of the distributions in both groups are similar, you can use Wilcoxon.

If this can't be done, I'd recommend looking at a randomisation test in this example.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.