I have a time series that is quite erratic, having lots of small increases and then large drops. I want to see if the mean is statistically significantly different to zero.

Using a t-test, I get a p-value of around 0.6, but using a wilcoxon ranksum test I get a p value of 0.0001.

How is this possible? I think the p-value of the t-test makes much more sense as I have only around 500 sample values, most of then +2 and some of them -200 so that the cumulative return of all of them is around 274. It should not be statistically significantly different to zero. Why does the ranksum test say otherwise? Why does it differ so much from the t-test?

last value of cumsum: 

stat, p_stat = ranksums(df['returns'].fillna(0).values, np.zeros(len(df['returns'].values)))
stat, p_stat = ttest_ind(df['returns'].fillna(0).values, np.zeros(len(df['returns'].values)))

The p_stat of the t-test is 0.6, but the p_stat of the ranksums is 2.9e-32. How can that be? For t-test I'm using scipy.stats.ttest_ind and for ranksums I use scipy.stats.ranksums.

  • $\begingroup$ If the standard deviation of the data is 170, the standard error of the mean is $170/\sqrt{500} = 7.6$, which would lead to a t-statistic p-value of somewhere around machine precision; $2.9e-32$ wouldn't be far off. Did you leave out that $\sqrt{500}$ term by any chance? $\endgroup$
    – jbowman
    Feb 11 '18 at 20:24
  • $\begingroup$ corrected the question and added more details $\endgroup$
    – Nickpick
    Feb 11 '18 at 20:52
  • $\begingroup$ What fraction of the values are $\leq 0$? $\endgroup$
    – jbowman
    Feb 11 '18 at 21:18
  • $\begingroup$ 33% are <=0. Here the full time series: goo.gl/8Ucxhw $\endgroup$
    – Nickpick
    Feb 11 '18 at 21:24
  • 2
    $\begingroup$ You won't have independent identically distributed observations within either sample so neither test could be appropriate. However it's not clear why you'd expect the p-values to be similar; it's easy to get cases where means are essentially equal but P(X>Y) differs substantially from 1/2. $\endgroup$
    – Glen_b
    Feb 11 '18 at 22:24

I am assuming you are trying to see if the mean of all data points in one sample over the time series is different from 0.

The choice of whether to use the t-test or the Mann-Whitney U test depends on how the data is distributed. The t-test is for normally-distributed data. The Mann-Whitney test is more general in that it can be applied to both normally and non-normally distributed data under more relaxed conditions (see https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test). From the description but having not seen the data, having a series of small increases and then large drops would suggest to me it is non-normal since I can't picture the data scattered around one mean with the bulk of the other data points distributed around the mean in a certain standard deviation. But without seeing the data in question or any other visualizations (e.g., a QQ-plot) it is hard to say definitively.

So you just need to decide whether the data is normally distributed or not when evaluating which test appears more appropriate. From what I remember, the Mann-Whitney U test result is fairly consistent with the t-test result when the data is normally distributed, so any deviations for large sample sizes between the two may imply the data is not normally distributed. But overall, if "most of then [values are] +2 and some of them -200", that would suggest to me that the mean isn't at 0.

  • $\begingroup$ You are correct in your assumptions. The data is not normally distributed, as most of the values are +2 and some ore between -1 and -200. Overall total cumulative return is around +50. So why are the two tests showing such large discrepancies. The result of the ranksum tests doesn't seem plausible. $\endgroup$
    – Nickpick
    Feb 11 '18 at 19:36
  • $\begingroup$ the mean is in fact +100 and the standard deviation is 170. The amount of samples is 500. $\endgroup$
    – Nickpick
    Feb 11 '18 at 19:46
  • $\begingroup$ Some more details: the p_stat of the t-test is 0.6, but the p_stat of the ranksums is 2.9e-32. How can that be? For t-test I'm using scipy.stats.ttest_ind and for ranksums I use scipy.stats.ranksums $\endgroup$
    – Nickpick
    Feb 11 '18 at 19:51
  • $\begingroup$ I am just really confused then. How can most of your values be around 2 and some around -200 but then your mean be positive 100? This means there is at least one value much higher than positive 2. In any case, everything you have told me suggests that the mean is not 0, so I am not sure why you are doubting the Mann-Whitney result especially when it is more robust is non-normal data sets. In this case the output of the t-test may not be reliable, and there isn't any guarantee it will have a similar output when the data is non-normal. $\endgroup$
    – Daniel
    Feb 11 '18 at 19:55
  • $\begingroup$ yes you are right, there is one value that is higher. But still, the variance is soo high, how can this be statistically significant? Or maybe the difference comes from the fact that I was looking for a way to see if the mean is positive rather than non-zero? If so, is there a way to do a one sided non-parametric test? $\endgroup$
    – Nickpick
    Feb 11 '18 at 20:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.