I have two independent samples of male and female order value.
Female number of observations = 26887
Male number of observations = 12928
Female mean order value = 133.03
Male mean order value = 145.24
Female sample variance = 10,406.38
Male sample variance = 17,563.87

Pooled variance = 12,730.36
enter image description here T-score = 0.11
enter image description here p-value = 0.50926

Same on the image enter image description here

Terrible p-value 🤦‍♂️

Samples distribution is far from normal enter image description here

1. What is the reason for such a bad p-value?
2. How can I improve it?
3. Is CLT applicable here?

  • $\begingroup$ The reason for such a high p-value is that you have made a calculation error somewhere (a Welch t-test gives an absolute t-value of about 9.24; an equal variance two sample t test gives an absolute t-value of about 10.11) . Please show what formulas you used, giving explicit steps $\endgroup$ – Glen_b -Reinstate Monica Feb 8 at 9:54
  • $\begingroup$ Added screenshots of the formulas $\endgroup$ – Svetoslav Dimitrov Feb 8 at 9:57
  • $\begingroup$ Yep, the error is quite plain. $\endgroup$ – Glen_b -Reinstate Monica Feb 8 at 9:58
  • $\begingroup$ However, I meant "please show the algebraic formulas" not "please show me spreadsheet formulas" $\endgroup$ – Glen_b -Reinstate Monica Feb 8 at 10:06

Your calculation of the test statistic is wrong. You're dividing the difference in means by the pooled standard deviation, rather than the standard error of the difference in means.

The formula for the (equal-variance) two sample t-test statistic would be:

$$\frac {{\bar {x}}_{1}-{\bar {x}}_{2}}{s_{p}\cdot {\sqrt {{\frac {1}{n_{1}}}+{\frac {1}{n_{2}}}}}}$$

but you seem to have

$$\frac {\bar{x}_1-\bar{x}_2}{s_{p}}$$

There may be additional issues.

In relation to your later questions:

  • I would also suggest that the Welch-type statistic would be more suitable, since the variances are not all that close; with such a large sample size there's no need to assume they're equal in the population, though it will make no difference to your conclusions.

  • The distributions are each quite skew but even so the samples are quite large -- seemingly large enough that you could reasonably treat the sample means as having a normal distribution (and you could probably reasonably treat the sample variances as effectively "known"). However the first sample is quite heavy tailed on the right, so there might perhaps be some question there about the appropriateness of assuming the variance is finite (however, knowledge of what the variable is removes that concern; dollar value of orders will be bounded).

    So while you could perhaps argue to use a z-test (though again, rather on the Welch statistic) on the basis that the statistic should have very close to a normal distribution, the bigger issue could be one of power; the test is going to be considerably less powerful than one with a more suitable distributional assumption (I don't really know enough about your variables to suggest something, but I'd expect something called "order values" might be roughly lognormalish or perhaps some other right skewed distribution like the Gamma; you have enough data to sample split and choose a reasonable model so you wouldn't necessarily need an additional source of information for model selection.)

    On the other hand, you seem to have such huge samples as to have power to spare, so it hardly matters.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.