Independent samples test for mean without normal distribution with T-score

Question

I have two independent samples of male and female order value.
Female number of observations = 26887
Male number of observations = 12928
Female mean order value = 133.03
Male mean order value = 145.24
Female sample variance = 10,406.38
Male sample variance = 17,563.87

Pooled variance = 12,730.36
T-score = 0.11
p-value = 0.50926

Same on the image

Terrible p-value 🤦‍♂️

Samples distribution is far from normal

Questions:
1. What is the reason for such a bad p-value?
2. How can I improve it?
3. Is CLT applicable here?

Answer 1

Your calculation of the test statistic is wrong. You're dividing the difference in means by the pooled standard deviation, rather than the standard error of the difference in means.

The formula for the (equal-variance) two sample t-test statistic would be:

$$\frac {{\bar {x}}_{1}-{\bar {x}}_{2}}{s_{p}\cdot {\sqrt {{\frac {1}{n_{1}}}+{\frac {1}{n_{2}}}}}}$$

but you seem to have

$$\frac {\bar{x}_1-\bar{x}_2}{s_{p}}$$

There may be additional issues.

In relation to your later questions:

• I would also suggest that the Welch-type statistic would be more suitable, since the variances are not all that close; with such a large sample size there's no need to assume they're equal in the population, though it will make no difference to your conclusions.

• The distributions are each quite skew but even so the samples are quite large -- seemingly large enough that you could reasonably treat the sample means as having a normal distribution (and you could probably reasonably treat the sample variances as effectively "known"). However the first sample is quite heavy tailed on the right, so there might perhaps be some question there about the appropriateness of assuming the variance is finite (however, knowledge of what the variable is removes that concern; dollar value of orders will be bounded).

  So while you could perhaps argue to use a z-test (though again, rather on the Welch statistic) on the basis that the statistic should have very close to a normal distribution, the bigger issue could be one of power; the test is going to be considerably less powerful than one with a more suitable distributional assumption (I don't really know enough about your variables to suggest something, but I'd expect something called "order values" might be roughly lognormalish or perhaps some other right skewed distribution like the Gamma; you have enough data to sample split and choose a reasonable model so you wouldn't necessarily need an additional source of information for model selection.)

  On the other hand, you seem to have such huge samples as to have power to spare, so it hardly matters.