0
$\begingroup$

I am currently testing with following sample data:

''' Student t test '''

np.random.seed(123)

data1 = np.random.randn(1000)*5 +50   #SO: this would have mean 50 and Std dev  5

sns.distplot(data1,hist=True)

data2 = np.random.randn(1000)*5 +55  #SO: this would have mean 55 and Std dev  5

sns.distplot(data2,hist=True)

np.std(data1) #5.003937687581167

np.std(data2) #4.790047775711126

stat, p = scipy.stats.ttest_ind(data1,data2)

print(stat)  # =  -23.90818904550741

print(p)  # = 2.6073235694238827e-111

When I see the plots they look as below.

two data distributions

Question is : P value which the ttest_ind function is giving is well below 0.05. But mean values of data1 and data2 are Not > 5 units apart. ( the Std dev value of both is 5.) AS I am thinking, if Data1 and data2 are > (2*std dev) units apart (approx) then p value should be less than 0.05... ( which means they have indeed statistically significant difference)

I referred to the official documentation of scipy here https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html

But did not quite understand using the documentation present there. Am I wrongly interpreting the result? OR : How to understand/Interpret this.?

$\endgroup$
1
$\begingroup$

You misunderstood the definition of p-value. The documentation is not very good, so I recommend reading a description from some statistics book.

The p-value for this test means:

If the first sample is taken from a normal distribution and the second sample is taken from a normal distribution with the same mean and variance, what is the probability of seeing a difference of at least -5.24 in the means of the two samples?

As you can see, this probability is very small. Since these samples are taken at random, it would be quite bizarre if the means of the samples differed by so much.

The reason why your picture seems to suggest otherwise is because you have plotted the distribution of the sample from each distribution, but the test is looking at the distribution of the means of samples of size 1000 from each distribution. If you take many samples of size 1000 from each distribution and calculate the means of those samples, the distribution of means will look a lot thinner.

In fact, you can do it using this code. In this code, we do the sampling 500 times and calculate the mean of each sample.

n_samples = 500
sample1_mu = []
sample2_mu = []

for i in range(n_samples):
    sample1 = np.random.randn(1000)*5 +50
    sample1_mu += [np.mean(sample1)]
    sample2 = np.random.randn(1000)*5 +55
    sample2_mu += [np.mean(sample2)]

sns.distplot(sample1_mu,hist=True)
sns.distplot(sample2_mu,hist=True)

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.