# Test that training and test sets are sampled consistently

I'm exploring the Titanic dataset as my first experience of a Kaggle. Firstly, I want to test that the training and test sets were indeed sampled randomly from the population.

For continuous variables, I considered the two sample Kolmogorov-Smirnov test and the Wilcoxon rank-sum test. Which of these is more appropriate? (They both appear to be non-parametric.)

For discrete variables, I used the chi2 test with a contingency table. This found that the Embarked variable was distributed in a significantly different way between the training and test sets. Is this the wrong test?

train_set = pd.read_csv('train.csv')

from scipy.stats import ks_2samp, chi2_contingency, ranksums

def ks_test(column):
_, p = ks_2samp(train_set[column], test_set[column])
print(f'{column}: p={p:.2f}')

def ranksum_test(column):
_, p = ranksums(train_set[column], test_set[column])
print(f'{column}: p={p:.2f}')

def chi2_test(column):
cont_table = pd.DataFrame({
'train': train_set[column].value_counts(),
'test': test_set[column].value_counts()
})
chi2, p, dof, exp = chi2_contingency(cont_table.values)
print(f'{column}: p={p:.2f}')

ks_test('Age')
ranksum_test('Age')
ks_test('Fare')
ranksum_test('Fare')
chi2_test('Pclass')
chi2_test('Sex')
chi2_test('Embarked')


Gives the output:

Age: p=0.80
Age: p=0.84
Fare: p=0.78
Fare: p=0.92
Pclass: p=0.60
Sex: p=0.74
Embarked: p=0.02


You're basically rolling a 50 sided dice 7 times. If you do that, there's a 14% chance that at least one of those dice comes up with the value 1, just by chance.