I'm exploring the Titanic dataset as my first experience of a Kaggle. Firstly, I want to test that the training and test sets were indeed sampled randomly from the population.

For continuous variables, I considered the two sample Kolmogorov-Smirnov test and the Wilcoxon rank-sum test. Which of these is more appropriate? (They both appear to be non-parametric.)

For discrete variables, I used the chi2 test with a contingency table. This found that the Embarked variable was distributed in a significantly different way between the training and test sets. Is this the wrong test?

train_set = pd.read_csv('train.csv')
test_set = pd.read_csv('test.csv')

from scipy.stats import ks_2samp, chi2_contingency, ranksums

def ks_test(column):
    _, p = ks_2samp(train_set[column], test_set[column])
    print(f'{column}: p={p:.2f}')

def ranksum_test(column):
    _, p = ranksums(train_set[column], test_set[column])
    print(f'{column}: p={p:.2f}')

def chi2_test(column):
    cont_table = pd.DataFrame({
        'train': train_set[column].value_counts(), 
        'test': test_set[column].value_counts()
    chi2, p, dof, exp = chi2_contingency(cont_table.values)
    print(f'{column}: p={p:.2f}')


Gives the output:

Age: p=0.80
Age: p=0.84
Fare: p=0.78
Fare: p=0.92
Pclass: p=0.60
Sex: p=0.74
Embarked: p=0.02

You're basically rolling a 50 sided dice 7 times. If you do that, there's a 14% chance that at least one of those dice comes up with the value 1, just by chance.

  • $\begingroup$ This doesn't really answer my question of what are the most appropriate statistical tests. $\endgroup$ – David Hall Apr 7 '17 at 19:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.