1
$\begingroup$

I got two sample sets a training set around 32k rows and I have a test set of 16k rows. I want to test if the two sets are randomly split. I checked the normal distribution and it is not met for none of the variables in the datasets. So I cannot use the Central Limit Theorem because I do have a large sample size but there are still skewed. Then I checked the variances of each variables and compared the train with the test set and the variances are equal on all variables. In order to test if the two sample size are random, I wanted to do a Chi Square test with each categorial variable e.g. "gender" test set with "gender" train set and in this way to check if the population is the same. For the continuous variables, since there are not normal distributed I cannot use t-test and some statistics did not work due to unequal length. Which test is the best to test for randomness given the unequal sample size and not normal distribution? And would you recommend to test each variable separately on the two datasets? Is this the right approach?

$\endgroup$
2
  • 1
    $\begingroup$ The CLT does not assume normality of a single observation but rather implies normality of the sample mean. Thus your question does not make much sense. $\endgroup$
    – Michael M
    Jun 12, 2022 at 7:40
  • $\begingroup$ Then I start from the beginning : What would be the best way to test if two sets are randomly split and there is no bias on one of the sample sets? $\endgroup$ Jun 12, 2022 at 9:58

1 Answer 1

1
$\begingroup$

I got two sample sets a training set around 32k rows and I have a test set of 16k rows

According to this description, it seems the training/test set is randomly split, then you don't need to test this. This is something like test random split is random (Does significance test make sense to compare randomised groups at baseline?).

Only some specific scenarios where we need to test whether a sequence is random, for example, we invent a new random number generation algorithm.

$\endgroup$
4
  • $\begingroup$ We don't know if it is randomly split and that is what I try to find out if these two samples are random or not. Based on exploratory analysis I do not want to draw conclusion. Hence I wanted to use statistical test on the same variables and test them against each other in the test vs train set. My proposal was to use ChiSquare for categorial and for the continuous variables Mann Whitney U Test. $\endgroup$ Jun 12, 2022 at 10:45
  • $\begingroup$ if the train and test data set are in hand, then you can just (re)do random split $\endgroup$
    – wei
    Jun 12, 2022 at 10:51
  • $\begingroup$ if you don't know a split is random, you just need test the actual count vs expected count using the chisq.test. This has nothing to do with the type of the feature $\endgroup$
    – wei
    Jun 12, 2022 at 10:58
  • $\begingroup$ I think they want to show that the two samples are balanced with respect to a couple of variables. $\endgroup$
    – Michael M
    Jun 12, 2022 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.