I have a dataset I want to use for machine learning that looks something like this:
Group_ID | Column_1 | Column_2 | Column_3 ... ========================================== A | 1 | 2 | 33 A | 2 | 2 | 3765 A | 3 | 6 | 3436 A | 4 | 8 | 32 B | 5 | 9 | 33 B | 3 | 34 | 385 B | 7 | 25 | 3 B | 3 | 1 | 38 C | 6 | 2 | 3 C | 8 | 2 | 4 D | 7 | 1 | 5 D | 6 | 9 | 11
Group_ID is an identifier which isn't used by the model, it's just a reference I have, and the rest of the columns are the predictors.
I want to create a training and test set where the rows are split say 70-30, but all rows belonging to a group are always in the same dataset.
So if we aggregate the counts of rows, and get something like this:
A | 145 B | 110 C | 60 D | 35
The final split should be ~70-30, so approx. 245-105, so I'd want the final training-test split to be:
Training set: Group A, C and D Test set: Group B
I want to keep them together to see if there is a difference between the groups in terms of how well my classifiers perform.
First of all:
- statistically speaking, is there anything wrong with splitting like this?
- is there a simple way of doing this kind of stratified sampling (in Python)?