I have a highly imbalanced dataset. My question is how to split the dataset for training and testing? I want to have a separate training set and a separate testing set.
One idea I had is just to shuffle the dataset and then take 70% for training and the rest for testing. Is that good strategy?
Stratified cross-validation is a good technique in the case of highly imbalanced classes. For binary classification with a training/test split rather than cross-validation, this involves the training set having the same proportion of positive-labeled points as the test set (and hence the same as the overall training set). Such a split is easily accomplished by splitting your points by label resulting in two sets, shuffling each of these sets, and then placing the first x% of each set into the training set and the last x% of each set into the test set.
While intuitively this method seems to provide a more representative training sample (since the label proportions match the overall set), it is easier to form generalization error guarantees if you instead just use a random (non-stratified) partition. This is all assuming that your data points are drawn iid of course. In any case, if you are dealing with small datasets, or datasets such that the number of points with the least frequent label is very small, then not using a stratified approach can lead to zero of the least frequent class's points occurring in the training or test set with non-trivial probability.
I wanted to comment but I don't have enough reputation, but I think your idea works fine. A heuristic is usually to randomly partition 50% of the data to the training set, 25% to validation set, and 25% to the test set. Where the validation set would be crucial if you were comparing different learning algorithms and trying to see which one performed best.
