I've been learning about Naive Bayes classifiers using the nltk package in Python. I'm working on a gender classification model. I have some labeled data for names with male/female probabilities, and to create the model I used a 80:20 split between training and testing sets.
I understand the importance of keeping these sets separate when you are optimizing your model, but once you've determined the features you want to include, doesn't it make sense to shift all of your existing labeled data into the training set when you're actually implementing the model on new data? My intuition is that this way when I apply the model to new, unseen names in the context of a real-world application, my model will have been trained on a larger data set. Is this correct, or are you supposed to keep your split and use only part of the data for training even after you've settled on the features you want to include?
If you do maintain the split, do you have to always use the same exact data in the training set or can you shuffle which data is in training and testing sets each time you run the model? (My intuition here is that the training set should stay fixed, but not sure)