Augmenting training data with cases that won't be in future data Background: I am working on coding survey responses, where the respondent writes in a description of their job. So the person might write in "McDonald's Employee" and get coded to something like 1002 which would be the code for fast food for example. The person might also write in something less clear like "I flip burgers" which should also be coded to 1002. For the clear-cut cases we have a reference file that automatically codes these. For the write ins that are not clear-cut we would like to use machine learning to code them (in particular we are using FastText). 
My question is: even though our future data will not contain these clear-cut cases as they will be automatically coded, should we still use them to train our model. Would they add valid information to the model even though we never expect to have to code those cases using the model? I suppose having them in our train/test split will bias optimistically our error rate as it will contain many east to predict cases we will never see, but if we could account for this would it be appropriate?
 A: This would depend on the resolution of your coding. 
Essentially, you need to train a model as a channel to pass the necessary information contained in the text description to the target coding. Take your example for instance, "I flip burgers" contains more information than "McDonald's Employee" if the target coding is restaurant type, as high-end restaurants might still need someone to flip burgers nicely. Therefore, the resolution of your coding scheme would determine if the extra piece of information in "I flip burgers" would be captured by your model. That is, the two descriptions make no difference if the target coding is "catering industry", but it does make a difference for restaurant types. 
Now back to your question, if you indeed want to encode different restaurant types, would adding "McDonald's employee" make your model biased? It depends on the data distribution. The embedding vector for "MacDonald's employee" would clearly point to the direction of "fast food". If there are other descriptions like "I flip Wagyu burgers", it might make "I flip burgers" close to high-end restaurants. But in general, we have a good chance of being right for classifying "I flip burgers" to fast food.  So, even in the high resolution case, the augmentation of adding clear-cuts might bring high quality data points for training. This is very helpful in the case of training deep neural networks, as high-quality data could bring you a good "lottery ticket" in descending on the loss surface.
Hope the discussion can spark some inspirations for you.  
