From the examples I've seen, it seems to me that generally people build a TFIDF transformer based on only on text in the training data, and then use that transformer to transform the test data.
Why isn't OK (or is it) to have the TFIDF transformer fit on text in both the training and test data? After all, one wouldn't be looking at the test data's labels, and thus one could practically do this in the real-world when applying to new datasets whose labels are unknown.
TL;DR - it violates "the rules" of cross-validation.
In Machine Learning your data comes all at once, so it's up to you to make a split and experiment with it, preferably many times over. Splitting data into training and test sets ensures that your methodology is robust on new data, not just accurate for an idiosyncratic subset. It also ensures that you get a good reading of your methodology measurement.
Training/fitting with test data is like giving schoolchildren their complete math test as a homework assignment - you shouldn't be surprised when they all come back with A's. But their letter grade, in this case, doesn't indicate real subject area knowledge, but rather that they knew all the answers to the questions beforehand. The analogy is nearly 1:1 with the thinking behind why you should partition your data.
While tf-idf itself doesn't do any prediction or grouping, what you want to show is that when fit it with an arbitrary subset of data, that tf-idf is accurate enough to work with new data points. If it doesn't work with a subset, it won't work on the whole, or so the thinking goes.
It's also worth pointing out:
Why isn't OK (or is it) to have the TFIDF transformer fit on text in both the training and test data? After all, one wouldn't be looking at the test data's labels, and thus one could practically do this in the real-world when applying to new datasets whose labels are unknown.
What you're describing here is actually cross-validation, it's just that the training set is the data you currently have and the test set is the stuff in the wild. The only problem is that often times you want to test and make sure your methodology makes sense before you go out and apply it in the wild, in part because it can be hard to continually gather and clean enough good data to make business sense. It's generally advised to sandbox and test your development with a closed dataset so that you can guard against the kind of unforeseen edge cases that might happen with a constant influx of new data.
Oftentimes the assumption is that you will receive the test data one sentence at a time. For example, if you are working on a search engine you wouldn't have more than one query at a time available for processing.
