Let's say you are a city builder and want to optimize the number of laundromats and convenience stores based on the satisfaction of people (just a made-up example). Your data might look like this:
Satisfaction # Laundromats # Convenience Stores
100% 4 10
... 50% 5 11
You could optimize for satisfaction by creating a regression on all your data and then finding the global maximum for the resulting equation, using all of the data points. For example, if your regression is
f(x)= B0 + B1*laundromats + B2*convenience, you might maximize
x=satisfaction, and the regression was built on the entire data set.
Alternatively, you could build a predictive model - build a model and then predict over a variety of combinations of
# of laundromatsand
# of convenience stores. Then choose the combination that predicts the highest
In the case of 1, you might build your regression such that R^2 or some other metric indicates the model is a good fit. You might do this on the entire data set, and not split into a training and test set.
In the case of 2, you might split your data into train and test sets, so you could evaluate the generalization of your model before running predictions for a variety of combinations of features.
In both cases, you're trying to answer the question: "How many laundromats and convenience stores should I build to make people the most happy?" In my mind, that is a predictive question. So you would want to split your data into training and test sets to test the out-of-sample generalization of your predictions.
On the other hand, you might argue that you want the best fitting equation to your data, so that you can maximize this equation. In which case, over-fitting might not be that big of a concern, and therefore doing some type of generalization test unnecessary (no train/test split).
What is the correct way to approach this?