Let's say you are a city builder and want to optimize the number of laundromats and convenience stores based on the satisfaction of people (just a made-up example). Your data might look like this:

Satisfaction # Laundromats # Convenience Stores 100% 4 10
... 50% 5 11

  1. You could optimize for satisfaction by creating a regression on all your data and then finding the global maximum for the resulting equation, using all of the data points. For example, if your regression is f(x)= B0 + B1*laundromats + B2*convenience, you might maximize f(x) where x=satisfaction, and the regression was built on the entire data set.

  2. Alternatively, you could build a predictive model - build a model and then predict over a variety of combinations of # of laundromats and # of convenience stores. Then choose the combination that predicts the highest Satisfaction.

In the case of 1, you might build your regression such that R^2 or some other metric indicates the model is a good fit. You might do this on the entire data set, and not split into a training and test set.

In the case of 2, you might split your data into train and test sets, so you could evaluate the generalization of your model before running predictions for a variety of combinations of features.

In both cases, you're trying to answer the question: "How many laundromats and convenience stores should I build to make people the most happy?" In my mind, that is a predictive question. So you would want to split your data into training and test sets to test the out-of-sample generalization of your predictions.

On the other hand, you might argue that you want the best fitting equation to your data, so that you can maximize this equation. In which case, over-fitting might not be that big of a concern, and therefore doing some type of generalization test unnecessary (no train/test split).

What is the correct way to approach this?

  • $\begingroup$ in NEITHER case will you be truly answering the question "How many laundromats and convenience stores should I build to make people the most happy?" Looking at existing data, you will see how correlated these cases are, and you might make an out-of-data inference that building one more laundromat would make people more satisfied, but it may be that some other set of variables is determining this. $\endgroup$ – zbicyclist Feb 21 '17 at 15:43
  • $\begingroup$ But to get closer to a direct answer: assuming you have enough data, it's clearly advantageous (and often humbling) to verify the regression results on a new set of data before trusting the regression. $\endgroup$ – zbicyclist Feb 21 '17 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.