(I found a more concise way of expressing my question - I'm leaving the original wording below for reference purposes)
Is there a difference in the way we approach supervised learning problems when the problem corresponds to an (unknown) true model/data generating process vs. the way we approach supervised learning problems where there is no true underlying model/process?
Does this impact the choice of loss function, regularization method, the choice of test-train split or cross validation approach, etc...?
Is there a difference between real supervised learning problems and constructed supervised learning problems, from a statistical point of view?
Consider the following 2 classification problems:
We have a data set of biometric measurements for two groups of people, healthy people and people with a certain disease, and we want to build a classifier that distinguishes between the two groups based on their biometric measurements.
We have a bunch of website browsing data and chosen items from customer visits to a e-comm site, and we want to predict whether a customer will be a return customer or not.
Both are binary classification tasks, but in the first case we are certain that the categories of healthy person and sick person are ontologically real categories that are "out there" in the world.
When building a classifier for this problem, we might struggle with which type of classifier to choose, which features to choose, which loss function, etc....but in so far as we are trying to model health/disease as a process, there is a real (but unknown) process out there towards which we could converge if we gather enough data and choose the right models.
In the second case, there is no real category of people that are "return customers". Different people have different reasons for returning to the site later to make a another purchase. There is no underlying process towards which we can converge if we get better data and more sophisticated classifiers. Instead we are just trying to do a better than average educated guess on the outcome of a random event. The category "return customer" is constructed in the sense that it was invented by the business, not a reflection of some concrete feature.
Is there a name for this distinction between the two types of supervised learning problem I just described?
And does this distinction have implications for which type of model, regularization, loss functions, etc...to choose in each case?
As an additional clarification, consider a similar distinction between time series forecasting problems: Ecological time series usually correspond to a real data generating process, whereas retail demand time series correspond to a constructed process.