(I found a more concise way of expressing my question - I'm leaving the original wording below for reference purposes)

Is there a difference in the way we approach supervised learning problems when the problem corresponds to an (unknown) true model/data generating process vs. the way we approach supervised learning problems where there is no true underlying model/process?

Does this impact the choice of loss function, regularization method, the choice of test-train split or cross validation approach, etc...?

Is there a difference between real supervised learning problems and constructed supervised learning problems, from a statistical point of view?

Consider the following 2 classification problems:

  1. We have a data set of biometric measurements for two groups of people, healthy people and people with a certain disease, and we want to build a classifier that distinguishes between the two groups based on their biometric measurements.

  2. We have a bunch of website browsing data and chosen items from customer visits to a e-comm site, and we want to predict whether a customer will be a return customer or not.

Both are binary classification tasks, but in the first case we are certain that the categories of healthy person and sick person are ontologically real categories that are "out there" in the world.

When building a classifier for this problem, we might struggle with which type of classifier to choose, which features to choose, which loss function, etc....but in so far as we are trying to model health/disease as a process, there is a real (but unknown) process out there towards which we could converge if we gather enough data and choose the right models.

In the second case, there is no real category of people that are "return customers". Different people have different reasons for returning to the site later to make a another purchase. There is no underlying process towards which we can converge if we get better data and more sophisticated classifiers. Instead we are just trying to do a better than average educated guess on the outcome of a random event. The category "return customer" is constructed in the sense that it was invented by the business, not a reflection of some concrete feature.

Is there a name for this distinction between the two types of supervised learning problem I just described?

And does this distinction have implications for which type of model, regularization, loss functions, etc...to choose in each case?

As an additional clarification, consider a similar distinction between time series forecasting problems: Ecological time series usually correspond to a real data generating process, whereas retail demand time series correspond to a constructed process.

  • 1
    $\begingroup$ I don't see how sick/healthy people are any more a real category than return vs nonreturning customers. Both are equally human abstractions which are useful for describing the arrangement of atoms in the universe (or a small corner of it) succinctly. And they both have ambiguity and fuzzy boundaries, which is typical of most human abstractions. $\endgroup$ – shimao Apr 10 '18 at 22:21
  • $\begingroup$ Different people have different reasons for returning. Different people also have different ways of getting sick. Does that mean there's no underlying process behind sickness? $\endgroup$ – shimao Apr 10 '18 at 22:31
  • 1
    $\begingroup$ @shimao "Different people also have different ways of getting sick." In my example I was thinking of people with a specific disease, for example type II diabetes or pancreatic cancer, or something like that. Not of the generic category "sick people". $\endgroup$ – Skander H. Apr 10 '18 at 22:37
  • 1
    $\begingroup$ Even a specific disease can have multiple transmission vectors. You can almost always keep breaking down any category into subcategories like this. $\endgroup$ – shimao Apr 10 '18 at 22:42
  • $\begingroup$ What do you have in mind when you say "no true underlying process"? In that case, where did the data come from? $\endgroup$ – user20160 Apr 25 '18 at 20:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.