A disclaimer:

By using an informal term such as "generalize", I am aware I am getting close to philosophical territory, and that my question could be seen as unsuitable for CV.

I will do my best to be specific enough in phrasing it, to allow for meaningful answers by the standards of this community.

I am trying to gain some broad overview of the methods with which ML researchers and practitioners, statisticians and mathematicians, tackle the question whether a model successfully learned to generalize -- or whether it failed to do so.

Or maybe better, to avoid framing the question categorically: to which degree a model learned to generalize?

In other words, I am asking which formally specified (and to remain practical: computable and computationally tractable) methods exist that can be seen as addressing the informal question posed in the title, that of a model's or method's ability to 'generalize'.

Does that question make any sense up to this point? And is it possible to answer it in the context of CV?

Some additional remarks, trying to clarify the question further:

  • What I'm asking about is maybe a taxonomy of sorts (happily accepting your personal taxonomy, in case no canonical one exists) of such methods, of both 'hard' formal results, and methods relying on empirical evaluation.

  • Closely related to 'generalization', and, I'm afraid, equally underspecified: the notion of systematicity. A model's ability to generalize often seems to be mentioned alongside the question whether the model found a systematical solution for a task it was trained on. Does that help in any way? (Probably not.)

  • Maybe the following distinction needs to be made: mentioning "models" above, I somewhat conflate the general learning algorithm or method, and particular instances of these methods, i.e. models that are constructed by the algorithm from training data.

    My question then contains at least two sub-questions: ways to speak about the 'generalization' ability of the algorithm itself, and which to evaluate the same for a trained model?

  • At least in the context of neural networks (the family of models I'm most familiar with), it seems to me that the 'generalization' question is answered mostly empirically, and mostly by one particular method only (perhaps the only one available, in reality?): by separating the data into distinct sets (standard being 2 for train/test, or 3 for train/test/evaluation), keeping data used for training and performance evaluation separate.

    As a consequence, we can then consider overfitting of a model (as measured by the model's performance on the withheld data), and find ways to combat it (regularization).

    Which brings me to consider another distinction: (i) 'generalization' as performance over unseen data that we (plausibly) know was generated by the same function that also generated the training data, in contrast to: (ii) generalization to (unseen) data that we might only hypothesize or believe to be produced by the same underlying, general function.

    By my own (still very incomplete) understanding, it appears then that we are usually only concerned with evaluating 'generalization' of the first type (i.e. by measuring and comparing performance over unseen data generated with some certainty by the same function as the training data -- I say "with some certainty", because we either chose the function generating the data ourselves, or following from the relatively natural assumption that the same function generated the particular data set we used, say, a set of 1 million 400 by 600 px pictures of dancing cats), while generalization of the second type is not usually measured or considered (i.e. performance over unseen examples that are in a sense truly new and different from the ones encountered during training, but that we might believe are the product of the same function that generated the training data).

    Here's another strong possibility: I am completely wrong with that characterization (not really surprising, considering how confused I still am about all things ML). If that's the case, my apologies for misrepresenting (and misunderstanding) the current approaches.

  • 4
    $\begingroup$ In my opinion, philosophy of statistics is definitely on topic here, although I don't know of any academic philosophers who are regulars. $\endgroup$ Jul 15 '16 at 20:12
  • $\begingroup$ @Kodiologist Thank you for making it through my wall of text. I should maybe say, I am not necessarily asking about the philosophical side of it alone: putting it more succinctly, I wonder which other ways to evaluate models are used, other than the train/test data separation mentioned in my last bullet point, that can be seen as related to the question of generalization? $\endgroup$ Jul 15 '16 at 20:20

One important notion of generalizability, especially in machine learning, is predictive accuracy: the degree to which a learner can predict the value of the dependent variable in cases it wasn't trained with. Predictive accuracy can be estimated with a wide variety of techniques, including a train-test split, cross-validation, and bootstrapping.

  • $\begingroup$ Thank you for answering. (I'd upvote it, but cannot yet on here). I know (or at least: heard of) the techniques you mention, but I guess I was thinking of a slightly broader view on the question. Then again, as you pointed out above in a comment to my question, that might be more philosophy of statistics than statistics proper. I'll leave the question open for another few days, but will mark your answer as accepted o/w. $\endgroup$ Jul 17 '16 at 11:54
  • 1
    $\begingroup$ I'm curious myself for a more thorough answer, so I've bountied the question. $\endgroup$ Jul 19 '16 at 14:18
  • $\begingroup$ In classification case looking at probabilities (if given) and performance measures based on them like the brier score or logarithmic loss, can give a broader picture of the goodness of a model. In the regression case (and classification) to look specifically at the variance of your prediction can be quite interesting. $\endgroup$
    – PhilippPro
    Jul 20 '16 at 5:28
  • $\begingroup$ In my view, proper scoring rules are indeed the right way to assess predictive accuracy in the context of probabilistic classification. $\endgroup$ Jul 20 '16 at 6:20

For medium to large datasets, most practitioners will use a holdout set. This is what you refer to as training, validation, and test sets. A holdout set consists of data that your model has never seen before. If your model generalizes well on the holdout set, then presumably it will generalize equally well on live production data.

Regarding your last bullet point on holdout sets -- there aren't two types of generalizability. There is only one type, and so it's simply called generalizability. If your training dataset doesn't represent real world data, then there is no point in building a model. Unfortunately there is no simple rule for knowing how close your training data will represent production data. You just have to use good judgement (e.g., model build data should be sourced from the same production systems that you would pull from when your model is deployed).

Generalizability is often not static just like the real world is often not static. If your model is used to make important decisions, you will often also do post-production monitoring of it to make sure that it continues to generalize well. As your model's generalizability decays over time, as is most often the case with real-world (e.g., financial) data, then you'll need to do a refit.

  • $\begingroup$ "there aren't two types of generalizability" — Really? Think about predictive accuracy versus how accurately the model describes the actual data-generating process. Are these not both some kind of generalizablility? $\endgroup$ Jul 19 '16 at 20:19
  • $\begingroup$ @Kodiologist Yes, from a practical perspective they are the same kind of generalizability. I would say accuracy (or any other loss metric for that matter) is the same as "how accurately the model describes the actual data-generating process." Trying to understand the true "mechanism" for how some data is generated sounds too abstract to be measured objectively other than by using a holdout set and applying a loss function $\endgroup$
    – Ryan Zotti
    Jul 19 '16 at 20:51
  • $\begingroup$ Where I work in industry, predictive power is everything and explanatory power does not matter (think neural networks and gradient boosting vs GLMs and p-values). I suppose in an academic setting there might be a distinction to make, but I'm not an academic, so to me and those I work with they are the same. $\endgroup$
    – Ryan Zotti
    Jul 19 '16 at 20:56
  • $\begingroup$ Yeah, scientists are often explicitly concerned with explanation or with trying to learn about true underlying causal or data-generating processes. $\endgroup$ Jul 19 '16 at 20:58
  • $\begingroup$ Thanks for your answer to my question. Two follow-ups: 1. You mention holdout sets, which fall within the context of cross-validation (mentioned by Kodiologist as well). That seems to be the most common technique to evaluate the 'generalization' of a model, but what I wanted to know: is there any other standard technique that does not fall under the crossvalidation umbrella, that I simply don't know about? Maybe to ML veterans this knowledge, that this is the only technique, is so self-evident it doesn't bear mentioning, but I'm still pretty ignorant on these things. (1/4) $\endgroup$ Jul 21 '16 at 20:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.