Return to Answer

For medium to large datasets, most practitioners will use a holdout set. This is what you refer to as training, validation, and test sets. A holdout set consists of data that your model has never seen before. If your model generalizes well on the holdout set, then presumably it will generalize equally well on live production data.

Regarding your last bullet point on holdout sets -- there aren't two types of generalizability. There is only one type, and so it's simply called generalizability. If your training dataset doesn't represent real world data, then there is no point in building a model. Unfortunately there is no simple rule for knowing how close your training data will represent production data. You just have to use good judgement (e.g., model build data should be sourced from the same production systems that you would pull from when your model is deployed).

Generalizability is often not static just like the real world is often not static. If your model is used to make important decisions, you will often also do post-production monitoring of it to make sure that it continues to generalize well. As your model's generalizability decays over time, as if most often the case with real-world (e.g., financial) data, then you'll need to do a refit.

For medium to large datasets, most practitioners will use a holdout set. This is what you refer to as training, validation, and test sets. A holdout set consists of data that your model has never seen before. If your model generalizes well on the holdout set, then presumably it will generalize equally well on live production data.

Regarding your last bullet point on holdout sets -- there aren't two types of generalizability. There is only one type, and so it's simply called generalizability. If your training dataset doesn't represent real world data, then there is no point in building a model. Unfortunately there is no simple rule for knowing how close your training data will represent production data. You just have to use good judgement (e.g., model build data should be sourced from the same production systems that you would pull from when your model is deployed).

Generalizability is often not static just like the real world is often not static. If your model is used to make important decisions, you will often also do post-production monitoring of it to make sure that it continues to generalize well. As your model's generalizability decays over time, as is most often the case with real-world (e.g., financial) data, then you'll need to do a refit.

For medium to large datasets, most practitioners will use a holdout set. This is what you refer to as training, validation, and test sets. A holdout set consists of data that your model has never seen before. If your model generalizes well on the holdout set, then presumably it will generalize equally well on live production data. Regarding your last bullet point on holdout sets -- there are not two types of generalizability; there is only one type. If your training dataset doesn't represent real world data, then there is no point in building a model.

If your model is used to make important decisions, you will often also do post-production monitoring of it to make sure that it continues to generalize well. As your model's generalizability decays over time, as if most often the case with real-world (e.g., financial) data, then you'll need to do a refit. Generalizability is often not static just like the real world is often not static.

For medium to large datasets, most practitioners will use a holdout set. This is what you refer to as training, validation, and test sets. A holdout set consists of data that your model has never seen before. If your model generalizes well on the holdout set, then presumably it will generalize equally well on live production data. 

Regarding your last bullet point on holdout sets -- there aren't two types of generalizability. There is only one type, and so it's simply called generalizability. If your training dataset doesn't represent real world data, then there is no point in building a model. Unfortunately there is no simple rule for knowing how close your training data will represent production data. You just have to use good judgement (e.g., model build data should be sourced from the same production systems that you would pull from when your model is deployed).

Generalizability is often not static just like the real world is often not static. If your model is used to make important decisions, you will often also do post-production monitoring of it to make sure that it continues to generalize well. As your model's generalizability decays over time, as if most often the case with real-world (e.g., financial) data, then you'll need to do a refit.

For medium to large datasets, most practitioners will use a holdout set. A holdout set consists of data that your model has never seen before. If your model generalizes well on the holdout set, then presumably it will generalize equally well on live production data.

If your model is used to make important decisions, you will often also do post-production monitoring of it to make sure that it continues to generalize well. As your model's generalizability decays over time, as if most often the case with real-world (e.g., financial) data, then you'll need to do a refit. Generalizability is often not static just like the real world is often not static.

For medium to large datasets, most practitioners will use a holdout set. This is what you refer to as training, validation, and test sets. A holdout set consists of data that your model has never seen before. If your model generalizes well on the holdout set, then presumably it will generalize equally well on live production data. Regarding your last bullet point on holdout sets -- there are not two types of generalizability; there is only one type. If your training dataset doesn't represent real world data, then there is no point in building a model.

If your model is used to make important decisions, you will often also do post-production monitoring of it to make sure that it continues to generalize well. As your model's generalizability decays over time, as if most often the case with real-world (e.g., financial) data, then you'll need to do a refit. Generalizability is often not static just like the real world is often not static.