Return to Answer

Consider two types of algorithms:

1. Algorithms come with hyperparameters, which do not change with different data subsets. Cross-validation might be used to evaluate

• the performance of different algorithms
• feature engineering
• feature input
• the selection of best hyperparameters.

However, with those algorithms (such as tree based methods), different data subsets in CV may have different splitting rules. Except the choice of your ML procedures and hyperparameter tuning, you need a final rule to be applied for your future data. Therefore, applying your final choices to the full dataset is normally required to obtain the final model.

2. Algorithms come with parameters that change with different data.

Statistical models such as (generalised) linear models might be easier to understand. CV might be used to evaluate

• the performance of different models
• the performance of different distributional assumptions
• feature engineering
• feature input

With such methods, after selecting a model and features going into the model based on CV, you need regression coefficients to be applied to the future data to make prediction.

Where are those coefficients coming from then?

One way is to apply the model to your entire dataset to obtain them. The other way is to take average on coefficients obtained from different data subsets if the full data is too large to run in one go.

If you are using training/validation/test splits, then I would normally treat training + validation as the full dataset, and the test split as future data to test on your final model.

Consider two types of algorithms:

1. Algorithms come with hyperparameters, which do not change with different data subsets. Cross-validation might be used to evaluate

• the performance of different algorithms
• feature engineering
• feature input
• the selection of best hyperparameters.

However, with those algorithms (such as tree based methods), different data subsets in CV may have different splitting rules. Except the choice of your ML procedures and hyperparameter tuning, you need a final rule to be applied for your future data. Therefore, applying your final choices to the full dataset is normally required to obtain the final model.

2. Algorithms come with parameters that change with different data.

Statistical models such as (generalised) linear models might be easier to understand. CV might be used to evaluate

• the performance of different models
• the performance of different distributional assumptions
• feature engineering
• feature input

With such methods, after selecting a model and features going into the model based on CV, you need regression coefficients to be applied to the future data to make prediction.

Where are those coefficients coming from then?

One way is to apply the final model to your entire dataset to obtain them. The other way is to take average on coefficients obtained from different data subsets if the full data is too large to run in one go.

If you are using training/validation/test splits, then I would normally treat training + validation as the full dataset, and the test split as future data to test on your final model.

Consider two types of algorithms:

1. Algorithms come with hyperparameters, which do not change with different data subsets. Cross-validation might be used to evaluate

   • the performance of different algorithms
   • feature engineering
   • feature input
   • the selection of best hyperparameters.

However, with those algorithms (such as tree based methods), different data subsets in CV may have different splitting rules. Except the choice of your ML procedures and hyperparameter tuning, you need a final rule to be applied for your future data. Therefore, applying your final choices to the full dataset is normally required to obtain the final model.

2. Algorithms come with parameters that change with different data.

Statistical models such as (generalised) linear models might be easier to understand. CV might be used to evaluate

• the performance of different models
• the performance of different distributional assumptions
• feature engineering
• feature input

With such methods, after selecting a model and features going into the model based on CV, you need regression coefficients to be applied to the future data to make prediction.

Where are those coefficients coming from then?

One way is to apply the model to your entire dataset to obtain them.

If you are using training/validation/test splits, then I would normally treat training + validation as the full dataset, and the test split as future data to test on your final model.

Consider two types of algorithms:

1. Algorithms come with hyperparameters, which do not change with different data subsets. Cross-validation might be used to evaluate

• the performance of different algorithms
• feature engineering
• feature input
• the selection of best hyperparameters.

However, with those algorithms (such as tree based methods), different data subsets in CV may have different splitting rules. Except the choice of your ML procedures and hyperparameter tuning, you need a final rule to be applied for your future data. Therefore, applying your final choices to the full dataset is normally required to obtain the final model.

2. Algorithms come with parameters that change with different data.

Statistical models such as (generalised) linear models might be easier to understand. CV might be used to evaluate

• the performance of different models
• the performance of different distributional assumptions
• feature engineering
• feature input

With such methods, after selecting a model and features going into the model based on CV, you need regression coefficients to be applied to the future data to make prediction.

Where are those coefficients coming from then?

One way is to apply the model to your entire dataset to obtain them. The other way is to take average on coefficients obtained from different data subsets if the full data is too large to run in one go.

If you are using training/validation/test splits, then I would normally treat training + validation as the full dataset, and the test split as future data to test on your final model.

Consider two types of algorithms:

1. Algorithms come with hyperparameters, which do not change with different data subsets. Cross-validation might be used to evaluate

   • the performance of different algorithms
   • feature engineering
   • feature input
   • the selection of best hyperparameters.

However, with those algorithms (such as tree based methods), different data subsets in CV may have different splitting rules. Except the choice of your ML procedures and hyperparameter tuning, you need a final rule to be applied for your future data. Therefore, applying your final choices to the full dataset is normally required to obtain the final model.

2. Algorithms come with parameters that change with different data.

Statistical models such as (generalised) linear models might be easier to understand. CV might be used to evaluate

• the performance of different models
• the performance of different distributional assumptions
• feature engineering
• feature input

With such methods, after selecting a model and features going into the model based on CV, you need regression coefficients to be applied to the future data to make prediction.

Where are those coefficients coming from then?

One way is to apply the model to your entire dataset to obtain them.

Consider two types of algorithms:

1. Algorithms come with hyperparameters, which do not change with different data subsets. Cross-validation might be used to evaluate

   • the performance of different algorithms
   • feature engineering
   • feature input
   • the selection of best hyperparameters.

However, with those algorithms (such as tree based methods), different data subsets in CV may have different splitting rules. Except the choice of your ML procedures and hyperparameter tuning, you need a final rule to be applied for your future data. Therefore, applying your final choices to the full dataset is normally required to obtain the final model.

2. Algorithms come with parameters that change with different data.

Statistical models such as (generalised) linear models might be easier to understand. CV might be used to evaluate

• the performance of different models
• the performance of different distributional assumptions
• feature engineering
• feature input

With such methods, after selecting a model and features going into the model based on CV, you need regression coefficients to be applied to the future data to make prediction.

Where are those coefficients coming from then?

One way is to apply the model to your entire dataset to obtain them.

If you are using training/validation/test splits, then I would normally treat training + validation as the full dataset, and the test split as future data to test on your final model.