# Model fitting VS model selection: what works best?

Suppose we have two candidate models to predict a variable $$y$$ given a variable $$x$$, where $$\alpha$$ is a model parameter.

$$\hat y=M_1(x,\alpha)$$ $$\hat y=M_2(x,\alpha)$$

Conceptually, we could combine them two into a single model, by introducing an additional parameter $$\beta$$.

$$M(x,\alpha,\beta)=\left\{\begin{array}{lr} M_1(x,\alpha), & \text{for } \beta=1 \\ M_2(x,\alpha), & \text{for } \beta=2 \\ \end{array}\right.$$

Now, what approach would produce the best out-of-sample predictor? Why?

Approach 1

• split the samples into in training and validation sets
• select the model that generalises best, out of $$M_1(x,\alpha)$$ and $$M_2(x,\alpha)$$
• fit the selected to the whole set of historical samples
• use the fitted model to predict

Approach 2

• fit the combined model $$M(x,\alpha,\beta)$$ to the historical samples
• use the fitted model to predict

thank you!

• How do you propose to learn the combined model? It seems that $\beta$ would be a hyperparameter and unlearnable in general. Unless you have a way around that, option B won’t work. Jun 20 at 15:44
• @Arya You can learn $\beta$ by doing grid search for example. The method you use to optimise a parameter (e.g. grid search, calculus, etc.) does not make it necessarily a parameter (which you would optimise with model fitting) versus an hyperparameter (which you would optimise with model selection). Does that make sense? Jun 21 at 5:53