Suppose we have two candidate models to predict a variable $y$ given a variable $x$, where $\alpha$ is a model parameter.

$$\hat y=M_1(x,\alpha)$$ $$\hat y=M_2(x,\alpha)$$

Conceptually, we could combine them two into a single model, by introducing an additional parameter $\beta$.

$$M(x,\alpha,\beta)=\left\{\begin{array}{lr} M_1(x,\alpha), & \text{for } \beta=1 \\ M_2(x,\alpha), & \text{for } \beta=2 \\ \end{array}\right.$$

Now, what approach would produce the best out-of-sample predictor? Why?

Approach 1

  • split the samples into in training and validation sets
  • select the model that generalises best, out of $M_1(x,\alpha)$ and $M_2(x,\alpha)$
  • fit the selected to the whole set of historical samples
  • use the fitted model to predict

Approach 2

  • fit the combined model $M(x,\alpha,\beta)$ to the historical samples
  • use the fitted model to predict

thank you!

  • 1
    $\begingroup$ How do you propose to learn the combined model? It seems that $\beta$ would be a hyperparameter and unlearnable in general. Unless you have a way around that, option B won’t work. $\endgroup$ Jun 20 at 15:44
  • $\begingroup$ @Arya You can learn $\beta$ by doing grid search for example. The method you use to optimise a parameter (e.g. grid search, calculus, etc.) does not make it necessarily a parameter (which you would optimise with model fitting) versus an hyperparameter (which you would optimise with model selection). Does that make sense? $\endgroup$
    – elemolotiv
    Jun 21 at 5:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.