In every machine learning discussion, the term "model" is used to describe how the prediction is made. Does this "model" refer to the learning algorithm used? What exactly is a model?


Think of machine learning as learning formula for a task (say making a prediction). Model is then the unique formula. Learning algorithm is the procedure by which you arrive at the right formula.

If you are building a machine learning model of the form (linear regression),

Y = a0 * x0 + a1 * x1 + ... + an * xn

Then the linear formula above has free parameters a0 .. an. For a fixed value of all parameters, you have a unique formula. This unique formula/equation is the model. Model is something that you can save and use to make predictions on new inputs.

  • $\begingroup$ Can we say the Theta (or Hypothesis) which is used to get predicted y (for a given x) is the Model ? When algorithms like Decision Trees or Random Forest is used, they derive upon a Model based on their training set and this model is used to make the prediction for the unseen data. Is this understanding correct ? $\endgroup$
    – yathirigan
    Jan 4 '18 at 9:16
  • 1
    $\begingroup$ $\theta$ is not a model but the parameters of a (parametric) model. $\endgroup$
    – Lucas
    Jan 4 '18 at 9:34

The term "model" is generally used in one of two slightly different ways.

It can refer to a statistical model. Formally, this is a set of probability distributions. A "learning algorithm" tries to find a distribution from this set which matches the data best. There are different types of statistical models. Linear models and most neural networks correspond to parametric models. Here, each element of the set is identified by a vector of parameters.

$$\mathcal{P} = \{ p_\theta : \theta \in \Theta \}$$

A linear regression model could be written as

$$\mathcal{P} = \{ \mathcal{N}(y; \mathbf{a}^\top \mathbf{x} + b, 1) : \mathbf{a} \in \mathbb{R}^n, b \in \mathbb{R}\}.$$

In neural networks, the network's architecture defines the set of possible distributions, and the weights and biases identify distributions from that set.

Other types of statistical models are graphical models, in which the family of distributions is determined by a graph, or non-parametric models, in which the number of parameters is not fixed.

The other use of "model" in machine learning is to refer to a particular instance of a statistical model. I.e., instead of $\mathcal{P}$, "model" can refer to a particular $p_\theta$ where $\theta$ might be the parameter vector found by a learning algorithm.

It is also common to use terms to describe a combination of models and algorithms. For example, the term "linear regression" is generally understood to mean maximum likelihood estimation in the linear model. Another example would be variational autoencoders, which refers to variational inference in certain neural network based generative models.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.