In my Machine Learning course, when computing MLE for a linear regression problem, we modeled the likelihood function as a Gaussian. I have trouble understanding why. Where does the uncertainty come from? In my opinion, if our model is of the form $w^T\phi(x)$, then the likelihood should be $p(\mathcal{D}|w)=\prod_{x,y\in\mathcal{D}}\mathbb{I}\{y=w^T\phi(x)\}$, where $\mathbb{I}$ is the indicator function. In other words, I do not understand where the uncertainty comes from. Although I understand that using a "soft" function instead of a hard indicator has advantages (it is differentiable, and also $\prod_{x,y\in\mathcal{D}}\mathbb{I}\{y=w^T\phi(x)\}=0$ if only a single element of the product is 0), I still don't get why it is legit to model $p(\mathcal{D}|w)$ that way, because again in my mind the process of determining $y$ from $w$ and $x$ involves no uncertainty.

Furthermore, I understand there is some uncertainty within the parameter $w$, but still from my perspective the probability of $(x,y)\in\mathcal{D}$ given a certain value of $w$ should be either 1 or 0: either $y=w^T\phi(x)$ or not.

I understand that this misunderstanding probably stems from some fundamental concept I am not seeing or grasping. Can somebody help me understand where this uncertainty comes from?

  • 2
    $\begingroup$ Well probably the noise was already there ... the model is not $y=w^\top\phi(x)$ but instead $y=w^\top\phi(x) + \epsilon$ with $\mathbb E[y \lvert x] = w^\top \phi(x)$. The noise has to be there because in practice no model is ever exact so the likelihood you suggest will just "always" be 0. But I am not sure this will answer your question since "where this uncertainty comes from?" is potentially a very philosophical question. $\endgroup$ – Jesper for President Nov 25 '19 at 14:48
  • $\begingroup$ @JesperHybel hi, thanks for the comment. I am aware that is the model we use, with the $\epsilon$ term. But still I do not understand why we do model it as such. $\endgroup$ – olinarr Nov 25 '19 at 14:59
  • 2
    $\begingroup$ 1) The world is random, 2) We can't model everything exactly. When you look at the data for your linear regression problem, does it in fact all fall exactly on a straight line? $\endgroup$ – jbowman Nov 25 '19 at 15:06
  • 2
    $\begingroup$ @jbowman Oh, I think your last question may have made everything click into place. I was understanding "the model" as the concrete object (algorithm, neural network, ...) that maps inputs to outputs, which contains no uncertainty. Now I see that when we say "model" is a description of our data, which does in fact contain uncertainty. That was kinda silly of me! $\endgroup$ – olinarr Nov 25 '19 at 15:11
  • 3
    $\begingroup$ Not really, we all have to make conceptual leaps at various times in our education :) $\endgroup$ – jbowman Nov 25 '19 at 16:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.