# When computing MLE for linear regression, where does the uncertainty come from?

In my Machine Learning course, when computing MLE for a linear regression problem, we modeled the likelihood function as a Gaussian. I have trouble understanding why. Where does the uncertainty come from? In my opinion, if our model is of the form $$w^T\phi(x)$$, then the likelihood should be $$p(\mathcal{D}|w)=\prod_{x,y\in\mathcal{D}}\mathbb{I}\{y=w^T\phi(x)\}$$, where $$\mathbb{I}$$ is the indicator function. In other words, I do not understand where the uncertainty comes from. Although I understand that using a "soft" function instead of a hard indicator has advantages (it is differentiable, and also $$\prod_{x,y\in\mathcal{D}}\mathbb{I}\{y=w^T\phi(x)\}=0$$ if only a single element of the product is 0), I still don't get why it is legit to model $$p(\mathcal{D}|w)$$ that way, because again in my mind the process of determining $$y$$ from $$w$$ and $$x$$ involves no uncertainty.

Furthermore, I understand there is some uncertainty within the parameter $$w$$, but still from my perspective the probability of $$(x,y)\in\mathcal{D}$$ given a certain value of $$w$$ should be either 1 or 0: either $$y=w^T\phi(x)$$ or not.

I understand that this misunderstanding probably stems from some fundamental concept I am not seeing or grasping. Can somebody help me understand where this uncertainty comes from?

• Well probably the noise was already there ... the model is not $y=w^\top\phi(x)$ but instead $y=w^\top\phi(x) + \epsilon$ with $\mathbb E[y \lvert x] = w^\top \phi(x)$. The noise has to be there because in practice no model is ever exact so the likelihood you suggest will just "always" be 0. But I am not sure this will answer your question since "where this uncertainty comes from?" is potentially a very philosophical question. – Jesper for President Nov 25 '19 at 14:48
• @JesperHybel hi, thanks for the comment. I am aware that is the model we use, with the $\epsilon$ term. But still I do not understand why we do model it as such. – olinarr Nov 25 '19 at 14:59
• 1) The world is random, 2) We can't model everything exactly. When you look at the data for your linear regression problem, does it in fact all fall exactly on a straight line? – jbowman Nov 25 '19 at 15:06
• @jbowman Oh, I think your last question may have made everything click into place. I was understanding "the model" as the concrete object (algorithm, neural network, ...) that maps inputs to outputs, which contains no uncertainty. Now I see that when we say "model" is a description of our data, which does in fact contain uncertainty. That was kinda silly of me! – olinarr Nov 25 '19 at 15:11
• Not really, we all have to make conceptual leaps at various times in our education :) – jbowman Nov 25 '19 at 16:09