Understanding probabilistic view of linear regression I am trying to get into some machine learning and am not very good at math.
Could someone please help me with the following queston?
I am reading this script that explains the basic ideas behind probabilistic ML:
https://www.cl.cam.ac.uk/teaching/1617/MLBayInfer/mlbi-1.pdf
On page 41, the author explains how he generated the data for the linear regression examples and tries to show, I think, how this can be viewed as a probabilistic process.
I quite simply don't understand how steps 2 and 3 give you $p(y_i|\vec{x_i},\vec{w})$
It says:
For the ith example:

*

*I sampled $x_i$ according to the uniform density on [0, 3]. So there is a distribution
$p(\vec{x})$.

*I computed the value $h_{\vec{w}}(\vec{x}_i)$.

*I sampled $\epsilon_i ∼ N (0, σ^2)$ with $σ^2 = 0.1$ and formed $y_i = h_{\vec{w}}(\vec{x}_i)+\epsilon_i$
And therfore $p(y_i|\vec{x_i},\vec{w}) = N(h_{\vec{w}}(\vec{x}_i), \sigma^2)$
I kinda understand that the rgression line is the expected value of the normal distribution at that point... but I don't quite understand how the normal distribution arises from steps 2 and 3.
EDIT
Thanks for the comments.
I'll definitely check out the book and youtube videos.
I think mhdadk's comment touches on where I have difficulties.
So okay, If I add a constant k to a normal distribution $N(0,1)$, it becomes $N(k,1)$.
But how exactly is $y_i = h_{\vec{w}}(\vec{x}_i)+\epsilon_i$ the cond. distribution of $y$ given $x$ and $w$?  In this scenario, both $x$ and $e$ are random variables, right?
When I see a conditional distribution $P(A|B,C)$, I think of $\frac{P(A,B,C)} {P(B,C)}$
So does that mean that $N(h_{\vec{w}}(\vec{x}_i), \sigma^2)$ is somehow the result of $\frac{P(y,x,w)}{P(x,w)}$ ?
 A: There are two sets of potentially confounded issues here with respect to conditional probabilities and likelihoods. One has to do with the construction of a data set $s$ to illustrate approaches to regression. The second has to do with the analysis of that data set.
Constructing the data set
With respect to constructing the data set $s$:

does that mean that $N(h_{\vec{w}}(\vec{x}_i), \sigma^2)$ is somehow the result of $\frac{P(y,x,w)}{P(x,w)}$ ?

Fundamentally yes, but I'd suggest that you think of conditional probability in a more formal way that directly relates to how the data set was constructed. Quoting from Wikipedia on conditional probability:

Formally, $P(A | B)$ is defined as the probability of A according to a new probability function on the sample space, such that outcomes not in B have probability 0 and that it is consistent with all original probability measures.

Starting from that formal definition the Wikipedia entry shows how to derive the way that you defined conditional probability,
$$P(A|B) = \frac{P(A \cap B)}{P(B)}, $$
where I've used the $A \cap B$ symbol to avoid potential ambiguity in use of commas.
This definition of conditional probability in terms of an event $B$ such that all other outcomes have probability 0 leads directly to how the data set was constructed.
First, the author chose a specific instance of the vector of parameter values $\vec{w}$ that determines the shape of the function $h_{\vec{w}}(x)$ that in turn defines the mean value of $y$. All other potential values of $\vec{w}$ have probability 0.
Second, the author chose a set of individual values of $x_i$. With the fixed $\vec{w}$ and a single choice of $x_i$, one thus temporarily defines "a new probability function on the sample space" in which the event $(\vec{w},x_i)$ has probability 1 and all other outcomes have probability 0.
The mean value of $Y$ given the event $(\vec{w},x_i)$ then is simply $h_{\vec{w}}(x_i)$. Sampling a value $\epsilon_i$ from a $N(0,\sigma^2)$ distribution and adding it to that mean value gives an instance $y_i$ distributed as $N(h_{\vec{w}}(x_i),\sigma^2)$.
In your terminology, you might think of this as representing $\frac{P(y,x,w)}{P(x,w)}$ but in a restricted space for each choice of $x_i$ in which there is only one value of $(x,w)$ having nonzero probability.
As it turns out, the author chose to sample multiple scalar values $x_i$ from a uniform distribution over [0,3]. Going back to your way of thinking about conditional probability, you could think of the entire process as leading to a sample from $$P(y,x,\vec{w}) = P(y|x,\vec{w}) P(x,\vec{w})$$ consistent with the fixed $\vec{w}$ and that distribution of $X$.

In this scenario, both $x$ and $e$ are random variables, right?

Yes, insofar as in constructing the data set $s$ you have sampled $x_i$ from a random variable $X$ that is uniformly distributed over [0,3] and have sampled $\epsilon_i$ from a distribution of $N(0,\sigma^2)$.
Analyzing the data set
When it comes to analyzing the constructed data set $s$, however, the distribution of $X$ is of at most secondary interest. The point of the exercise, from page 39 of the linked resource, is this:

Remember: you don’t know what $w$ is: you need to identify it by analysing $s$.

That's where likelihoods come in. To estimate $\vec{w}$ you find the value that maximizes the likelihood of the data set $s$. That directly leads to a relationship between that optimal estimate of $\vec{w}$ and the conditional probabilities $P(y_i|x_i,\vec{w})$, as adapted from page 45 of that resource:
$$\vec{w}_{opt} = \underset{\vec{w}}{\operatorname{argmax}}  \sum_{i=1}^m \log P(y_i|x_i,\vec{w}) $$
The distribution of the $x_i$ doesn't enter into the optimization itself. The resource then goes on to show how a Bayesian approach to regression treats the set of parameter values $\vec{w}$ as a random variable in its own right with a distribution to estimate, not some "true" point value that we are trying to estimate in a frequentist approach.
A: Usually, we tend to think about linear regression as outputting a single number, that is (hopefully) close to the real label. The optimization process try to make these single number predictions as close as possible to the true values (labels).
Here the author suggests a different view. Instead of a single number, the output is modeled by a normal distribution. The "regular" output of the linear regression $h(w, x_i)$ will be now the mean of the normal distribution, but we also have noise around it. In this specific example the $\sigma$ is constant, but you can think about cases in which each example will have its own $\sigma_i$.
The author created a dataset, in which each example is normally distributed, with mean of the output of the linear regression and variance of $\sigma^2$.
The way he did it is by generating a normal distributed number $\epsilon_i$ and add it to the linear regression output, $h(w, x_i)$. The addition of normally distributed random variable to a constant is also normally distributed variable.
Once we have a distribution (probabilistic view) we can ask how probable a value $y_i$ given the current linear regression weight and features $x_i$.
This can also be our loss function. If the value has high probability (i.e. it is close to the mean, which is linear regression output) we will have low loss because this value is probable under our model.
