2
$\begingroup$

I have the following linear regression model, $y = f(x;w) + n$, where $y$ is the vector of true labels, $x$ is the observed data, $f(x;w) = w^Tx$, and $n$ ~ $N(0, \sigma^2)$ is the noise. Why then does $p(y|x;w) = N(w^Tx;\sigma^2)$ (taken from page 12 of these notes: http://cs229.stanford.edu/notes/cs229-notes1.pdf)? Could someone please derive this for me?

$\endgroup$
3
  • 1
    $\begingroup$ Are you aware that adding a constant $\mu$ to a Normal$(0,\sigma^2)$ variable gives a Normal$(\mu,\sigma^2)$ variable? $\endgroup$
    – whuber
    Oct 16, 2014 at 2:50
  • $\begingroup$ Why is this true? $\endgroup$
    – eric
    Oct 16, 2014 at 2:57
  • $\begingroup$ For many people, it's the definition. One first defines a Normal$(0,1)$ variable (its probability density at any number $x$ is proportional to $\exp(-x^2/2)$) and then defines a Normal$(\mu,\sigma^2)$ variable to be a Normal$(0,1)$ variable that has been multiplied by $\sigma$ and had $\mu$ added to it. Evidently that is not your definition, so what is your definition? $\endgroup$
    – whuber
    Oct 16, 2014 at 3:00

2 Answers 2

3
$\begingroup$

I think whuber boils the questions down to its essentials with

adding a constant $μ$ to a Normal$(0,σ^2)$ variable gives a Normal$(μ,σ^2)$ variable

So how do we know this is true? How we show that depends on what you're prepared to start with. To talk about the normal at all we must begin with some kind of definition, so it depends on what you define the general normal to be. (As whuber points out, Wikipedia defines $N(\mu,\sigma^2)$ as a scaled, shifted standard normal, from which the pdf can be derived).

We can go the other way; we could begin with say a $N(0,\sigma^2)$ (your $n$ variable) and investigate the effect of adding $\mu$ to it. Here's the density of $n$:

$$\frac{1}{\sigma\sqrt{2\pi}}\, e^{-\frac{n^2}{2 \sigma^2}}$$

Let $Y=n+\mu$.

Now, how do we work out the density of the transformed (shifted) random variable? Again, it depends on what you're prepared to assume. It can be done from first principles ($F_Y(y)=P(Y\leq y)=P(n+\mu\leq y)=P(n\leq y-\mu)=F_n(y-\mu)$ and then differentiate $F_Y(y)$ to obtain the density $f_Y(y)$).

Or we could use the transformation result $f_Y(y) = f_X(g^{-1}(y))|\frac{d}{dy}g^{-1}(y)|$

So $n=Y-\mu$, hence $g=y-\mu$ and $dy = dn$

Hence $f_Y(y)=f_n(y-\mu).|1|$

$$=\frac{1}{\sigma\sqrt{2\pi}}\, e^{-\frac{(y-\mu)^2}{2 \sigma^2}}$$

which is a $N(\mu,\sigma^2)$

$\endgroup$
4
  • $\begingroup$ +1 I would like to suggest that a simpler and fully general way to show the effect of an affine transformation would be to contemplate the CDF rather than the PDF: when $Y=\sigma X+\mu$ (with $\sigma\gt 0$), then $F_Y(x)={\Pr}(Y\le x)={\Pr}(X\le (x-\mu)/\sigma)$, whence each "$x$" in the PDF (if it exists) must be replaced by "$(x-\mu)/\sigma$" and the $dx$ (which is implicitly in the PDF) must be replaced by $d((x-\mu)/\sigma)=dx/\sigma$. This doesn't really even require any knowledge of Calculus, but only of how to obtain slopes of lines. $\endgroup$
    – whuber
    Oct 16, 2014 at 14:30
  • $\begingroup$ Why is $N(w^Tx, \sigma^2)$ then set equal to $p(y|x;w)$? It seems just as reasonable to let $p(y,x;w) = N(w^Tx, \sigma^2)$. $\endgroup$
    – eric
    Oct 16, 2014 at 15:58
  • $\begingroup$ @whuber indeed so; via CDFs in essentially the way you did it is the way I prefer to look at it. I guessed that since the fact wasn't immediately obvious to the OP, that such an explanation might carry less impact than the pdf version that essentially derives from it. $\endgroup$
    – Glen_b
    Oct 16, 2014 at 21:40
  • $\begingroup$ @eric throughout the calculation, we're conditioning on $x$. The idea is to model the conditional expectation (we don't necessarily know the distribution of $x$, only the values we have -- it may not even be a random variable; it might be experimentally controlled, for example). $\endgroup$
    – Glen_b
    Oct 16, 2014 at 21:42
2
$\begingroup$

To complement what whuber said in the comments. You can do some simple manipulations to derive the distribution of $y^{(i)}$. In the regression equation:

$$y^{(i)} = \theta^T x^{(i)} + \epsilon^{(i)}$$

We isolate $\epsilon^{(i)} $ and than replace it in the density formula:

$$p(\epsilon^{(i)}) = \frac{1}{\sqrt{2\pi\sigma}}exp{\left(-\frac{(\epsilon^{(i)})^2}{2\sigma^2} \right)} $$

$$p(y^{(i)} - \theta^T x^{(i)}) = \frac{1}{\sqrt{2\pi\sigma}}exp{\left(-\frac{(y^{(i)} - \theta^T x^{(i)})^2}{2\sigma^2} \right)} $$

Then we rewrite the probability term in a more convenient way, considering $\theta$ as a fixed parameter and $x^{(i)}$ as a given variable. So

$$p(y^{(i)}| x^{(i)}; \theta) = \frac{1}{\sqrt{2\pi\sigma}}exp{\left(-\frac{(y^{(i)} - \theta^T x^{(i)})^2}{2\sigma^2} \right)} $$

$\endgroup$
4
  • 1
    $\begingroup$ +1 yours posted when I was almost finished composing mine. I guess mine shows some details the OP might seek, so I'll leave it there for now. $\endgroup$
    – Glen_b
    Oct 16, 2014 at 4:58
  • $\begingroup$ In your final equation, why is it correct to assume that $x^{(i)}$ is fixed? It seems more appropriate to define $p(y^{(i)}, x^{(i)};\theta) = N(\theta^Tx, \sigma^2)$. $\endgroup$
    – eric
    Oct 16, 2014 at 14:19
  • $\begingroup$ In order to write $y^{(i)} = \theta^T x^{(i)} + \epsilon^{(i)}$ you assumed that $x$ was your predictor variable (called the "input" variable in the notes), thus by definition it is a given variable, you know its value. $\endgroup$ Oct 16, 2014 at 16:41
  • 1
    $\begingroup$ Furthermore, you're not picking randomly a $(x, y)$ pair and measuring its probability. You're choosing instead a $x$ and then measuring the probability of the response variable $y$ taking a particular value. $\endgroup$ Oct 16, 2014 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.