In section 1.4.5 of Kevin Murphy ML textbook, he introduces linear regression where for a given data $x$, the target $y$ is assumed to be obtained through

$$y(x) = w^Tx + \epsilon, \text{ where } \epsilon \sim \mathcal{N}(\mu, \sigma^2)$$

Since $\epsilon$ is a random variable, therefore $y$ is a random variable as well induced solely by $\epsilon$.

However, the author then defines $$p(y|x, \theta) = \mathcal{N}(y|\mu(x), \sigma^2(x))$$

where $\theta = (w, \sigma^2)$

First of all, $x$ here is explicitly just a vector with no assumption that is was generated according to a distribution. Secondly, $\theta$ here is just the parameters associated with the Gaussian as well as the model, there is no probablistic interpretation as well.

How does the author condition on two deterministic variables?

Relevant text below:

enter image description here

  • $\begingroup$ The author doesn't condition: this is merely an abuse of notation. The two other variables are parameters. $\endgroup$
    – whuber
    Jul 24 '20 at 13:29
  • $\begingroup$ @whuber But this doesn't make sense if you consider the parallel development for logistic regression. It is always $P(y|x)$ conditioned on $x$, web.stanford.edu/class/archive/cs/cs109/cs109.1166/pdfs/… $\endgroup$
    – Norman
    Jul 24 '20 at 19:32
  • $\begingroup$ It is not. In logistic regression there is no need to assume $x$ is a random variable. One proof of that is that there is no probability (or density) associated with $x$ in the expression for the likelihood given in the "Derivations" section. $\endgroup$
    – whuber
    Jul 24 '20 at 19:35

If $x$ is not stochastic, then $y$ is just an affine transformation of the multivariate normal variable $\epsilon$.

When $x$ is stochastic, no distributional assumption is needed. All that is required is that the conditional distribution of $y$ given $x$ is normal and the marginal distribution of $x$ is unrestricted (see Hansen, Econometrics, p. 141).

As to $\theta$, in a frequentist setting parameters are just unknown numbers, and one would rather write $p(y;x,\theta)$, but in a bayesian setting parameters are random variables. And Murphy is going to introduce Bayesian concept learning...

  • $\begingroup$ The first characterization is deceiving, because which transformation $y$ is depends on $x.$ And it's an affine transformation: the linear part merely is $\epsilon$ itself. $\endgroup$
    – whuber
    Jul 24 '20 at 14:01
  • $\begingroup$ @whuber Yes, "affine" is more technical. I was just saying that if $x$ is non stochastic, then $y=w^Tx+\epsilon$ is like the familiar "linear" (more technically: affine) transformation $Y=a+bX$. $\endgroup$
    – Sergio
    Jul 24 '20 at 14:08
  • $\begingroup$ The concern isn't linear vs affine; it's that even if we use affine, it's not the case that "$y$ is just a [linear or affine] transformation of a normal variable." $\endgroup$
    – whuber
    Jul 24 '20 at 16:35
  • $\begingroup$ Please, could you explain this point? $\endgroup$
    – Sergio
    Jul 24 '20 at 16:39
  • 1
    $\begingroup$ I've edited my answer. As to how $y$ (a random variable) is related to $x$ (when a non stochastic variable) another time :) Thanks! $\endgroup$
    – Sergio
    Jul 24 '20 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.