Could somebody explain to me in simple terms what regression functions are? In bishop's pattern recognition it says

"Consider a pair of random variables θ and z governed by a joint distribution p(z, θ). The conditional expectation of z given θ defines a deterministic function

$$f(\theta )=\int zp(z|\theta)dz $$

Functions defined in this way are called regression functions."

can someone please expand on this?

  • $\begingroup$ The regression function is important because if you want to approximate $z$ by a function $g(\theta)$, then choosing as $g$ the regression function, i.e., $\mathbb{E}[z|\theta]$, minimizes the squared loss. See for example this answer: stats.stackexchange.com/questions/176313/… where however $t$ corresponds to your $z$ and $x$ corresponds to your $\theta$. $\endgroup$
    – DeltaIV
    Jul 28, 2017 at 21:38

1 Answer 1



Say you know $\theta = 3$. You want this to estimate $z$. You use the regression function to estimate $z$ as $$ E[z|\theta=3] = f(3) = \int z p(z|\theta=3)dz. $$

Say the above equals five. Okay, then you guess that $z$ is 5, assuming you know $\theta =3$.


A regression function gives you a number for $z$ for any input $\theta$ you have.


Well there are a lot of functions that can take $\theta$ and give you a guess for $z$. Why do we use the conditional expectation? Well because it's the "best" or "closest" in a sense. That's what DeltaIV is referring to.

Say you have your conditional expectation function $f(\theta)$. Is it better than any other function $g(\theta)$? Well each of them will be off by $Z-f(\theta)$ and $Z-g(\theta)$, respectively.

We want these numbers to be positive all the time, though, so we square them. That means we're looking at $(Z-f(\theta))^2$ and $(Z-g(\theta))^2$ now.

One more problem: these numbers are random. So we take the expectation of these random distances. $f$ is better than any other $g$ in the sense that

$$ E[(Z-f(\theta))^2] \le E[(Z-g(\theta)^2]. $$ For a proof, you can see the link DeltaIV posted.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.