# regression functions

Could somebody explain to me in simple terms what regression functions are? In bishop's pattern recognition it says

"Consider a pair of random variables θ and z governed by a joint distribution p(z, θ). The conditional expectation of z given θ defines a deterministic function

$$f(\theta )=\int zp(z|\theta)dz$$

Functions defined in this way are called regression functions."

can someone please expand on this?

• The regression function is important because if you want to approximate $z$ by a function $g(\theta)$, then choosing as $g$ the regression function, i.e., $\mathbb{E}[z|\theta]$, minimizes the squared loss. See for example this answer: stats.stackexchange.com/questions/176313/… where however $t$ corresponds to your $z$ and $x$ corresponds to your $\theta$. Jul 28, 2017 at 21:38

## Basics

Say you know $\theta = 3$. You want this to estimate $z$. You use the regression function to estimate $z$ as $$E[z|\theta=3] = f(3) = \int z p(z|\theta=3)dz.$$

Say the above equals five. Okay, then you guess that $z$ is 5, assuming you know $\theta =3$.

## General

A regression function gives you a number for $z$ for any input $\theta$ you have.

## Optimality

Well there are a lot of functions that can take $\theta$ and give you a guess for $z$. Why do we use the conditional expectation? Well because it's the "best" or "closest" in a sense. That's what DeltaIV is referring to.

Say you have your conditional expectation function $f(\theta)$. Is it better than any other function $g(\theta)$? Well each of them will be off by $Z-f(\theta)$ and $Z-g(\theta)$, respectively.

We want these numbers to be positive all the time, though, so we square them. That means we're looking at $(Z-f(\theta))^2$ and $(Z-g(\theta))^2$ now.

One more problem: these numbers are random. So we take the expectation of these random distances. $f$ is better than any other $g$ in the sense that

$$E[(Z-f(\theta))^2] \le E[(Z-g(\theta)^2].$$ For a proof, you can see the link DeltaIV posted.