Regression function of "non-regressible" data I have some background in probability, and now trying to understand statistics, which sometimes leads to the questions of the following kind. Let $X$ and $Y$ be two random variables that represent the data that we have in hand. Let's say we are interested in case when $X$ and $Y$ are not independent, and let $\mathbb P$ be their joint distribution. If we knew $\Bbb P$ we could deduce the marginal distribution of $X$, say $\mu(\mathrm dx)$ and a conditional distribution of $Y$ given $X$, say $m(x,\mathrm dy)$. In practice of course we do not know $\Bbb P$, rather we only have several samples from that distribution.
Let's consider the case when $Y$ is real-valued. As far as I understand, the regressiong function 
$$
  r(x):=\mathbb E[Y|X = x] = \int y \, m(x,\mathrm dy)
$$
is what we are often trying to find. In case $\Bbb P$ is given by $Y = f(X) + \varepsilon$ and $\varepsilon$ is some noise, of course finding $r \approx f$ would give us quite a lot information. However, if $Y$ is distributed (given $X$) not as a nice curve $\pm$ some noise, but in a more complex way, then I would not expect any regression to provide us relevant model, not matter whether it is simple linear regression or some more complicated non-linear method. For example, if $Y = \delta\cdot f_1(X) + (1-\delta)f_2(X)$ where $\delta$ follows symmetric Bernoulli distribution, then $r(x) = \frac12(f_1(x) + f_2(x))$ but it is gonna be quite a bad of an estimate of $Y$.
I thus wonder, whether instead of a regression function we can focus on estimating just conditional distribution of $Y$ given $X$, and if we can, why would not we do this all the time? Please tell me in case the question is not clear.
 A: I'm not entirely clear about what you're asking, but I think what you're after is non-parametric regression. The standard method is Nadaraya-Watson kernel regression, which is derived from the fact that $E[Y|X=x]=\int y f(y|x)dy = \int y \frac{f(y,x)}{f(x)}dy$. The numerator is a joint density, and the denominator is a density. These can be estimated using kernel density methods (basically smoothed histograms), yielding the estimator $$\hat{m}_h(x) = \frac{\sum K_h(x-x_i)y_i}{\sum K_h(x-x_i)} $$, where $K_h$ is a kernel with bandwidth $h$ (basically any probability density function can be used as a kernel). 
Anyway, this estimator is derived with no assumptions about the functional form of the relationship between $Y$ and $X$. So why don't we always use kernel regression? 
First, there's always a trade-off between assumptions and precision - the more structure I put on a problem, the more precise estimates I will be able to get with a given sample (but the risk of bias increases). Kernel regression has a slower rate of convergence than OLS, for example. 
Second, kernel regression quickly gets computationally expensive, especially as the number of independent variables increases. 
Third, kernel regression is harder to interpret - there will be a different marginal effect of each $X$ depending on the values of all the other $X$ variables. With a linear model, you get constant marginal effects.
A: Assessing the quality of predictors can be done only if we specify a criterion to use for the assessment (or a collection of criteria). So general statements like "...it is gonna be quite a bad of an estimate of $Y$", that are included in the OP's post are not really meaningful.
For example, by construction, the conditional expectation is such that
$$r(X)=\mathbb E[Y\mid X ] \Rightarrow Y = r(X) + u, \;\; E(u\mid X) =0$$
Not a bad property if you ask me (now, I am making statements that are not really meaningful, by silently assuming/implying a ton of things).
The conditional expectation is the best predictor in $L^2$ norm, and this has nothing to do with the specific functional form it may take.  
To quote from Williams' Probability with Martingales (1991, ch. 9.4, p. 85)

(...) No surprise then that conditional expectation (and the martingale
  theory which develops it) is crucial in filtering and control- of
  space-ships, of  industrial processes, or whatever.

ADDENDUM
Responding to the OP's comment-clarification below, to the question "why estimate conditional expectation and not conditional distribution function", the answer is easy: for a given set of available data, the more unknowns we want to estimate and combine, the worse our estimator's properties will be. Estimating a distribution is a much more delicate and uncertain endeavor (as I think I have written also elsewhere), than estimating an expected value.
A: There are two issues you should pay attention:


*

*We don't need the distribution of any kind for $Y$ - we only want to be able to predict $y$ values using given $x$. Many people how use linear regression don't derive it using statistics, rather they minimize $l_2$ norm of residuals - and achieve the same result. 

*In some cases we can't evaluate marginal distribution, while we are still able to calculate mean value of this distribution. See, for example, paper on heteroskedastic Gaussian processes regression http://www.icml-2011.org/papers/456_icmlpaper.pdf, at the end of the section 4: we can evaluate mean value of predictive distribution, but we can't calculate density for predictive distribution (as it is analytically intractable). 
