# Non parametric estimation/regression for conditional distribution

Context : one continuous variable $Y$ dependent on $X$ ($X$ can be anything)

Linear regression, generalized linear model... focus on estimating the conditional expectation $E(Y\mid X)$. I want to focus on estimating the conditional distribution.

Linear regression can be seen as a parametric model for the conditional distribution: $Y\mid X\hookrightarrow \mathcal{N}(X\beta,\sigma^2)$ with parameter $\beta$. It assumes that given $X$, $Y$ has a normal distribution.

I'm looking for methods where the distribution of $Y$ given $X$ is not assumed to belong to a predefined family of distributions : just having a certain regularity and maybe vaguely a certain shape. How this distribution is assumed to change when $X$ changes is something that may be predefined or not. Of course this may require quite a lot of training data.

For non conditional distributions (just the distribution of $Y$, there is no $X$), I know a simple method : kernel density estimator.

What methods exist for the conditional case ?

• Lots of methods are consistent for a conditional mean, and given that conditional mean, further conditional moments can be looked-at. Can't you just take any consistent-for-the-mean estimator and look at its conditional higher moments via the residuals? – generic_user Jun 3 '17 at 1:14
• What kind of formula would you use for the moments ? For example, how would you expect the variance would depend on $X$ ? Linear ? Quadratic like $\sigma^2=(X\gamma)^2$ ? – Benoit Sanchez Jun 3 '17 at 13:25
• If you're prepared to assume a common distribution at each combination of predictors, you might fit a nonparametric regression first (e.g. local linear regression) and then try to fit a kernel density estimate to the residuals. – Glen_b Jun 4 '17 at 8:18
• I've heard of 3 approaches. Kernel estimator of conditional density: try npcdens in R's np package: cran.r-project.org/web/packages/np/np.pdf Or: use lots of (nonparametric) quantile regressions to to estimate the quantiles of the conditional distributions. Or: "distribution regression" as in Section 3 of arxiv.org/pdf/0904.0951 which also discusses/compares with quantile regression. – David M Kaplan Jun 6 '17 at 15:52
• Thanks David, this would be worth an answer :-) I've started reading. – Benoit Sanchez Jun 8 '17 at 11:19

In a time series context you can take a look at non-parametric predictive distribution estimation for $p(X_{t+1} \mid X_t, X_{t-1}, \ldots)$. For this I suggest to look at hard LICORS, mixed LICORS, LICORS cabinet, and Hilbert space embeddings of predictive state representations -- to name a couple (disclaimer: I have been working on the first 2.)
As a pure regression framework, I just finished work on "Predictive State Smoothing (PRESS): Scalable non-parametric regression for high-dimensional data with variable selection", which makes almost no assumptions on the distributions for $y \mid \mathbf{X}$ and estimates an optimal Kernel smoother. As others have already pointed to npreg or other kernel smoother implementations, I want to highlight that PRESS does not suffer from curse of dimensionality. In the paper I show examples of a kernel smoother for an $\mathbf{X} \in R^{60,000 \times 784}$, i.e., $784$ covariates with $N = 60,000$ observations. PRESS can estimate the kernel smoothing matrix for that within a couple of seconds. Traditional kernel smoothers break down (or just won't work) for more than a couple (5-ish) dimensions.