4
$\begingroup$

Context : one continuous variable $Y$ dependent on $X$ ($X$ can be anything)

Linear regression, generalized linear model... focus on estimating the conditional expectation $E(Y\mid X)$. I want to focus on estimating the conditional distribution.

Linear regression can be seen as a parametric model for the conditional distribution: $Y\mid X\hookrightarrow \mathcal{N}(X\beta,\sigma^2)$ with parameter $\beta$. It assumes that given $X$, $Y$ has a normal distribution.

I'm looking for methods where the distribution of $Y$ given $X$ is not assumed to belong to a predefined family of distributions : just having a certain regularity and maybe vaguely a certain shape. How this distribution is assumed to change when $X$ changes is something that may be predefined or not. Of course this may require quite a lot of training data.

For non conditional distributions (just the distribution of $Y$, there is no $X$), I know a simple method : kernel density estimator.

What methods exist for the conditional case ?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Lots of methods are consistent for a conditional mean, and given that conditional mean, further conditional moments can be looked-at. Can't you just take any consistent-for-the-mean estimator and look at its conditional higher moments via the residuals? $\endgroup$
    – generic_user
    Jun 3, 2017 at 1:14
  • $\begingroup$ What kind of formula would you use for the moments ? For example, how would you expect the variance would depend on $X$ ? Linear ? Quadratic like $\sigma^2=(X\gamma)^2$ ? $\endgroup$
    – Benoit Sanchez
    Jun 3, 2017 at 13:25
  • 1
    $\begingroup$ If you're prepared to assume a common distribution at each combination of predictors, you might fit a nonparametric regression first (e.g. local linear regression) and then try to fit a kernel density estimate to the residuals. $\endgroup$
    – Glen_b
    Jun 4, 2017 at 8:18
  • 1
    $\begingroup$ I've heard of 3 approaches. Kernel estimator of conditional density: try npcdens in R's np package: cran.r-project.org/web/packages/np/np.pdf Or: use lots of (nonparametric) quantile regressions to to estimate the quantiles of the conditional distributions. Or: "distribution regression" as in Section 3 of arxiv.org/pdf/0904.0951 which also discusses/compares with quantile regression. $\endgroup$
    – David M Kaplan
    Jun 6, 2017 at 15:52
  • $\begingroup$ Thanks David, this would be worth an answer :-) I've started reading. $\endgroup$
    – Benoit Sanchez
    Jun 8, 2017 at 11:19

1 Answer 1

Reset to default
3
$\begingroup$

In a time series context you can take a look at non-parametric predictive distribution estimation for $p(X_{t+1} \mid X_t, X_{t-1}, \ldots)$. For this I suggest to look at hard LICORS, mixed LICORS, LICORS cabinet, and Hilbert space embeddings of predictive state representations -- to name a couple (disclaimer: I have been working on the first 2.)

As a pure regression framework, I just finished work on "Predictive State Smoothing (PRESS): Scalable non-parametric regression for high-dimensional data with variable selection", which makes almost no assumptions on the distributions for $y \mid \mathbf{X}$ and estimates an optimal Kernel smoother. As others have already pointed to npreg or other kernel smoother implementations, I want to highlight that PRESS does not suffer from curse of dimensionality. In the paper I show examples of a kernel smoother for an $\mathbf{X} \in R^{60,000 \times 784}$, i.e., $784$ covariates with $N = 60,000$ observations. PRESS can estimate the kernel smoothing matrix for that within a couple of seconds. Traditional kernel smoothers break down (or just won't work) for more than a couple (5-ish) dimensions.

Improve this answer
$\endgroup$
1
  • $\begingroup$ OK, those are some great acronyms. $\endgroup$
    – naught101
    Jan 29, 2020 at 1:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.