This is a hypothetical and pretty general question. Apologies if it is too vague. Suggestions on how to better focus it are welcome.

Suppose you are interested in the relationship between one endogenous variable $y$ and a few exogenous variables $x_1,...,x_k$. The ultimate goal is forecasting new realizations of $y$ given new realizations of $x$'s. You have little clue what functional form the relationship could take.

Suppose you have a sufficiently large sample, so that you may obtain a reasonably accurate estimate of the joint probability density (by kernel density estimation or similar) of $y$ and the $x$'s.

Then you could use
(A) kernel density estimation (or some similar alternative);
(B) machine learning techniques (penalized regression like LASSO, ridge, elastic net; random forests; other)

(There are certainly other alternatives, but including those would make the question way too wide.)


  1. When would you prefer A over B and when B over A?
  2. What would be the key determinants of the choice?
  3. What main trade-offs do we face?

Feel free to comment on special cases and add your own assumptions.

  • 3
    $\begingroup$ the main determining factor is the value of $k$ $\endgroup$
    – user603
    Jan 17 '15 at 0:23
  • $\begingroup$ @user603: What is the reason? Do you mean that sample size has to increase exponentially with respect to $k$ to maintain roughly the same precision of kernel density estimation? Note that I indicated in the question (paragraph 3) that you have sufficiently large sample size so that KDE is reasonably accurate. $\endgroup$ Jan 17 '15 at 10:02
  • $\begingroup$ @RichardHardy I m not sure at which level of abstraction you want an answer. Very soon, as $k$ becomes large, the sample size and computational costs of KDE makes it infeasible. Another problem is that for $k>2$ you cannot visualize the KDE fit, so that makes it impractical. $\endgroup$
    – user603
    Jan 17 '15 at 11:31

(First off, I'd consider kernel density estimation a form of a machine learning model, so that's a strange dichotomy to make. But anyway.)

If you really do have enough samples to do good density estimation, then the Bayes classifier formed via KDE, or its regression analogue the Nadaraya-Watson model, converges to the optimal model. Any drawbacks of this approach are then purely computational. (Naive KDE requires comparing each test point with every single training point, though you can get much better than that if you're clever.) The other problem is the enormous issue of bandwidth selection, but with a good enough training set this is again only a computational issue.

In practice, however, you rarely actually have a good enough sample to perform highly accurate density estimation. Some issues:

  • As the dimension increases, KDE rapidly needs many more samples; vanilla KDE is rarely useful beyond the order of 10 dimensions.
  • Even in low dimensions, a density estimation-based model has essentially no ability to generalize; if your test set has any examples outside the support of your training distribution, you're likely screwed.

The reason for this drawbacks is that density estimation-type models assume only that the function being learned is fairly smooth (with respect to the kernel). Other models, by making stronger assumptions, can learn with many fewer training points when the assumptions are reasonably well-met. If you think it's likely that the function you're trying to learn is more or less a sparse linear function of its inputs, then LASSO will be much better at learning that model with a given number of samples than KDE. But if it turns out to be $f(x) = \begin{cases} 1 & \lVert x \rVert > 1\\0 & \text{otherwise}\end{cases}$, LASSO will do essentially nothing and KDE will learn more or less the right model pretty quickly.

  • $\begingroup$ though you can get much better than that if you're clever do you have a reference handy? Do you have a reference where KDE is used on a 10 dimensional data-set? I have not paid interest to KDE for years so the developments you mention are impressive to me! $\endgroup$
    – user603
    Jan 17 '15 at 11:32
  • 1
    $\begingroup$ @user603 By that, I meant that you can use approximate nearest-neighbor indices to accelerate the kernel lookip: if most points are outside the support of your kernel (or so far away that they contribute almost nothing), you don't need to compute them. Here is an example; it's also implemented eg in scikit-learn's KernelDensity class. $\endgroup$
    – Danica
    Jan 17 '15 at 17:31
  • 1
    $\begingroup$ @user603 I mentioned 10 dimensions as more or less an upper bound; successful applications there are rare. But this 1991 article proposes that it shouldn't be too bad. And we got good classification results in 50 dimensions using something more-or-less based on a $k$-NN density estimator. In practice, you usually need to turn to sparse graphical model estimators like forest density estimation or the nonparanormal, or parametric models for good density estimation at that point. $\endgroup$
    – Danica
    Jan 17 '15 at 17:42

From your statement,

The ultimate goal is forecasting new realizations of $y$ given new realizations of $x$'s.

, this already suggests that you want to do regression. I would go for (B). That is to estimate $\mathbb{E}[y | x]$. I am not so sure what you plan to use KDE on. I definitely will not use it to model the density of $x$ because that is not needed. Your goal is to predict $y$ given $x$. There is no need to care about the density of $x$. Perhaps you mean to use KDE to somehow estimate the conditional density $p(y|x)$. But then again, this is overkill because presumably $\mathbb{E}[y | x]$ should be enough for predicting $y$. Estimating $\mathbb{E}[y | x]$ is an easier problem compared to estimating $p(y|x)$. The methods you mention in (B) are for estimating $\mathbb{E}[y | x]$.

  • $\begingroup$ Thanks, good point! Your answer helps me understand my own question better :) Indeed, if the ultimate goal is prediction given new realizations of $x$ then I would be only interested in $p(y|x)$. (While the joint density of $y$ and $x$'s could potentially be useful for other tasks than prediction.) Being able to predict the conditional density instead of just the conditional mean might occasionally be useful (e.g. when the loss function is asymmetric) and then KDE would help more than just $\mathbb{E}[y|x]$ from some method in (B). $\endgroup$ Jan 17 '15 at 11:42
  • $\begingroup$ Right. $p(y|x)$ contains more information than just $\mathbb{E}[y|x]$. In most cases, you may want the predictive variance $\mathbb{V}[y|x]$. If that is the case, it might be easier to assume some noise model on $y$ (usually Gaussian is asssumed), and derive the variance. Still easier than estimating $p(y|x)$. But you may have more reasons for $p(y|x)$. Go for it if you need. :) $\endgroup$
    – wij
    Jan 17 '15 at 11:46
  • $\begingroup$ This is the general argument in favor of discriminative rather than generative learning. It's a strong argument, but sometimes generative approaches have their own strengths. In any case, conditional KDE is also a thing. $\endgroup$
    – Danica
    Jan 17 '15 at 17:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.