This is a hypothetical and pretty general question. Apologies if it is too vague. Suggestions on how to better focus it are welcome.
Suppose you are interested in the relationship between one endogenous variable $y$ and a few exogenous variables $x_1,...,x_k$. The ultimate goal is forecasting new realizations of $y$ given new realizations of $x$'s. You have little clue what functional form the relationship could take.
Suppose you have a sufficiently large sample, so that you may obtain a reasonably accurate estimate of the joint probability density (by kernel density estimation or similar) of $y$ and the $x$'s.
Then you could use
(A) kernel density estimation (or some similar alternative);
(B) machine learning techniques (penalized regression like LASSO, ridge, elastic net; random forests; other)
(There are certainly other alternatives, but including those would make the question way too wide.)
- When would you prefer A over B and when B over A?
- What would be the key determinants of the choice?
- What main trade-offs do we face?
Feel free to comment on special cases and add your own assumptions.