Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

When we fit a generalized linear regression (e.g., logistic regression, gamma regression) we are estimating the population average Y given the predictors $X$ ( i.e., $E(Y | X)$ ).

When we fit a machine learning model such as an ANN, SVM, or a decision tree, does this notion still apply? In other words, are we estimating the population average value of $Y$ or isn't that idea applicable and we are just predicting "Y"?

ADD After Dikran response:

I. What aspect of the theory of a predictive modeling algorithm tells us that we are modeling E(Y|X) versus just Y|X ? Is it the use of an error term that follows a certain distribution? For instance, what is it about ANN versus a decision tree which tells us the former models E(Y|X) while the latter is modeling Y|X?

II. Is there any connection between these and say a confidence interval versus prediction interval in linear regression?

share|improve this question
Imo one can't answer a question like "is all of stats or ML this or that way" in a meaningful way other than by saying "No" (although the type of question pops up regularly). It depends on the technique, not the field. GLM, trees, SVM and ANN are different. Take your question: The $E(Y|X)$ part only makes sense for some probability measure to relate the expectation to, so if the technique has that then maybe, else no. SVM for example can be constructed completely devoid of probability. – Momo Dec 25 '12 at 1:54
Ok so to enumerate, I was interested in 1) gradient boosted regression trees, 2) svm and 3) artificial neural networks specifically. So is the deciding factor to my question , if the technique is derived in a probabilistic manner or not? – B_Miner Dec 25 '12 at 5:07
@B_Miner I know this is an old questions but did you figure this out? I'm currently thinking about confidence intervals and prediction intervals for ML predictions (GBM and RF in particular) and first I need to figure out the more fundamental questions. Hoping you're able to share some of your learnings. – organic agave May 24 at 19:38
up vote 5 down vote accepted

There are many machine learning methods that do aim to estimate the conditional mean of the data, such as artificial neural networks, but also there are many that do not (such as SVMs, decision trees etc.). The motivation of the SVM is that it is better to solve the particular problem at hand directly, rather than solve a more general problem and simplify the result. So if you are only interested in a hard binary classification, in principle that ought to be easier than estimating the a-posteriori probability of class membership and then thresholding at 0.5. Whether that is true in practice is debatable, but also in my experience in practice you often do want the a-posteriori probabilities becase training set and operational class frequencies are different or variable, or equivalently the misclassification costs are not known at training time or are variable, or you need a reject option etc. So whether a particular method estimates the conditional mean of the response variable depends on what task the method intended to solve.

Note for the SVM there is an alternative that does estimate the conditional mean of the data, namely kernel logistic regression for classification and kernel ridge regression for regression problems.

The loss function that is minimised has a lot to do with whether the model predicts the conditional mean of the response variable, pretty much any method that minimises a sum of squared error loss (or cross-entropy for classification) will have this property, see e.g.

Saerens, M., "Building cost functions minimizing to some summary statistics", IEEE Transactions on Neural Networks, volume: 11 , issue: 6, pages 1263 - 1271, 2000.

share|improve this answer
Hi Dikran, I added an addition to my original question based on your response. – B_Miner Dec 25 '12 at 22:32
Thanks for the additional info! I am hoping someone can show an example - I don't have the intuition as to why the loss function being minimized tells us that what is being estimated is E(Y|X). (also have not yet found a free access to this article). But if the presence of a loss function minimization during training is what counts, then would gradient boosted regression trees (GBM) constitute estimating the average value of Y (although a decision tree alone does not - but GBM does minimize a loss function). – B_Miner Dec 26 '12 at 18:42

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.