Should predictive tasks always be regression instead of classification? "Any predictive classification task can be formulated as a regression task with a threshold, and we should use the regression because it gives you freedom to choose the threshold and/or compare the most likely answer." 
Is this correct? Is there any advantage in using e.g. Classification trees instead of a regression tree and deciding the threshold yourself? 
 A: The first statement seems to make sense because when Y is categorical, we link a decision threshold ($Y = 1$ when $Y > Y^*$) and a regression ($Y^*=X+e$). And we can tune the threshold $Y^*$ to improve the prediction accuracy. Logistic regression works exactly in this way.
When you apply tree-based models, you would primarily focus on tuning the so-called "hyperparameters", such as the maximum depth and splitting criteria of the tree model. Most importantly, tree-based models work differently from regressions. In this sense, a regression tree is a type of tree model (when $Y$ is a continuous variable), which still splits the data using Xs as the splitting nodes at each level, and the $\hat{Y}$ (predicted value) will the $\bar{Y}$ (group average) of observations at an ending node.
A: Perhaps you mean binary classification, to do this for multi-class classification would not be straightforward. 
The answer is no. Formulating as a classification task in general will optimise separability, while regression will optimise line fit. Think about a simple SVM classifier as an example - the decision boundary will be based on the observations nearest to the boundary, while regression will have to model all the observations.
Another argument - if you have different costs of missclassification for different classes, then formulating the task as a classification task can optimise for minimising those costs directly, while formulation as a regression will optimise a non-cost sensitive fit, and it would not be straightforward to include discrete costs into the optimisation criteria.
A: I think there is a split between MLers (classifier) and statisticians ([logistic] regression). [I don't think regression trees are good probabilistic classifiers] - I am on the logistic regressin side
I would say from a business perspective I would agree with that statement ( you don't want to retrain the model everytime the business changes their threshold....  
Its easy to get a classification from a probabilistic classifier, but getting probabilities from a random forest etc is not. all the methods suggested are basically hacks (see eg https://people.dsv.su.se/~henke/papers/bostrom08b.pdf). In addition these non probabilistic models have problems with typical real world problem of 'rare' classes (do I have cancer?) so it is suggested to eg undersample your data etc..
However, regression trees are not trained to output probabilities ( they use some other metric AFAIK) so I wouldn't recommend them.  In addition trees make the assumption of full 'non linearity', so you will need many more splits  to achieve a continuous output - as opposed to a classifier tree, which in the limit only has to split the data into 2 groups. so if you want to use trees then a classifier is likely to work better than regression tree; if you are not tied to trees then I would suggest logistic/multinomial regression (with nonlinear inputs) or NNs.
