I'm dealing with a supervised regression problem, a financial data set with 5m observations and a few features (about 5 to 10). All the features are continuous numeric values, without any missing values. I am considering only non-linear methods for prediction, and NN and GBM are my two candidates.
However, I am considering GBM for reasons below. Please correct me what I misunderstood.
1) (Much) Easier to get result
It takes (much) less time and effort to "tune" the model. Yes, NN will do better in theory, but in practice I doubt myself getting there as a non-professional in NN tuning. it is highly likely for me to end up with local optimum, even after poking around tens of times with hyper-parameters set.
- I already tried NN for a moment: a perturbation in learning rate immediately leads to prediction of mean value everywhere. Without perturbation it takes hours (if not days) to train, and that's where I stopped.
On the other hand, GBM is easier to train. It does have many hyper-parameters, but which can easily be "tuned" and not so sensitive to those values. For my problem, it works good enough even with default hyper-parameters.
2) A few number of features (<10)
By limiting a number of features, I am trading-off between predictability and interpretability. It's impossible to provide causal relationship for between each feature and
I expect <10 features will be enough to predict y well enough. Hence interaction effects wouldn't be that severe and higher-order, so GBM will do fine and DL will be an overkill.
DL might try to search higher-order effects while there's not, so it may capture spurious interaction effects, leading to less accurate predictions.
This is ambiguous. Ensemble of trees may be seen as not-so-interpretable, unlike single tree.
Feature importance can also be measured for NN (say, SHAP values of Lundberg, Lee (2017)). I cannot come up with any interpretation methods that work only for GBM.
But, above are why GBM might do well, NOT why it might do BETTER than NN. For example, NN is "known" to perform bad with small number of observations. However, I am not sure whether I can say 5m obs is small or large. It looks large enough, but it seems that judgements on size are made ex-post, not ex-ante.
So, the problem is this. When asked by customer/supervisor "Why don't you do NN? It's the best method", I cannot just say "Well, it takes time and practically hard. I will just take an alternative approach". I should provide theoretical grounds why GBM is (or can be) better than NN in my application ex-ante. Before I do the project I should be able to anticipate it to persuade him/her.
However, I cannot come up with any reason why GBM should/might do better than NN. Nothing is given in comparative sense. To my understanding, NN should outperform any other models in theory. So, for a fair comparison, I should always include NN as I cannot rule out NN performing the best, right? (On the other hand, at least I can rule out linear regression if I expect highly non-linear relation).
Noisy data can make NN do worse, but noisiness are identified ex-post, not ex-ante.
In short, my question is: Can one justify using ML method in place of NN, in ex-ante? If NN didn't do better, how can one be sure that it's not his/her fault not torturing the model enough, but it's due to noisy data or others and conclude "GBM is the best method for this particular application"?