0
$\begingroup$

I'm dealing with a supervised regression problem, a financial data set with 5m observations and a few features (about 5 to 10). All the features are continuous numeric values, without any missing values. I am considering only non-linear methods for prediction, and NN and GBM are my two candidates.

However, I am considering GBM for reasons below. Please correct me what I misunderstood.

1) (Much) Easier to get result

  • It takes (much) less time and effort to "tune" the model. Yes, NN will do better in theory, but in practice I doubt myself getting there as a non-professional in NN tuning. it is highly likely for me to end up with local optimum, even after poking around tens of times with hyper-parameters set.

    • I already tried NN for a moment: a perturbation in learning rate immediately leads to prediction of mean value everywhere. Without perturbation it takes hours (if not days) to train, and that's where I stopped.

  • On the other hand, GBM is easier to train. It does have many hyper-parameters, but which can easily be "tuned" and not so sensitive to those values. For my problem, it works good enough even with default hyper-parameters.

2) A few number of features (<10)

  • By limiting a number of features, I am trading-off between predictability and interpretability. It's impossible to provide causal relationship for between each feature and y ex-ante.

  • I expect <10 features will be enough to predict y well enough. Hence interaction effects wouldn't be that severe and higher-order, so GBM will do fine and DL will be an overkill.

  • DL might try to search higher-order effects while there's not, so it may capture spurious interaction effects, leading to less accurate predictions.

3) Interpretability

  • This is ambiguous. Ensemble of trees may be seen as not-so-interpretable, unlike single tree.

  • Feature importance can also be measured for NN (say, SHAP values of Lundberg, Lee (2017)). I cannot come up with any interpretation methods that work only for GBM.

But, above are why GBM might do well, NOT why it might do BETTER than NN. For example, NN is "known" to perform bad with small number of observations. However, I am not sure whether I can say 5m obs is small or large. It looks large enough, but it seems that judgements on size are made ex-post, not ex-ante.

So, the problem is this. When asked by customer/supervisor "Why don't you do NN? It's the best method", I cannot just say "Well, it takes time and practically hard. I will just take an alternative approach". I should provide theoretical grounds why GBM is (or can be) better than NN in my application ex-ante. Before I do the project I should be able to anticipate it to persuade him/her.

However, I cannot come up with any reason why GBM should/might do better than NN. Nothing is given in comparative sense. To my understanding, NN should outperform any other models in theory. So, for a fair comparison, I should always include NN as I cannot rule out NN performing the best, right? (On the other hand, at least I can rule out linear regression if I expect highly non-linear relation).

Noisy data can make NN do worse, but noisiness are identified ex-post, not ex-ante.

In short, my question is: Can one justify using ML method in place of NN, in ex-ante? If NN didn't do better, how can one be sure that it's not his/her fault not torturing the model enough, but it's due to noisy data or others and conclude "GBM is the best method for this particular application"?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ 1) What theoretical results did you have in mind suggesting superior performance for neural nets? I'm not aware of any such results, and this sounds implausible to me. The issue is that performance is highly problem dependent, and the relevant properties of real-world problems tend not to be exactly known. 2) Ease of implementation is a perfectly justifiable factor to consider when choosing a method. The time you spend getting something to work has real costs of various kinds. $\endgroup$
    – user20160
    Oct 16, 2019 at 10:17
  • $\begingroup$ For 1), well I explicitly haven't read a "theory" that NN is superior in every aspect per se, but most of the articles talk about inferiority of all other models, but does not say much about what NN is incapable of. That's why I concluded that way. For 2), I personally agree with it, but I'm still not sure whether I can persuade this "fact" to non-ML guys. Anyway, I see you saying I cannot expect which model is superior before I actually do the job and compare the results. Thank you so much for the comment. $\endgroup$
    – crux26
    Oct 16, 2019 at 12:25
  • $\begingroup$ I agree with @user20160 that I am unaware of such results. That being said, if that case is just to blag people we can refer to the Kolmogorov-Arnold representation theorem for GBMs. It is as "useless" as the Universal approximation theorem in the context of NNs or the Stone-Weierstrass theorem in the context of polynomial regression. $\endgroup$
    – usεr11852
    Oct 16, 2019 at 15:16
  • $\begingroup$ @usεr11852 Thank you so much for bringing those theorems up. I've seen people "bragging" NN delivered with Universal approximation theorem, but I always thought it is meaningless in reality, where everything is obscure. So there's an equivalent version for the trees for blagging. So one can approximate true relation well enough with arbitraily many parameters, but the point is how well it works with given number of hyper-parameters bounded by practical reasons. And those "theoretical" relationships only state "existence", not "convergence speed" or "path". $\endgroup$
    – crux26
    Oct 17, 2019 at 4:29

1 Answer 1

Reset to default
3
$\begingroup$

Deep neural networks are very successful over a wide range of applications on raw and unstructured data, such as image pixels, signals, texts, etc. Their main advantage in these contexts is their ability to identify relevant features in the data. This task-oriented, data-driven feature identification usually outperforms generic hand-crafted feature extractors and models built on top of them.

In your case, you have a very few features (possibly hand-crafted to represent aspects of the data that a domain expert would consider important?), which takes away the main advantage of neural networks. I do not see a reason why they should perform considerably better in this context.

At the same time, as mentioned in the comments, neural networks require more engineering and parameter tuning, implying higher development costs.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you so much for the answer. Most of the things I've read was about the generic cases, and DNN talkings seem to be somewhat "tilted" toward non-tabular data, which is not of my interest. Let's say GBM, NN performs similar in tabular data but NN outperforms significantly in image case. Then it is not that NN not working well in tabular data: it is GBM not working well for image while NN is performing "similarly well" in image as well. This is because NN delivers "special" layers such as convolution layers to reflect the data characteristic whereas GBM hardly can model such characteristics. $\endgroup$
    – crux26
    Oct 17, 2019 at 4:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.