6
$\begingroup$

There is significant literature on using deep nets on classification problems. I'm currently trying to apply a deep learning to a regression problem, and getting mediocre results compared to tree based methods. I'm assuming that theoretically, a deep net should offer similar if not superior performance if the correct architecture, activation functions, etc. are specified. Is there any relevant literature, or practical advice available for deep learning as it applies to regression?

Edit Just to give some context, I'm talking about high-dimensional data with mixed types (categorical, continuous) and where the features may be obfuscated, and the goal is to build the most accurate model according to the RMSE on a holdout set. Think benchmarks, machine learning contests, etc.

Improve this question
$\endgroup$
4
  • $\begingroup$ I don't know of any great examples on regression problems. Speech Synthesis is one example, but DNNs aren't SotA for that, yet. $\endgroup$
    – capybaralet
    Commented Aug 6, 2015 at 2:28
  • 3
    $\begingroup$ I am curious about what body of "theory" you are invoking to conclude that deep nets are "superior" in regression--and superior to what method, exactly, and according to what metrics and for what purposes? $\endgroup$
    – whuber ♦
    Commented Aug 6, 2015 at 2:29
  • $\begingroup$ No, I'm not concluding that. I haven't been able to find any literature on deep nets for regression. It's more of a conjecture that eventually they will provide the SoA results on high-dimensional regression benchmarks. Think machine learning contests, where you're trying a black-box approach. I would of course not use deep nets for regression for simpler problems where domain knowledge abounds, and interpretation is important. $\endgroup$
    – Ben Rollert
    Commented Aug 6, 2015 at 2:37
  • $\begingroup$ Do you realize that classification is a regression problem? $\endgroup$
    – Frans Rodenburg
    Commented Nov 20, 2017 at 8:40

1 Answer 1

Reset to default
4
$\begingroup$

Deep neural networks such as CNNs or RNN/LSTMs are only truly effective if the data has a structure that can be exploited in the modeling. For instance, this can be spatial (pixels in images) or sequential (words in a sentence). If indeed your data has some underlying structure that is not captured effectively with the current feature-based models, then deep learning might very well give state-of-the-art results on both classification and regression tasks. However, based on your question it seems that your data does not have this kind of structure, and I would therefore stay with tree-based models if I were you.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.