After layerwise pretraining (typically unsupervised) the dnn, we tend to fine tune it in a supurvised manner. I know how to do the fine-tuning if the last layer is a softmax classifer, but how to do it if I have a Random Forest being the last layer? What is the loss function then and how to calculate the gradient w.r.t. the weights?


1 Answer 1


Due to the branching of a random forest, you have many discontinuities. More so, you have flat regions within each leaf. That means even if you can calculate the gradients, they will be zero everywhere except at the borders of partitions--where it is undefined.

That makes it somewhat impossible to use them as a top layer for deep nets and training with gradient-based techniques.

  • 1
    $\begingroup$ Thanks, @bayerj. So the features learnt from a dnn can only be feed into classifiers like softmax? $\endgroup$
    – avocado
    Oct 10, 2014 at 14:48
  • $\begingroup$ They can be fed into anything; but you cannot fine tune the DNN with gradient-based techniques on the loss of the RF. $\endgroup$
    – bayerj
    Oct 11, 2014 at 20:16
  • $\begingroup$ Yes, that's what I meant. $\endgroup$
    – avocado
    Oct 12, 2014 at 0:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.