So I have a net which is working pretty well(93%+ on the validation set which is the state of the art[https://yoniker.github.io/]) on some problem.

I want to squeeze even more performance out of it, so I intentionally took examples it misclassified (I thought that those examples will get it closer to the true hypothesis as the gradient is proportional to the loss which is higher for mispredicted examples,and the "price" in terms of time of getting those kind of examples is almost the same as getting any example,mispredicted or not).

  • What hyperparameters (learning rate in particular) should I use when it comes to the new examples? (the gradient is bigger so the ones which i previously found are not working anymore).
  • Should I search again for new hyperparameters for the 'new' problem (training more a trained net)?
  • Should I use the previous examples as well?
  • If so, what should be the ratio between the 'old' examples and the 'new' ones?
  • Are there known and proved methods for this particular situation?

Focusing in the samples in which the classifier performs less is very common in boosting. It is usually being done by training one classifier, seeing were it doesn't perform well and train another classifier focusing there and so on. If you will use boosting you will enjoy proven techniques and plenty of implementations.

Do you want to do it without iterations in order to get a single net? If so, most boosting are linear composition of the classifiers so you can merge them this way to a single one. You can integrate them, by really combining the networks (e.g., like in Graph-Based Algorithms for Boolean Function Manipulation) but I guess that you would like a quicker solution.

Please note that modifying the weights of the samples, based on a classifier, relay on that classifier. You will still need the classification of the first one. Other than that, the new classifier won't be trained on your natural distribution (were your samples come from and were you will be evaluated), so this direction is risky.

  • 1
    $\begingroup$ +1. There is a huge literature on boosting with different base learners. I'd recommend that the OP read one of these tutorials. $\endgroup$ – Stephan Kolassa Oct 16 '17 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.