So I have a net which is working pretty well(93%+ on the validation set which is the state of the art[https://yoniker.github.io/]) on some problem.
I want to squeeze even more performance out of it, so I intentionally took examples it misclassified (I thought that those examples will get it closer to the true hypothesis as the gradient is proportional to the loss which is higher for mispredicted examples,and the "price" in terms of time of getting those kind of examples is almost the same as getting any example,mispredicted or not).
- What hyperparameters (learning rate in particular) should I use when it comes to the new examples? (the gradient is bigger so the ones which i previously found are not working anymore).
- Should I search again for new hyperparameters for the 'new' problem (training more a trained net)?
- Should I use the previous examples as well?
- If so, what should be the ratio between the 'old' examples and the 'new' ones?
- Are there known and proved methods for this particular situation?