Is there any algorithm which I can use to smooth the ROC curve and maybe another algorithm to fix points which are below the reference (chance) line?
Rufibach in arXiv:1103.1787 has, besides the description of a new smoothing method, a pretty good introduction with an overview of your options to smooth ROC curves. I suggest starting from there.
To "fix" the points below the diagonal you will need a more discriminative detector. You can't really fix bad data with statistics.