How much is ROC biased towards the minority class? It's known that ROC is overly optimistic in case of imbalanced data sets. How big can this bias be? For example if I read paper where they report 0.75 ROC on a dataset with 5 percent of samples being form the minority class, how would the ROC change if the dataset is balanced? 
 A: One of the features of ROC curves is that they are insensitive to changes in class distribution (see here).  To answer your question, the ROC curve would not change much, or not change at all, if the dataset is balanced.
The ingredients of a ROC curve are true positive rate = TP/P (# positives correctly classified / total positives in dataset) and false positive rate FP/N (# negatives incorrectly classified / total negatives).  Imagine if you balance the dataset, for example, by cutting N in half.  Certainly TP/P doesn't change at all.  FP/N might not change at all, and probably won't change much.  When you eliminate negative cases, you'll probably eliminate properly classified and improperly classified cases without prejudice.
I'm guessing the "ROC overly optimistic for imbalanced dataset" comment has its roots in this paper.  There are some good points and findings in this paper about the relationship between ROC and PR curves.  However I don't believe the authors have properly justified their claim of over-optimism.  They show that a PR curve has more space in the optimal area than the ROC for a specific example.  So what? "Over optimistic" to me is a "bad" word, meaning the performance is worse than the diagnostic states.  I'm pretty sure the FPRs and TPRs are what the ROC curves says they are.  In any case, they are not going to change much by balancing the dataset.
A: Probably difficult to answer without the data.  Sensitivity and specificity do a very good job when there are small class sizes.  
Think of this example.  Assume you have 100 patients, and 5 are diseased and 95 are normal.  If the classifier is a "shoebox" and does nothing and assigns normal to all 100 patients, the predictive accuracy of the classifier is still 95%, since only 5/100 are misclassified.  But if you apply sens, spec, they will be terribly low.  Thus, sens, spec can catch cases like this.  
