I understand why this metric is good for object detection tasks but for instance segmentation tasks it does not give any clue about the quality of the predicted masks. Shouldn't it be combined in some way with IOU between predicted and ground truth masks like it is done for semantic segmentation with mIOU ?
Key takeaway: modern datasets and instance segmentations use pixel-wise IOU for instance to instance overlap calculations during matching, essentially as you suggest above.
mIOU gives average intersection over union across all segmentation classes in a semantic segmentation problem, giving all classes equal weight importance.
mAP gives average accuracy of predicted object locations across all object predictions, matched to ground truth object predictions, and giving each object equal importance. (with a few caveats. First off, it is normally done separately per class so you get an mAP value for each class, and secondly, mAP is done using bounding boxes with a set threshold of overlap to be considered a match, though theoretically could be done with pixel-wise IoU for a finer analysis).
Reasons to use mAP:
If you have class imbalance where most of the instances belong to a single class, minority classes would be underweighted using mIOU, whereas mAP counts on a per object basis. EDIT ADDITION: While most evaluation challenges use class-balanced mAP, for your own training and evaluation you may not want to give each class equal importance in scoring because this will heavily weight classes with fewer examples. mAP can be used without equal class weighting, in which case each object will be given equal importance. If mIOU without class averaging were used instead of mAP, each pixel would be given the same importance. In many cases the latter may not be a desirable effect (e.g. one object class is generally larger, so would thus be given more weight by the metric).
Instance segmentations are often subsequently used for object detection tasks (as in Mask RCNN Paper), and in these cases it is nice to have a metric to compute their accuracy on this subsequent task. In the early days of instance segmentation (5 or so years ago) there likely weren't many datasets with instance segmentation labels so comparing against bounding box ground truth labels was more esesential.
Now, to get to the main answer to your question, the only difference between mAP for object detection and instance segmentation is that when calculating overlaps between predictions and ground truths, one uses the pixel-wise IOU rather than bounding box IOU. Recent instance segmentation papers have used this pixel-wise IOU rather than bounding box IOU (see this paper). You'd have to look at specific datasets and benchmarks to see which form of mAP they use for evaluation.