I'm having trouble understanding something when going over Faster R-CNN paper (https://arxiv.org/pdf/1506.01497.pdf).
From what I understood, if k is the number of anchors per position on the feature map, RPN outputs a 4*k vector (for BB regression) and a 2*k vector (for obj/not obj classification) for each position on the feature map (so if i understood well the actual output is a 14x14x4*k and 14x14x2*k).
In the 3.1.3 Training RPNs section, it's said "we randomly sample 256 anchors in an image to compute the loss function of a mini-batch".
So my question is what are we suppose to do with anchors not used during training ? How not to take them into account while there is a value expected for them (class and reg)?