I am trying to implement Region Proposal Networks, as have been described in the original Faster R-CNN paper. One thing which is not very clear is how to generate the region proposals (the anchors). So, for a single feature on the feature map, where the RPN module slides a small window, we assign $k$ anchor boxes of different sizes to this feature. My question is, to where exactly. The paper says:
An anchor is centered at the sliding window in question, and is associated with a scale and aspect ratio
This sounds very ambiguous to me. I think of two possible alternatives:
1) For each feature in the last feature map, we back-project the feature into the original image, so it corresponds to the center of its receptive field in the original input space . (We would require some receptive field arithmetic as described in https://medium.com/syncedreview/a-guide-to-receptive-field-arithmetic-for-convolutional-neural-networks-42f33d4378e0) So, we can directly compare the anchors with the ground truth boxes in the original input space, calculate IoUs and regress them. This looks to me as the correct way to do it, since the anchors can have sizes like $512^2$ which are too large for the last convolutional layer after numerous downsamplings.
2)For a given image, we project the ground truth boxes in the original image onto the last feature map. We execute regression here as well. This sounds to me more unlikely, since the anchor boxes will be way too large compared to the projected ground truth boxes in the final layer.
My question is which of these approaches are true for generating region proposals? If both are wrong, what is the correct way then?
