As in the paper I can understand SSD try to predict object locations and their relevant class scores from different feature maps .

So for each layers there can be different predictions with respect to number of anchor(reference) boxes in different scale.

So if one convolutional feature map has 5 reference boxes there should be class scores and bbx coordinates for each of the reference box .

We do above predictions by sliding a window(kernel Ex: 3*3) over the feature maps of different layers . So what I not clear is connection from sliding window at a position to score layer .

1. It just connection of convolution window output to score layer in a fully connected way ? 2.Or we do some other operation for convolution window output before connecting it to score layer ?