# What are "Grids" and Detection at different scales" in YOLOV3?

I've recently started working with Yolov3 and the more I go in depth, the more confused I get. In the simplest terms what I think about YOLOV3 (On 416 input, 80 classes, 3 BB) is that:

1. It Extract features from DarkNet-53 and then again use 53 Convolution networks throughout.
2. At 3 different levels (at layers number: 82,96,106), there are feature maps created of size :13,26,52 respectively with same depth 256. The features at 82 are upscaled till layer 96 and features from 96 upscaled till 106.( **What are these? Feature Map size?)

What I want to know from now here is that :

what are Grid cells? Are these pixels or slices of image?

Are there 3 different outputs predicted (one for each of the 3 layers) and total loss is calculated. If so? How is this achieved? Because we input 1 image with annotations? How would that give us Ground BB for the downscaled images?

OR Is the size same at each level? IF so, what are we up sampling?

In the end, How is the total loss calculated?

Apart from reading the original paper, I have referred to many blogs and videos but I could not get the meaning of how exactly works. Could someone please explain in simple terms of how an example would go in and get predicted.

• I edited my answer, hope it helps Feb 10, 2021 at 17:53
• Thanks a lot buddy! Feb 11, 2021 at 10:56

Regarding the first comment of the question:

1. It Extract features from DarkNet-53 and then again use 53 Convolution networks throughout.

The network used to extract features is just DarkNet-53 as stated by the author in the YOLOv3 paper:

We use a new network for performing feature extraction [...]
It has 53 convolutional layers so we call it.... wait for it..... Darknet-53!

Regarding the second comment of the question:

1. At 3 different levels (at layers number: 82,96,106), there are feature maps created of size: 13, 26, 52 respectively with same depth 256. The features at 82 are upscaled till layer 96 and features from 96 upscaled till 106. (What are these? Feature Map size?)

Yes, the sizes of 13, 26 and 52, refer to the feature map dimensions at the prediction layers. These sizes are directly related to the grid cells as we are going to see in the following lines. Just a sidenote, the correct depth (number of feature maps) is 255 and not 256 (this will also be explained hereafter).

## Grid cells

Grid cells were first explained in the original paper You Only Look Once. They are part of a regular grid (of simensions $$S\times S$$) into wich the input image is divided. Concretely, in YOLOv1 the author uses a grid of 7x7 cells. This is an example:

Images extracted from YOLO CVPR slides

The purpose of this type of grid (which is mantained in YOLOv3) is to serve as the output of the architecture $$\Rightarrow$$ $$B$$ bounding boxes are being predicted at each of the grid cells. Specifically, in YOLOv3, $$B=3 \Rightarrow$$ We are predicting 3 bounding boxes at each grid cell.

But how can this be encoded in the network?

The authors used what you mentioned in the question i.e. 3 different prediction layers/ grids composed of $$13\times 13, 26\times 26$$ and $$52\times 52$$ grid cells.

Now, we should remember that $$B=3$$ bounding boxes are being predicted at each of these grid cells. These boxes are encoded as a vector of $$85$$ features/ dimensions $$\Rightarrow$$ The depth (number of feature maps) needed is $$85\times B = 255$$, as you said in the question.

To visualize this, first we can focus on how a certain box of a certain cell is encoded:

• $$\mathbf{C}$$ : Is the objecteness of the associated bounding box (sort of a confidence measure of how likely a box is containing, or not, an object).
• $$\mathbf{(t_x, t_y)}$$ : Features related to the offsets of the bounding box center w.r.t. a corner of the grid cell.
• $$\mathbf{(t_w, t_h)}$$ : Features related to the dimensions of the bounding box.
• $$\mathbf{p_1,...,p_{80}}$$ : Conditional probabilities, $$P(\text{Class}| \text{Object})$$ , for each of the $$80$$ classes being considered in the paper.

By doing this we end up predicting a tensor of dimensions $$S\times S\times B \times 85$$, which represents the output of the architecture, as commented above:

However, we are predicting 3 bounding boxes for each grid cell $$\Rightarrow$$ the total output is the result of concatenating three of these tensors together, making a dimensions total of $$S\times S \times 85 \times 3 = S\times S \times 255$$.

Moreover, as you said in the question, YOLOv3 uses three of these regular grids as outputs $$\Rightarrow$$ We are predicting 3 of these $$S_i\times S_i \times 255$$ tensors at 3 different depths of the architecture.

## Predictions and training with YOLOv3

Due to the 3 different grid resolutions $$(S_i = \{13, 26, 52\})$$, we will be predicting outputs at 3 different depths of the architecture. This is shown in the next image extracted from the great post "What’s new in YOLO v3?":

At these 3 different layers (outputs), the same object can be detected, however this is not very likely to happen. This is because, as said by the authors, each of these regular grids uses anchor boxes with different sizes $$\Rightarrow$$ The lowest resolution grid ($$S=13$$) uses bigger anchor boxes, and the highest resolution grid ($$S=52$$) uses smaller anchor boxes.

The purpose of this, is to be able of detecting smaller objects which are difficult to detect in a low resolution grid. This way, in order to detect bigger objects at he highest resolution grid, the difference w.r.t. these anchor boxes would be significant, difficulting the learning procedure of the network.

However, having a look at the label generation for VOC dataset:

Now we need to generate the label files that Darknet uses. Darknet wants a .txt file for each image with a line for each ground truth object in the image that looks like:

<object-class> <x> <y> <width> <height>

Where x, y, width, and height are relative to the image's width and height

There isn't distinction of scale for the ground-truth objects. This means that the network will be learning to predict the ground-truth bounding boxes at the 3 different grids $$\Rightarrow$$ A ground-truth object will have influence at the loss function of all 3 output grids.

This influence in learning is by backpropagation of the gradients of the 3 loss function w.r.t. the weights/ biases. One loss function for each output grid. However, note that, for example, not all the weigths/ biases will have an influence in the first and second grid loss functions.

This is because, for example, a change in the weights and biases values of one of the final layers won't affect the predictions of the first output grid $$\Rightarrow$$ The associated gradient will be zero.

• What a thoughtful answer man. Appreciate it. Feb 11, 2021 at 10:57
• One more question. If we have 3 different detections at 3 levels, how do we interpret results? Let us suppose an object A was detected in all of the 3 stages, object B in 2 and object c in just  stage, how do we differentiate / normalise or combine the results in the very end? Jun 14, 2021 at 11:35
• @Deshwal That can be dealt via Non-Maximum Suppresion (NMS). Its goal is to filter redundant bounding boxes that have significant IOU with the correct bounding box (BB). As it's stated in the link, this is done by getting the most confident BB of a certain class that has significant IOU with others of the same class. An official implementation can be viewed at: github.com/pjreddie/darknet/blob/master/python/darknet.py Jun 14, 2021 at 16:26
• But NMS~ works on the same layer and same grid. I am asking about different level of detection. How would the model know that it's the same object or not? See this elaborate questuion Jun 15, 2021 at 2:34
• @Deshwal NMS is applied globally. Suppose you have multiple detected BBs in different layers for the same object $\to$ This lead to a global image where all these detections are checked. Note that different grids have differente sizes, but they correspond to the same image $\to$ We don't get as output 3 different images, we get one $\to$ All detections are related to the same image. Jun 15, 2021 at 9:34