I'm having a hard time understanding some on the inner-working of YOLO, especially the loss function depicted in this seminal paper. Bear in mind that I'm nowhere closed to being a specialist in deep learning and computer vision (in fact, I started studying the subject two weeks ago, so I do have some shortcomings).
So what I've understood so far : we want to build a neural network that, given an image, predicts bounding boxes and classes probability for those bounding boxes. Therefore, given some input image, our network will output some tensor of size $S * S * (B * 5 + C)$ (see the paper mentioned above).
In order to do that, we need to train our network on some dataset (e.g. Pascal VOC for instance), backpropagating our loss measurement through the all network to optimize our weights through gradient descent (to put in nutshell : the usual stuff !).
The loss function given by Redmon et al. is the following : $\begin{align*} \color{blue}{\lambda_\textbf{coord} \sum_{i = 0}^{S^2} \sum_{j = 0}^{B} {\mathbb{1}}_{ij}^{\text{obj}} \left[ \left( x_i - \hat{x}_i \right)^2 + \left( y_i - \hat{y}_i \right)^2 \right] \\\\ + \lambda_\textbf{coord} \sum_{i = 0}^{S^2} \sum_{j = 0}^{B} {\mathbb{1}}_{ij}^{\text{obj}} \left[ \left( \sqrt{w_i} - \sqrt{\hat{w}_i} \right)^2 + \left( \sqrt{h_i} - \sqrt{\hat{h}_i} \right)^2 \right]} \\\color{green}{ + \sum_{i = 0}^{S^2} \sum_{j = 0}^{B} {\mathbb{1}}_{ij}^{\text{obj}} \left( C_i - \hat{C}_i \right)^2 \\ + \lambda_\textrm{noobj} \sum_{i = 0}^{S^2} \sum_{j = 0}^{B} {\mathbb{1}}_{ij}^{\text{noobj}} \left( C_i - \hat{C}_i \right)^2 \\} \color{red}{ + \sum_{i = 0}^{S^2} {{1}}_i^{\text{obj}} \sum_{c \in \textrm{classes}} \left( p_i(c) - \hat{p}_i(c) \right)^2} \end{align*} $
I've highlighted in blue the first part of our loss function, which we could called coordinate loss. I totally understand this part as it is quite intuitive and natural.
My problem is that I'm not sure to quite understand the last terms of this function. Here some of my guesses :
The red part : since our neural network should output classes probabilities $p(c_i | object)$ for bounding boxes, those $(p(c_i|object))_{i = 1 \dots C}$ should indeed be considered as parameters and therefore it should make sense that they appear in the loss function. So my question would be: since we are training on some labeled dataset, it means that $p_i(c)$ should be zero except for one class $c$, right ? (it is deterministic! )
The green part : I'm a little bit lost about that one. I mean, since $\widehat{C_i}$ is defined as the confidence of grid cell $i$, that is $\widehat{C_i} = Pr(object) \times IOU_{pred}^{truth}$, and $C_i$ is defined to be the 'real' confidence score (that is $IOU_{pred}^{truth}$), it seems silly as this everything is deterministic (we're on the training set), then we know whether or not there is an object in our grid cell, therefore $Pr(object) \in \{0,1\}$...I mean, I don't really know how to explain, but I'm confused with this loss function, especially with the real sense of confidence score in the model.
Thanks a lot for your help!
EDIT : maybe a first question would be : why are we interested in confidence score? At the end of the neural net, do we have some decision algorithm that says : if this bounding box as confidence above threshold $c_0$ then displays it and choose class with highest probability ?