I want to do bbox-regression as in the rcnn paper which comes down to predicting scalar values by minimizing a smoothed L1 loss.
I have to predict relatively small offsets (a fex pixels) in around 20000 patches containing centered objects or their shifted versions. Girshick's parametrization is: $(offset_{x},offset_{y})=(\frac{\Delta_{x}}{width},\frac{\Delta_{y}}{height})$
So if my patches are 45x45 and my offsets range from -5 to 5 pixels in x and y. I get as a max value for the offset to predict $\frac{5}{45}=0.1111$.
I used different kind of normalizations for the offsets like dividing by 5 to have a maximum at 1, etc. but it did not change the outcome.
I first experimented with IMAGENET descriptor+a layer of linear regression as in the original paper (like AlexNet fc6) but I could not make it converge to a satisfying solution with SGD or even using an exact linear solver. I figured maybe it was because of the pooling layers which destroyed the translation information so the descriptors are in fact not that separable but the explanation is not entirely satisfying because by resizing my patches to be 227x227 I magnified the apparent shift too by a factor of 5.
I am now trying to learn this offsets with relatively shallow convnets WITHOUT pooling or with only one layer of pooling with SGD but it seems the optimization is still difficult and I get stuck on relatively high values even in the training set. Even when augmenting the number of fully connected layers or depth. Of course I am using all the known "tricks" that work well with cross-entropy loss like He initialization, ReLU, Adam optimizer and so forth.
Did anyone experience the same things when doing a simple L2/L1 regression on images ? Do you have any trick that proved to work ? Any references ? Is it because the magnitude of the shifts is too low ?
EDIT: Karpathy in CS231 mentions this difficulty and proposes to transform the problem into a classification problem. However that is not always feasible/practical.
EDIT 2 This recent article on arXiv Designing Deep Convolutional Neural Networks for Continuous Object Orientation Estimation mentions a regression problem where it seems the most successfull approach is to convert it to a classification problem first but they are avoiding the issue. It seems they have experimented with regression too but they do not provide enough technical details on the difficulty of optimizing for regression.
EDIT 3 I am aware that you cannot relu the output of your networks obviously...
