I am training a network ESNet in Pytorch to predict vanishing point as per VPGNet ICCV 2017 paper. I am using SGD with 0.1 learning rate and ReducedLR scheduler with patience = 5.
Any idea what is happening?
Visually the network predicts nearly the same point in almost all the validation images.
PS: Y axis is loss and X is the epoch. The network is trained for 150 epochs.