# What should we do when changing SGD optimizer to Adam optimizer?

Adam is one popular method of the optimization policies with adaptive learning rate. I'm focusing on a image segmentation project using fully convolutional networks. All weights were initialized by truncated normal distributions. Initially, I used the Adam optimizer, and got some convergence of loss on both training sets and test sets with reasonable accuracies (say 0.8). But when I tried to use the SGD optimizer, the loss seems converged, but the accuracy is nearly zero. So my question is, when we adopted different optimizers, what do we need to change for successful network training? Weight initialization?