Basically, it's an efficient implementation of the gradient descent algorithm. Gradient of a function points towards the steepest increase direction. So, the inverse of it is the steepest decrease direction. By moving towards the negative gradient (with a suitable learning rate), your cost function will decrease. If we were maximizing maximizing, we'd go ...
The weights can resemble some random distribution which we avoided in
the first place. How do you guarantee the weights are always
distributed according to Xavier/He initialization after
You don't. As long as the weights start in the right place, the rest of training usually goes pretty well in terms of the gradient dynamics.
I can assure you one thing after reading this blog you will have all your questions answered just read it religiously.This blog has the whole implementation from scratch in python just using numpy.