How normalizing helps to increase the speed of the learning? I am training my brain with machine learning concepts. I happen to read this post.
I understood the concept of mean, standard deviation, but unable to related with the below statement
"We normalize the input layer by adjusting and scaling the activations. For example, when we have features from 0 to 1 and some from 1 to 1000, we should normalize them to speed up learning"
Could someone help me understand better?
 A: If you don't normalize the inputs, the gradient for weights related to one feature will be significantly larger than the gradient for the other feature during training,since gradients are dependent on the input.
This will hinder gradient descent algorithms from converging to a minima, as shown in below image.

Normalization also allows you to use larger learning rates, as normalized inputs reduce the risk of exploding gradient.
A: Overall each training iteration will become slower because of the extra normalisation calculations during the forward pass and the additional hyperparameters to train during back propagation. However, training can be done fast as it should converge much more quickly, so training should be faster overall.
Few factors that influence faster training:
Allows higher learning rates  -  Gradient descent usually requires small learning rates for the network to converge, this is because of gradient vanishing problem. As networks get deeper, gradients get smaller during back propagation, and so require even more iterations to converse(gradient vanishing problem). Using batch normalisation allows much higher learning rates, increasing the speed at which networks train.
Makes weights easier to initialise  -   Choice of initial weights are very important crucial and can also influence training time. Weight initialisation can be difficult, especially when creating deeper networks. Batch normalisation helps reduce the sensitivity to the initial starting weights.
Makes more activation functions viable  -  Some activation functions don’t work well in certain situations. 
A: Apart from the previous answers, I wanted to double check this behaviour. A simple linear regression was run using normalized data-points, and non-normalized. This is what you get, graphically, for normalized data over iterations:

And non normalized:

The $y$ axis contain huge values, and the reason is that the cost is blown up (values of the same order).
It's suspicious how one of the lines seems to be fitting the data, but this may be just smaller values of $W$ and it's flatten out by the plot. Weird enough, the training set ranges from $10$ to $40$, but the cost depends on the square, and the square of the square of $40$ is pretty large already.
I'm a beginner so can't say the code is correct. But is looks like it is for the first fit. The code was this:
# Linear Regression as described on the docs
import numpy as np 
import matplotlib.pyplot as plt

X = np.array([[10],[20],[30],[40]])
X = (X - np.mean(X))/(max(X)-min(X)) # comment for the non-normalized.
Y = np.array([[1.01,2.02,2.88,4.1]])

# forward propagation
def cost(X, Y, Yp):
    m = X.shape[1] 
    A = Y - Yp 
    cost = 1/m*np.dot(A,A.T).flatten()
    return A, cost

def predict(X,w,b):
    Yp = np.dot(w,X.T)+b # 1x4
    return Yp

def initialize(dim):
    # dim is X.shape[0]
    w = np.zeros((1, dim))
    b = 0
    return w, b

# backward propagation
def update(X,A,w,b,lr=0.01):
   m = X.shape[1]
   dw = 1/m*np.dot(2*A, X)*lr
   w = w + dw 
   b = b + np.sum(1/m*2*A*lr)
   return w, b

def model(X, Y, numIt):
    w,b = initialize(X.shape[1])
    plt.scatter(X.flatten(), Y.flatten(), label="original")
    plt.title("Fit line over cycles, features high")
    for i in range(numIt):
        Yp = predict(X,w,b)
        A, c = cost(X,Y,Yp)
        w,b = update(X,A,w,b,lr=0.01)
        if i%10==0:
            plt.plot(X.flatten(),Yp.flatten(), label="it: "+str(i))
    plt.legend(loc="best")
    plt.show()
    print("cost: ", c)
    return w, b

w, b = model(X,Y,50)
#print("printing w,b", w,b)
#X = [[1.9],[2.11]]
#Yp = np.dot(w,X)+b # 1x4
#print(Yp)

