4
$\begingroup$

I am training my brain with machine learning concepts. I happen to read this post.

I understood the concept of mean, standard deviation, but unable to related with the below statement

"We normalize the input layer by adjusting and scaling the activations. For example, when we have features from 0 to 1 and some from 1 to 1000, we should normalize them to speed up learning"

Could someone help me understand better?

$\endgroup$
2
  • 2
    $\begingroup$ Mean and standard deviation don't really relate to normalisation. Normalisation is a way to bring data to a uniform scale and the author explains how it speeds up batch processing in the paragraph following the one you quoted. $\endgroup$
    – Zimano
    Jan 12, 2018 at 6:56
  • 2
    $\begingroup$ I think if you consider linear regression as your toy model, things will be easier to understand. Essentially gradient descent works badly on 'ravines', ie where the curvature of the error surface is wildly different in different directions... Curvature of error surface depends on variance of inputs, so normalising makes error surface have same curvature on each input direction ( but not diagonals). $\endgroup$
    – seanv507
    Jan 14, 2018 at 0:40

3 Answers 3

7
$\begingroup$

If you don't normalize the inputs, the gradient for weights related to one feature will be significantly larger than the gradient for the other feature during training,since gradients are dependent on the input.

This will hinder gradient descent algorithms from converging to a minima, as shown in below image. enter image description here

Normalization also allows you to use larger learning rates, as normalized inputs reduce the risk of exploding gradient.

$\endgroup$
1
  • $\begingroup$ This is a perfect reason on why to normalize data inputs, but how does this speed up training/convergence when we're still applying the same number of computations for the same number of epochs? $\endgroup$ May 30, 2021 at 5:12
5
$\begingroup$

Overall each training iteration will become slower because of the extra normalisation calculations during the forward pass and the additional hyperparameters to train during back propagation. However, training can be done fast as it should converge much more quickly, so training should be faster overall.

Few factors that influence faster training:

Allows higher learning rates  -  Gradient descent usually requires small learning rates for the network to converge, this is because of gradient vanishing problem. As networks get deeper, gradients get smaller during back propagation, and so require even more iterations to converse(gradient vanishing problem). Using batch normalisation allows much higher learning rates, increasing the speed at which networks train.

Makes weights easier to initialise -  Choice of initial weights are very important crucial and can also influence training time. Weight initialisation can be difficult, especially when creating deeper networks. Batch normalisation helps reduce the sensitivity to the initial starting weights.

Makes more activation functions viable -  Some activation functions don’t work well in certain situations.

$\endgroup$
1
1
$\begingroup$

Apart from the previous answers, I wanted to double check this behaviour. A simple linear regression was run using normalized data-points, and non-normalized. This is what you get, graphically, for normalized data over iterations: enter image description here And non normalized:

enter image description here

The $y$ axis contain huge values, and the reason is that the cost is blown up (values of the same order).

It's suspicious how one of the lines seems to be fitting the data, but this may be just smaller values of $W$ and it's flatten out by the plot. Weird enough, the training set ranges from $10$ to $40$, but the cost depends on the square, and the square of the square of $40$ is pretty large already.

I'm a beginner so can't say the code is correct. But is looks like it is for the first fit. The code was this:

# Linear Regression as described on the docs
import numpy as np 
import matplotlib.pyplot as plt

X = np.array([[10],[20],[30],[40]])
X = (X - np.mean(X))/(max(X)-min(X)) # comment for the non-normalized.
Y = np.array([[1.01,2.02,2.88,4.1]])

# forward propagation
def cost(X, Y, Yp):
    m = X.shape[1] 
    A = Y - Yp 
    cost = 1/m*np.dot(A,A.T).flatten()
    return A, cost

def predict(X,w,b):
    Yp = np.dot(w,X.T)+b # 1x4
    return Yp

def initialize(dim):
    # dim is X.shape[0]
    w = np.zeros((1, dim))
    b = 0
    return w, b

# backward propagation
def update(X,A,w,b,lr=0.01):
   m = X.shape[1]
   dw = 1/m*np.dot(2*A, X)*lr
   w = w + dw 
   b = b + np.sum(1/m*2*A*lr)
   return w, b

def model(X, Y, numIt):
    w,b = initialize(X.shape[1])
    plt.scatter(X.flatten(), Y.flatten(), label="original")
    plt.title("Fit line over cycles, features high")
    for i in range(numIt):
        Yp = predict(X,w,b)
        A, c = cost(X,Y,Yp)
        w,b = update(X,A,w,b,lr=0.01)
        if i%10==0:
            plt.plot(X.flatten(),Yp.flatten(), label="it: "+str(i))
    plt.legend(loc="best")
    plt.show()
    print("cost: ", c)
    return w, b

w, b = model(X,Y,50)
#print("printing w,b", w,b)
#X = [[1.9],[2.11]]
#Yp = np.dot(w,X)+b # 1x4
#print(Yp)
$\endgroup$
1
  • $\begingroup$ great job. And if you turn down the learning rate of the un normalized version, you will find it actually converges with a very slow speed $\endgroup$
    – hsc
    Apr 20, 2022 at 10:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.