# Do I apply normalization per entire dataset, per input vector or per feature?

One of the ways to standardize input data for Neural network training is:

\begin{equation} X = \frac{X - mean(X)}{std(X)} \end{equation}

However if I have have $n$ training examples which have each $m$ features: \begin{matrix} x_{11} & \ldots & a_{1m}\\ \vdots & \vdots & \vdots \\ x_{n1} & \ldots & a_{nn} \end{matrix}

But on what level do I apply this "mean to zero, variance to 1"?

What to apply, and what is the explanation? (I bet the answer is standardize across entire space!?)

• {Standardize across entire space} Calculate the mean/std (1 values) for the entire matrix and subtract/divide this element wise for each cell.

• {Standardize on row/input case level} Calculate the mean/std for each entire row, and subtract this element wise on each features in that row.

• {Standardize on column/feature basis} Calculate the mean/std for each entire column (feature), and subtract/divide this element wise all cells in that column.

This make a difference, If I have as input features with a different scale:

\begin{matrix} Feature 1: & Feature 2: & Feature 3:\\ case1: 100 & 0.1 & 5\\ case2: 150 & 0.9 & 2\\ \end{matrix}

Normalization across entire matrix will result in:

\begin{matrix}Feature 1: & Feature 2: & Feature 3:\\ Case1: 0.95363115 & -0.71773292 & -0.6357541 \\ case2: 1.79014971 & -0.70434862 & -0.68594522] \\ \end{matrix}

Normalization across each entire row (per case) will result in:

\begin{matrix}Feature 1: & Feature 2: & Feature 3:\\ Case1:1.41287463 & -0.75971915 & -0.65315549\\ Case2: 1.41418448 & -0.71494618 & -0.69923831]\\ \end{matrix}

Normalization across each entire column (per feature) will result in:

\begin{matrix}Feature 1: & Feature 2: & Feature 3:\\ Case1: -1 & -1 & 1\\ Case2: 1 & 1 & -1]\\ \end{matrix}

Python code to calculate examples:

import numpy as np
a = np.array([[100,0.1,5],
[150,0.9,2]])
print(a)
a-=a.mean()
a/=a.std()
print("Normalize over the entire matrix")
print(a)

c = np.array([[100,0.1,5],
[150,0.9,2]])

a = c
mean=a.mean()
std=a.std()
a -= mean
a /= std

mean=a.mean()
std=a.std()
a -= mean
a /= std

print("Normalize per input vector")
print(a)

a = c

mean=a[:,0].mean()
std=a[:,0].std()
a[:,0] -= mean
a[:,0] /= std

mean=a[:,1].mean()
std=a[:,1].std()
a[:,1] -= mean
a[:,1] /= std
print(a)

mean=a[:,2].mean()
std=a[:,2].std()
a[:,2] -= mean
a[:,2] /= std

print("Normalize per feature")
print(a)