12
$\begingroup$

One of the ways to standardize input data for Neural network training is:

\begin{equation} X = \frac{X - mean(X)}{std(X)} \end{equation}

However if I have have $n$ training examples which have each $m$ features: \begin{matrix} x_{11} & \ldots & a_{1m}\\ \vdots & \vdots & \vdots \\ x_{n1} & \ldots & a_{nn} \end{matrix}

But on what level do I apply this "mean to zero, variance to 1"?

What to apply, and what is the explanation? (I bet the answer is standardize across entire space!?)

  • {Standardize across entire space} Calculate the mean/std (1 values) for the entire matrix and subtract/divide this element wise for each cell.

  • {Standardize on row/input case level} Calculate the mean/std for each entire row, and subtract this element wise on each features in that row.

  • {Standardize on column/feature basis} Calculate the mean/std for each entire column (feature), and subtract/divide this element wise all cells in that column.

This make a difference, If I have as input features with a different scale:

\begin{matrix} Feature 1: & Feature 2: & Feature 3:\\ case1: 100 & 0.1 & 5\\ case2: 150 & 0.9 & 2\\ \end{matrix}

Normalization across entire matrix will result in:

\begin{matrix}Feature 1: & Feature 2: & Feature 3:\\ Case1: 0.95363115 & -0.71773292 & -0.6357541 \\ case2: 1.79014971 & -0.70434862 & -0.68594522] \\ \end{matrix}

Normalization across each entire row (per case) will result in:

\begin{matrix}Feature 1: & Feature 2: & Feature 3:\\ Case1:1.41287463 & -0.75971915 & -0.65315549\\ Case2: 1.41418448 & -0.71494618 & -0.69923831]\\ \end{matrix}

Normalization across each entire column (per feature) will result in:

\begin{matrix}Feature 1: & Feature 2: & Feature 3:\\ Case1: -1 & -1 & 1\\ Case2: 1 & 1 & -1]\\ \end{matrix}

Python code to calculate examples:

import numpy as np
a = np.array([[100,0.1,5],
              [150,0.9,2]])
print(a)
a-=a.mean()
a/=a.std()
print("Normalize over the entire matrix")
print(a)

c = np.array([[100,0.1,5],
              [150,0.9,2]])

a = c
mean=a[0].mean()
std=a[0].std()
a[0] -= mean
a[0] /= std

mean=a[1].mean()
std=a[1].std()
a[1] -= mean
a[1] /= std

print("Normalize per input vector")
print(a)

a = c

mean=a[:,0].mean()
std=a[:,0].std()
a[:,0] -= mean
a[:,0] /= std

mean=a[:,1].mean()
std=a[:,1].std()
a[:,1] -= mean
a[:,1] /= std
print(a)

mean=a[:,2].mean()
std=a[:,2].std()
a[:,2] -= mean
a[:,2] /= std

print("Normalize per feature")
print(a)
$\endgroup$

1 Answer 1

11
$\begingroup$

Rather than thinking of the problem in abstract terms, imagine a real-life example. You want to predict job satisfaction (1-5) based on features such as age (in years), work experience (in years), monthly salary (thousands of $), and gender (0 or 1). What would be the point of subtracting the global mean (highly influenced by monthly earnings) of gender or the number of years of work experience? Global or row-wise normalization does not make any sense in most cases. On the other hand, in the column-wise case, you end up with each of the columns having a mean of zero and a standard deviation of one -- each of the features is centered at zero and equally scaled.

$\endgroup$
4
  • $\begingroup$ So whats your point? Don't scale at all? Obs, you have to treat each feature separately in terms of scaling but then as you said mean = zero and std = 1 in most cases. $\endgroup$
    – thigi
    Dec 6, 2017 at 12:50
  • $\begingroup$ @thigi I don't know what is your question about. In most of the cases, you don't need scaling. If you need scaling, then how you do it depends on what you are doing it for: if you just need it for numerical issues then you stick to global scaling of the features if you need it to change meaning of your variables, then what you do depends on your pourpose. $\endgroup$
    – Tim
    Dec 6, 2017 at 12:54
  • $\begingroup$ What are some examples of when it might make sense to use global scaling or sample-wise (row-wise) scaling? $\endgroup$
    – fabiomaia
    Dec 26, 2018 at 3:43
  • $\begingroup$ @fabiomaia Row-wise scaling is the heart of Layer Normalization, which is becoming an increasing popular NN technique $\endgroup$ Aug 2, 2022 at 20:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.