# Regularization for approximation in neural networks

In the case of approximation tasks using neural networks, should we standardize the data, as in the classification ?

• Please clarify what you mean by "regularize the data"? Do you mean standardize the data or regularize the parameters in the loss-function?
– Jim
Apr 29, 2018 at 12:29
• standardize the data, sorry for misleading
– atos
Apr 29, 2018 at 12:35
– Jim
Apr 29, 2018 at 12:38
• guess you mean scaling data, so setting mean zero and covariance to unit. And yes you should! Apr 29, 2018 at 12:41
• Ok, but should we also standardize the output variable ?
– atos
Apr 29, 2018 at 12:45

Let' think of an example: So we have data $\boldsymbol{x}_{1} = \left[\begin{array} -1,1 \end{array} \right]^{T} + \boldsymbol{b}$ and $\boldsymbol{x}_{2} = \left[\begin{array} 11 ,1 \end{array} \right]^{T} + \boldsymbol{b}$ with $\lVert b \rVert| \gg 1$. And $y_{1} = 0$, $y_{2}=2$.
The loss is assumed to be MSE: $L(n) = \lVert y_{n} - \boldsymbol{w}_{1}\boldsymbol{x}_{n} \rVert ^{2}$. Now let's calculate the derivatives: $\frac{\partial}{\partial \boldsymbol{w}_{1}} L =\left[ \begin{array} \frac{\partial}{\partial w_{12}} L , \frac{\partial}{\partial w_{12}} L\end{array} \right]^{T}=-2\boldsymbol{x}_{n}(y_{n}-\boldsymbol{w}_{1}\boldsymbol{x}_{n})$.
Calculating this for $n=1$
$\frac{\partial}{\partial \boldsymbol{w}_{1}} L \approx -2\left[ \begin{array} bb ,b \end{array} \right]^{T} (y_{1}-\boldsymbol{w}_{1}\boldsymbol{x}_{1})$.
So we see that the offset $\boldsymbol{b}$ is the dominant part.