In the case of approximation tasks using neural networks, should we standardize the data, as in the classification ?

  • $\begingroup$ Please clarify what you mean by "regularize the data"? Do you mean standardize the data or regularize the parameters in the loss-function? $\endgroup$
    – Jim
    Apr 29, 2018 at 12:29
  • $\begingroup$ standardize the data, sorry for misleading $\endgroup$
    – atos
    Apr 29, 2018 at 12:35
  • $\begingroup$ Then please edit your question accordingly. Thank you. $\endgroup$
    – Jim
    Apr 29, 2018 at 12:38
  • $\begingroup$ guess you mean scaling data, so setting mean zero and covariance to unit. And yes you should! $\endgroup$ Apr 29, 2018 at 12:41
  • $\begingroup$ Ok, but should we also standardize the output variable ? $\endgroup$
    – atos
    Apr 29, 2018 at 12:45

1 Answer 1


Let' think of an example: So we have data $\boldsymbol{x}_{1} = \left[\begin{array} -1,1 \end{array} \right]^{T} + \boldsymbol{b} $ and $\boldsymbol{x}_{2} = \left[\begin{array} 11 ,1 \end{array} \right]^{T} + \boldsymbol{b} $ with $\lVert b \rVert| \gg 1 $. And $y_{1} = 0$, $y_{2}=2$.

The loss is assumed to be MSE: $L(n) = \lVert y_{n} - \boldsymbol{w}_{1}\boldsymbol{x}_{n} \rVert ^{2} $. Now let's calculate the derivatives: $\frac{\partial}{\partial \boldsymbol{w}_{1}} L =\left[ \begin{array} \frac{\partial}{\partial w_{12}} L , \frac{\partial}{\partial w_{12}} L\end{array} \right]^{T}=-2\boldsymbol{x}_{n}(y_{n}-\boldsymbol{w}_{1}\boldsymbol{x}_{n})$.

Calculating this for $n=1$

$\frac{\partial}{\partial \boldsymbol{w}_{1}} L \approx -2\left[ \begin{array} bb ,b \end{array} \right]^{T} (y_{1}-\boldsymbol{w}_{1}\boldsymbol{x}_{1})$.

So we see that the offset $\boldsymbol{b}$ is the dominant part.

  • $\begingroup$ Actually, can somebody tell me what's wrong with the TeX in here? $\endgroup$ Apr 29, 2018 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.