In the case of approximation tasks using neural networks, should we standardize the data, as in the classification ?
5
-
$\begingroup$ Please clarify what you mean by "regularize the data"? Do you mean standardize the data or regularize the parameters in the loss-function? $\endgroup$– JimApr 29, 2018 at 12:29
-
$\begingroup$ standardize the data, sorry for misleading $\endgroup$– atosApr 29, 2018 at 12:35
-
$\begingroup$ Then please edit your question accordingly. Thank you. $\endgroup$– JimApr 29, 2018 at 12:38
-
$\begingroup$ guess you mean scaling data, so setting mean zero and covariance to unit. And yes you should! $\endgroup$– pythonic833Apr 29, 2018 at 12:41
-
$\begingroup$ Ok, but should we also standardize the output variable ? $\endgroup$– atosApr 29, 2018 at 12:45
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
1
Let' think of an example: So we have data $\boldsymbol{x}_{1} = \left[\begin{array} -1,1 \end{array} \right]^{T} + \boldsymbol{b} $ and $\boldsymbol{x}_{2} = \left[\begin{array} 11 ,1 \end{array} \right]^{T} + \boldsymbol{b} $ with $\lVert b \rVert| \gg 1 $. And $y_{1} = 0$, $y_{2}=2$.
The loss is assumed to be MSE: $L(n) = \lVert y_{n} - \boldsymbol{w}_{1}\boldsymbol{x}_{n} \rVert ^{2} $. Now let's calculate the derivatives: $\frac{\partial}{\partial \boldsymbol{w}_{1}} L =\left[ \begin{array} \frac{\partial}{\partial w_{12}} L , \frac{\partial}{\partial w_{12}} L\end{array} \right]^{T}=-2\boldsymbol{x}_{n}(y_{n}-\boldsymbol{w}_{1}\boldsymbol{x}_{n})$.
Calculating this for $n=1$
$\frac{\partial}{\partial \boldsymbol{w}_{1}} L \approx -2\left[ \begin{array} bb ,b \end{array} \right]^{T} (y_{1}-\boldsymbol{w}_{1}\boldsymbol{x}_{1})$.
So we see that the offset $\boldsymbol{b}$ is the dominant part.
-
$\begingroup$ Actually, can somebody tell me what's wrong with the TeX in here? $\endgroup$ Apr 29, 2018 at 13:03