Standardizing neural network inputs with a linear layer? I'm contributing to a ML software project and noticed something weird in the code:
They perform standardization by introducing a linear layer right after the inputs. This linear layer has the same number of nodes as the inputs. I've never heard of such a practice... How does this standardize the inputs?
The only comment in the code regarding this says: We disguise standardization in the first linear layer to keep it seamlessly in a sequential PyTorch object.
But I do not understand how this is considered "standardization".
 A: One way to do standardization is to subtract some value (e.g. the sample mean $\hat \mu$) and divide by another value (e.g. the sample standard deviation $\hat \sigma$):
$$
z = \frac{x - \hat \mu}{\hat \sigma}.
$$
When $X$ is a matrix, we can compute the columns' means and standard deviations; each is a vector. Then we can center and scale each vector $x$ with these vectors:
$$\begin{align}
z &= \left(\hat \sigma I\right)^{-1}(x - \hat \mu) \\
&= \left(\hat \sigma I\right)^{-1}x - \left(\hat \sigma I\right)^{-1} \hat\mu  \\
&= Ax + b
\end{align}$$
This should be recognizable as the same operations of a linear layer: matrix-vector multiplication and vector-vector addition.
In other words, if you assign $A,b$ the exact values that you want to use and then never update those values, the linear layer will do standardization for you. This is the stated design goal: put the manipulations for standardization into the PyTorch object, instead of standardizing the data prior to handing it off to the model.
