My understanding is most of the tutorials recommend normalizing / scaling the data prior feeding the tensorflow models. Doesn't normalization require that data conforms to the normal parametric distribution? What good is a non-linear model if scaling / normalizing is a pre-requisite prior using the non-linear model such as tensorflow?
So back to the question, should I always normalize / scale my data prior feeding my tensorflow models?
Yes, normalisation/scaling is typically recommended and sometimes very important. Especially for neural networks, normalisation can be very crucial because when you input unnormalised inputs to activation functions, you can get stuck in a very flat region in the domain and may not learn at all. Or worse, you can end up with numerical issues.
One very obvious reason is that you need to tune (but you don't) the weight initialisations in the network according to the input range corresponding to that weight, e.g. let $x_1,x_2$ be two distinct features and $w_1,w_2$ be the corresponding weights. Also let the range of the feature be as follows: $x_1\in[0,1000],x_2\in[0,1]$. When you initialise $w_i$ with numbers within $[-1,1]$ for example, it won't mean the same for $x_1$ and $x_2$. Probably, the sum $w_1x_1+w_2x_2$ will be dominated by $w_1x_1$ and you won't see the effect of $w_2x_2$ for some time unless you're very lucky, and learning will be hindered significantly until the network is finally able to learn what $w_1$ should have been in the first place.
Doesn't normalization require that data conforms to the normal parametric distribution?
No, normalisation has nothing to do with normal distribution. One form of normalisation, called standardising, which is subtracting the mean and dividing by the deviation is very common in the literature and typically used for converting a normal RV into standard normal RV. Although the idea may stem from normal distributions, the operation has nothing to do with normal distribution.
The mention of Tensorflow in the question is a red herring -- the reasons that scaling is beneficial are not unique to Tensorflow, but instead are common to all methods that update model parameters using gradient descent. The reason we scale data is to improve gradient descent dynamics, and backpropagation using Tensorflow is just one application of gradient descent among many.
Optimization via gradient descent proceeds more easily when the optimization surface is more "circular" and less "elliptical." The optimization proceeds more quickly in directions corresponding to the largest eigenvalue and more slowly in directions corresponding to the smallest eigenvalue. In other words, the optimization procedure is easier when the eigenvalues of the Hessian are on the same scale.
None of this depends on the input data conforming to any particular parametric distribution; rescaling the inputs to have a common variance has the effect of preconditioning the Hessian matrix.
More information: In Machine learning, how does normalization help in convergence of gradient descent?
In general, real-valued inputs can be rescaled.
There are some corner cases where rescaling inputs doesn't make any sense. For example, embeddings use a lookup table to transform integer-coded inputs to a specific vector. Rescaling this to have 0 mean and variance 1 (or vary between 0 and 1) is meaningless and not helpful, because it breaks the lookup table property.
