Is it necessary to standardize data for neural networks? I'm learning about neural networks and I've been trying to figure out wether it's a good idea to normalize/standardiza data before training. From what I've read there's divided opinions about this, some sources says it will imrpove the result/make the training faster while other sources state there's no real need for it. I guess the general answere is "it depends" on the situation. Here's my use case and I'd like to hear what you think is the better approach:
I have a dataset with about 15 independent parameters of different types, some are categorical strings ("cat","dog,"horse"...,"elefant"), others are categorical integers (1,2,3,4...,10) and yet others are continous numerical data (1.32,3.123,2.132...,1.234). Some values are missing for some of the parameters (sometimes most of the values are missing). In my current preprossecing I do this:


*

*Transforms categoriacal strings to integers starting from 1: ("cat","dog,"horse"...) -> (1,2,3...)

*Categorical integers and numerical data are left as is.

*Replaces all missing values (NA) with 0


The training is done in R using rxNeuralNet and the output is a continous value (regression). From what I've read there are ways to standardize the data which might improve the performance (scaling, centering to zero mean etc). In this case, would it be meaningful to do this preprocessing?
 A: Talking about feature scaling - it's optional, and need of it depends on data values. The motivation behind feature scaling is to speed up your backpropagation. Because to update your weights you will use some algo (i.e. gradient descend). And most of this algorithms works better on scaled features (in fact, performs worse on non-scaled).
So, depending on your data values you may not even see the speed difference, but more your feature values differs, the more it affects the backptop performance.
And, by the way, idea to map words into categorical integers may end up with bad ANN accuracy (with high odds). Because integers are linear, not categorical discrete. I.e. $dog \ngtr cat$, while $2 > 1$. And your ANN probably will be "tricked" with this. In some cases, it will increase your performance/accuracy, but in most cases it will hurt.
More "classic" approach for this is to map categories into sparse vectors, just like with termvectors in NLP - single dimension for single category.
A: I agree with Slam, I would not map strings to integers, I think you could map it into a one-hot vector, for example:
cat dog bird

if your class is a cat, it would be:
1 0 0

You can map each possible animal to one different vector.
Regarding the normalization issue, I have been told that is strongly recommended to normalize or standardize them (Scikit ANN tips ) (sorry I usually work with Python),  because if one feature is ranging from 0 to 0.1 and other features range from 1-1000 the ANN can give more importance to certain features.
It's a bit like if like your features are kg and cm or kg and m, it might make a difference when it shouldn't. This is what I have heard, but I have never carried out tests to know if it's really true.
A: For tabular data converting categorical to integers and using an embedding layer is rather standard nowadays. Especially if you have a lot of categories, this tends to be a much better approach than one-hot-encoding (or inputting category number as a numeric value that gets treated as a continuous predictor), and that's one of the main scenarios, in which neural networks get really competitive with gradient boosted trees for tabular data.
For continuous inputs standardizing makes sense to speed up convergence/make things easier, unless somehow the scales across variables are truly comparable, when maybe keeping original scales makes sense. The more interesting question is whether additional transformations make sense (e.g. log-transformation for strictly positive variables with a long right-hand-tail, rank-Gauss transformation etc.), which is probably rather problem dependent. I.e. you'd ideally want to be on a scale, where your output can be explained by a linear combination of input features, like in a traditional regression equation, in which case you don't need many layers or non-linear activations (obviously, that will only ever be an approximation). While people talk a lot about neural networks "doing their own feature engineering", they still benefit from good features and "optimally" transformed features.
It's always worth looking at previous work, such as the book by the fast.ai team (link is to the github repository for the book), the code of the standard tabular model in the fastai library (plus the related paper on how to regularize such models), TabNet, SAINT, pytorch-widedeep, this from Yandex and so on. There are currently a lot of papers appearing on neural networks for tabular data.
