1
$\begingroup$

Can you use the sigmoid function to normalize continuous features that have no theoretical maximum value but tend to cluster around [-1, 1]?

Although using the sigmoid function would be a non-linear normalization, my intuition is that the deep neural network or machine learning model that I am training would learn that the continuous feature is not linearly normalized, therefore adapting and doing fine. Is this correct? Can the sigmoid function or any other non-linear normalization method (ex. tanh) be used for continuous features?

$\endgroup$
1
$\begingroup$

The description you give is basically what a sigmoid feed-forward neural network does in its hidden layers: find $a,b$ so that $\sigma(x|a,b)$ minimizes some loss, where $\sigma$ is any sigmoid function, for example you could choose $\sigma(x|a,b)=\tanh(ax+b)$. Depending on the choice of $a,b$, the function could be basically constant at a large value, basically constant at a small value, or approximately linear, or some kind of mix of all three.

This is presented in terms of scalar-valued functions, but NNs with more than one unit uses matrix-vector products.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.