I have a neural network that takes categorical and quantitative features as inputs. The quantitative features are scaled in $[0,1]$. I apply an embedding layer to get a continuous representation of the categorical features and then i concatenate them with the quantitative features again.
My question is, in this case, what is the best way to ensure the embeddings and the quantitative features have the same scale before feeding them to the next layer?
There are a number of options. Here are a few:
The easiest way to get a consistent scale between the quantitative features and the embedding representation is to use $z$-score scaling (subtract the mean, divide by the standard deviation). Then just put a batch normalization layer after the embedding, before you concatenate. You just care about the running mean and running standard deviation estimates, so you don't have to train the affine parameters. This doesn't meet the $[0,1]$ scaling requirement, but it's dirt-simple to implement.
If you must have $[0,1]$ scaling, then you'll need to extract the values of the embedding matrix, select the min and max, and then use them to achieve $[0,1]$ scaling in the usual way. This will require knowledge of how the software works, but should be achievable in every modern library.
Use a sigmoid transformation. The function $\sigma(x) = \frac{1}{\exp(-x)+1}$ is bounded between 0 and 1. The function $\sigma(ax+b)$ changes the steepness of the function in $a$ and shifts where the midpoint occurs in $b$.
Depending on what you need to achieve, "do nothing" may be an acceptable solution. That is, you might not need to scale the embedding outputs to have the same scale as the rest of the parameters to estimate the model. One way this could happen is if the embeddings themselves do not present any conditioning problems for gradient descent. Another way is that, instead of scaling the embedding outputs, the embeddings are constrained to be self-scaling, such as by requiring them to have a unit norm.
