Return to Answer

added 247 characters in body

Source Link

edited Dec 22, 2020 at 17:21

There are at least two waysa number of options. Here are a few:

The easiest way to get a consistent scale between the quantitative features and the embedding representation is to use $z$-score scaling (subtract the mean, divide by the standard deviation). Then just put a batch normalization layer after the embedding, before you concatenate. You just care about the running mean and running standard deviation estimates, so you don't have to train the affine parameters. This doesn't meet the $[0,1]$ scaling requirement, but it's dirt-simple to implement.
If you must have $[0,1]$ scaling, then you'll need to extract the values of the embedding matrix, select the min and max, and then use them to achieve $[0,1]$ scaling in the usual way. This will require knowledge of how the software works, but should be achievable in every modern library.

Use a sigmoid transformation. The function $\sigma(x) = \frac{1}{\exp(-x)+1}$ is bounded between 0 and 1. The function $\sigma(ax+b)$ changes the steepness of the function in $a$ and shifts where the midpoint occurs in $b$.
Depending on what you need to achieve, "do nothing" may be an acceptable solution. That is, you might not need to scale the embedding outputs to have the same scale as the rest of the parameters to estimate the model. One way this could happen is if the embeddings themselves do not present any conditioning problems for gradient descent. Another way is that, instead of scaling the embedding outputs, the embeddings are constrained to be self-scaling, such as by requiring them to have a unit norm.

There are at least two ways.

The easiest way to get a consistent scale between the quantitative features and the embedding representation is to use $z$-score scaling (subtract the mean, divide by the standard deviation). Then just put a batch normalization layer after the embedding, before you concatenate. You just care about the running mean and running standard deviation estimates, so you don't have to train the affine parameters. This doesn't meet the $[0,1]$ scaling requirement, but it's dirt-simple to implement.
If you must have $[0,1]$ scaling, then you'll need to extract the values of the embedding matrix, select the min and max, and then use them to achieve $[0,1]$ scaling in the usual way. This will require knowledge of how the software works, but should be achievable in every modern library.
Depending on what you need to achieve, "do nothing" may be an acceptable solution. That is, you might not need to scale the embedding outputs to have the same scale as the rest of the parameters to estimate the model. One way this could happen is if the embeddings themselves do not present any conditioning problems for gradient descent. Another way is that, instead of scaling the embedding outputs, the embeddings are constrained to be self-scaling, such as by requiring them to have a unit norm.

There are a number of options. Here are a few:

The easiest way to get a consistent scale between the quantitative features and the embedding representation is to use $z$-score scaling (subtract the mean, divide by the standard deviation). Then just put a batch normalization layer after the embedding, before you concatenate. You just care about the running mean and running standard deviation estimates, so you don't have to train the affine parameters. This doesn't meet the $[0,1]$ scaling requirement, but it's dirt-simple to implement.
If you must have $[0,1]$ scaling, then you'll need to extract the values of the embedding matrix, select the min and max, and then use them to achieve $[0,1]$ scaling in the usual way. This will require knowledge of how the software works, but should be achievable in every modern library.

Use a sigmoid transformation. The function $\sigma(x) = \frac{1}{\exp(-x)+1}$ is bounded between 0 and 1. The function $\sigma(ax+b)$ changes the steepness of the function in $a$ and shifts where the midpoint occurs in $b$.
Depending on what you need to achieve, "do nothing" may be an acceptable solution. That is, you might not need to scale the embedding outputs to have the same scale as the rest of the parameters to estimate the model. One way this could happen is if the embeddings themselves do not present any conditioning problems for gradient descent. Another way is that, instead of scaling the embedding outputs, the embeddings are constrained to be self-scaling, such as by requiring them to have a unit norm.

Source Link

created Dec 22, 2020 at 15:18

There are at least two ways.

The easiest way to get a consistent scale between the quantitative features and the embedding representation is to use $z$-score scaling (subtract the mean, divide by the standard deviation). Then just put a batch normalization layer after the embedding, before you concatenate. You just care about the running mean and running standard deviation estimates, so you don't have to train the affine parameters. This doesn't meet the $[0,1]$ scaling requirement, but it's dirt-simple to implement.
If you must have $[0,1]$ scaling, then you'll need to extract the values of the embedding matrix, select the min and max, and then use them to achieve $[0,1]$ scaling in the usual way. This will require knowledge of how the software works, but should be achievable in every modern library.
Depending on what you need to achieve, "do nothing" may be an acceptable solution. That is, you might not need to scale the embedding outputs to have the same scale as the rest of the parameters to estimate the model. One way this could happen is if the embeddings themselves do not present any conditioning problems for gradient descent. Another way is that, instead of scaling the embedding outputs, the embeddings are constrained to be self-scaling, such as by requiring them to have a unit norm.