Return to Answer

Some observations

1. This could be a numerical issue. Floating point arithmetic has finite precision, so values $\epsilon$ and $1 - \epsilon$ for some small $\epsilon >0$ may be rounded to 0 and 1.

2. This could be a printing precision issue. When displaying these values, you may only be printing the first several digits, and when the values are very close to 0 and 1, the result is rounded to 1 and 0. Try changing your print statements to print 16 decimal digits.

3. It could be that the absolute values of the weights are so large that the inputs to the softmax activation are saturated very rapidly.

a. You can bound weights away from large values using $L^2$ regularization.

b. Smaller learning rates could help, too, because then the weights will not move very quickly from small absolute values to values absolute values.

4. This is tangential to the question you asked, but dropout and batch norm don't work well together.

Günter Klambauer, Thomas Unterthiner, Andreas Mayr and Sepp Hochreiter. "Self-Normalizing Neural Networks"

To robustly train very deep CNNs, batch normalization evolved into a standard to normalize neuron activations to zero mean and unit variance. Layer normalization also ensures zero mean and unit variance, while weight normalization ensures zero mean and unit variance if in the previous layer the activations have zero mean and unit variance. Natural neural networks also aim at normalizing the variance of activations by reparametrization of the weights. However, training with normalization techniques is perturbed by stochastic gradient descent (SGD), stochastic regularization (like dropout), and the estimation of the normalization parameters. Both RNNs and CNNs can stabilize learning via weight sharing, therefore they are less prone to these perturbations. In contrast, FNNs trained with normalization techniques suffer from these perturbations and have high variance in the training error (see Figure 1). This high variance hinders learning and slows it down. Furthermore, strong regularization, such as dropout, is not possible as it would further increase the variance which in turn would lead to divergence of the learning process. We believe that this sensitivity to perturbations is the reason that FNNs are less successful than RNNs and CNNs.

My recommendation is to use either dropout or batch normalization; alternatively, you may wish to implement a network using SELUs and alpha dropout as suggested in this paper.

5. I noticed that all of your features are categorical. I've found that entity embeddings can be a powerful strategy for this kind of data.

Cheng Guo, Felix Berkhahn. "Entity Embeddings of Categorical Variables".

We map categorical variables in a function approximation problem into Euclidean spaces, which are the entity embeddings of the categorical variables. The mapping is learned by a neural network during the standard supervised training process. Entity embedding not only reduces memory usage and speeds up neural networks compared with one-hot encoding, but more importantly by mapping similar values close to each other in the embedding space it reveals the intrinsic properties of the categorical variables. We applied it successfully in a recent Kaggle competition and were able to reach the third position with relative simple features. We further demonstrate in this paper that entity embedding helps the neural network to generalize better when the data is sparse and statistics is unknown. Thus it is especially useful for datasets with lots of high cardinality features, where other methods tend to overfit. We also demonstrate that the embeddings obtained from the trained neural network boost the performance of all tested machine learning methods considerably when used as the input features instead. As entity embedding defines a distance measure for categorical variables it can be used for visualizing categorical data and for data clustering.

6. Training a neural network is hard. Here are some general suggestions. What should I do when my neural network doesn't learn?

Some observations

1. This could be a numerical issue. Floating point arithmetic has finite precision, so values $\epsilon$ and $1 - \epsilon$ for some small $\epsilon >0$ may be rounded to 0 and 1.

2. This could be a printing precision issue. When displaying these values, you may only be printing the first several digits, and when the values are very close to 0 and 1, the result is rounded to 1 and 0. Try changing your print statements to print 16 decimal digits.

3. It could be that the absolute values of the weights are so large that the inputs to the softmax activation are saturated very rapidly.

a. You can bound weights away from large values using $L^2$ regularization.

b. Smaller learning rates could help, too, because then the weights will not move very quickly from small absolute values to values absolute values.

4. This is tangential to the question you asked, but dropout and batch norm don't work well together.

Günter Klambauer, Thomas Unterthiner, Andreas Mayr and Sepp Hochreiter. "Self-Normalizing Neural Networks"

To robustly train very deep CNNs, batch normalization evolved into a standard to normalize neuron activations to zero mean and unit variance. Layer normalization also ensures zero mean and unit variance, while weight normalization ensures zero mean and unit variance if in the previous layer the activations have zero mean and unit variance. Natural neural networks also aim at normalizing the variance of activations by reparametrization of the weights. However, training with normalization techniques is perturbed by stochastic gradient descent (SGD), stochastic regularization (like dropout), and the estimation of the normalization parameters. Both RNNs and CNNs can stabilize learning via weight sharing, therefore they are less prone to these perturbations. In contrast, FNNs trained with normalization techniques suffer from these perturbations and have high variance in the training error (see Figure 1). This high variance hinders learning and slows it down. Furthermore, strong regularization, such as dropout, is not possible as it would further increase the variance which in turn would lead to divergence of the learning process. We believe that this sensitivity to perturbations is the reason that FNNs are less successful than RNNs and CNNs.

My recommendation is to use either dropout or batch normalization; alternatively, you may wish to implement a network using SELUs and alpha dropout as suggested in this paper.

5. Another tangential observation: I noticed that all of your features are categorical. I've found that entity embeddings can be a powerful strategy for this kind of data.

Cheng Guo, Felix Berkhahn. "Entity Embeddings of Categorical Variables".

We map categorical variables in a function approximation problem into Euclidean spaces, which are the entity embeddings of the categorical variables. The mapping is learned by a neural network during the standard supervised training process. Entity embedding not only reduces memory usage and speeds up neural networks compared with one-hot encoding, but more importantly by mapping similar values close to each other in the embedding space it reveals the intrinsic properties of the categorical variables. We applied it successfully in a recent Kaggle competition and were able to reach the third position with relative simple features. We further demonstrate in this paper that entity embedding helps the neural network to generalize better when the data is sparse and statistics is unknown. Thus it is especially useful for datasets with lots of high cardinality features, where other methods tend to overfit. We also demonstrate that the embeddings obtained from the trained neural network boost the performance of all tested machine learning methods considerably when used as the input features instead. As entity embedding defines a distance measure for categorical variables it can be used for visualizing categorical data and for data clustering.

6. Training a neural network is hard. Here are some general suggestions. What should I do when my neural network doesn't learn?

Some observations

1. This could be a numerical issue. Floating point arithmetic has finite precision, so values $\epsilon$ and $1 - \epsilon$ for some small $\epsilon >0$ may be rounded to 0 and 1.

2. This could be a printing precision issue. When displaying these values, you may only be printing the first several digits, and when the values are very close to 0 and 1, the result is rounded to 1 and 0. Try changing your print statements to print 16 decimal digits.

3. It could be that the absolute values of the weights are so large that the inputs to the softmax activation are saturated very rapidly.

a. You can bound weights away from large values using $L^2$ regularization.

b. Smaller learning rates could help, too, because then the weights will not move very quickly from small absolute values to values absolute values.

4. This is tangential to the question you asked, but dropout and batch norm don't work well together.

Günter Klambauer, Thomas Unterthiner, Andreas Mayr and Sepp Hochreiter. "Self-Normalizing Neural Networks"

To robustly train very deep CNNs, batch normalization evolved into a standard to normalize neuron activations to zero mean and unit variance [17]. Layer normalization 2 also ensures zero mean and unit variance, while weight normalization [25] ensures zero mean and unit variance if in the previous layer the activations have zero mean and unit variance. Natural neural networks [6] also aim at normalizing the variance of activations by reparametrization of the weights. However, training with normalization techniques is perturbed by stochastic gradient descent (SGD), stochastic regularization (like dropout), and the estimation of the normalization parameters. Both RNNs and CNNs can stabilize learning via weight sharing, therefore they are less prone to these perturbations. In contrast, FNNs trained with normalization techniques suffer from these perturbations and have high variance in the training error (see Figure 1). This high variance hinders learning and slows it down. Furthermore, strong regularization, such as dropout, is not possible as it would further increase the variance which in turn would lead to divergence of the learning process. We believe that this sensitivity to perturbations is the reason that FNNs are less successful than RNNs and CNNs.

My recommendation is to use either dropout or batch normalization; alternatively, you may wish to implement a network using SELUs and alpha dropout as suggested in this paper.

5. I noticed that all of your features are categorical. I've found that entity embeddings can be a powerful strategy for this kind of data.

Cheng Guo, Felix Berkhahn. "Entity Embeddings of Categorical Variables".

We map categorical variables in a function approximation problem into Euclidean spaces, which are the entity embeddings of the categorical variables. The mapping is learned by a neural network during the standard supervised training process. Entity embedding not only reduces memory usage and speeds up neural networks compared with one-hot encoding, but more importantly by mapping similar values close to each other in the embedding space it reveals the intrinsic properties of the categorical variables. We applied it successfully in a recent Kaggle competition and were able to reach the third position with relative simple features. We further demonstrate in this paper that entity embedding helps the neural network to generalize better when the data is sparse and statistics is unknown. Thus it is especially useful for datasets with lots of high cardinality features, where other methods tend to overfit. We also demonstrate that the embeddings obtained from the trained neural network boost the performance of all tested machine learning methods considerably when used as the input features instead. As entity embedding defines a distance measure for categorical variables it can be used for visualizing categorical data and for data clustering.

6. Training a neural network is hard. Here are some general suggestions. What should I do when my neural network doesn't learn?

Some observations

1. This could be a numerical issue. Floating point arithmetic has finite precision, so values $\epsilon$ and $1 - \epsilon$ for some small $\epsilon >0$ may be rounded to 0 and 1.

2. This could be a printing precision issue. When displaying these values, you may only be printing the first several digits, and when the values are very close to 0 and 1, the result is rounded to 1 and 0. Try changing your print statements to print 16 decimal digits.

3. It could be that the absolute values of the weights are so large that the inputs to the softmax activation are saturated very rapidly.

a. You can bound weights away from large values using $L^2$ regularization.

b. Smaller learning rates could help, too, because then the weights will not move very quickly from small absolute values to values absolute values.

4. This is tangential to the question you asked, but dropout and batch norm don't work well together.

Günter Klambauer, Thomas Unterthiner, Andreas Mayr and Sepp Hochreiter. "Self-Normalizing Neural Networks"

To robustly train very deep CNNs, batch normalization evolved into a standard to normalize neuron activations to zero mean and unit variance. Layer normalization also ensures zero mean and unit variance, while weight normalization ensures zero mean and unit variance if in the previous layer the activations have zero mean and unit variance. Natural neural networks also aim at normalizing the variance of activations by reparametrization of the weights. However, training with normalization techniques is perturbed by stochastic gradient descent (SGD), stochastic regularization (like dropout), and the estimation of the normalization parameters. Both RNNs and CNNs can stabilize learning via weight sharing, therefore they are less prone to these perturbations. In contrast, FNNs trained with normalization techniques suffer from these perturbations and have high variance in the training error (see Figure 1). This high variance hinders learning and slows it down. Furthermore, strong regularization, such as dropout, is not possible as it would further increase the variance which in turn would lead to divergence of the learning process. We believe that this sensitivity to perturbations is the reason that FNNs are less successful than RNNs and CNNs.

My recommendation is to use either dropout or batch normalization; alternatively, you may wish to implement a network using SELUs and alpha dropout as suggested in this paper.

5. I noticed that all of your features are categorical. I've found that entity embeddings can be a powerful strategy for this kind of data.

Cheng Guo, Felix Berkhahn. "Entity Embeddings of Categorical Variables".

We map categorical variables in a function approximation problem into Euclidean spaces, which are the entity embeddings of the categorical variables. The mapping is learned by a neural network during the standard supervised training process. Entity embedding not only reduces memory usage and speeds up neural networks compared with one-hot encoding, but more importantly by mapping similar values close to each other in the embedding space it reveals the intrinsic properties of the categorical variables. We applied it successfully in a recent Kaggle competition and were able to reach the third position with relative simple features. We further demonstrate in this paper that entity embedding helps the neural network to generalize better when the data is sparse and statistics is unknown. Thus it is especially useful for datasets with lots of high cardinality features, where other methods tend to overfit. We also demonstrate that the embeddings obtained from the trained neural network boost the performance of all tested machine learning methods considerably when used as the input features instead. As entity embedding defines a distance measure for categorical variables it can be used for visualizing categorical data and for data clustering.

6. Training a neural network is hard. Here are some general suggestions. What should I do when my neural network doesn't learn?

Günter Klambauer, Thomas Unterthiner, Andreas Mayr and Sepp Hochreiter. "Self-Normalizing Neural Networks"

To robustly train very deep CNNs, batch normalization evolved into a standard to normalize neuron activations to zero mean and unit variance [17]. Layer normalization 1 also ensures zero mean and unit variance, while weight normalization [25] ensures zero mean and unit variance if in the previous layer the activations have zero mean and unit variance. Natural neural networks [6] also aim at normalizing the variance of activations by reparametrization of the weights. However, training with normalization techniques is perturbed by stochastic gradient descent (SGD), stochastic regularization (like dropout), and the estimation of the normalization parameters. Both RNNs and CNNs can stabilize learning via weight sharing, therefore they are less prone to these perturbations. In contrast, FNNs trained with normalization techniques suffer from these perturbations and have high variance in the training error (see Figure 1). This high variance hinders learning and slows it down. Furthermore, strong regularization, such as dropout, is not possible as it would further increase the variance which in turn would lead to divergence of the learning process. We believe that this sensitivity to perturbations is the reason that FNNs are less successful than RNNs and CNNs.

Günter Klambauer, Thomas Unterthiner, Andreas Mayr and Sepp Hochreiter. "Self-Normalizing Neural Networks"

To robustly train very deep CNNs, batch normalization evolved into a standard to normalize neuron activations to zero mean and unit variance [17]. Layer normalization 2 also ensures zero mean and unit variance, while weight normalization [25] ensures zero mean and unit variance if in the previous layer the activations have zero mean and unit variance. Natural neural networks [6] also aim at normalizing the variance of activations by reparametrization of the weights. However, training with normalization techniques is perturbed by stochastic gradient descent (SGD), stochastic regularization (like dropout), and the estimation of the normalization parameters. Both RNNs and CNNs can stabilize learning via weight sharing, therefore they are less prone to these perturbations. In contrast, FNNs trained with normalization techniques suffer from these perturbations and have high variance in the training error (see Figure 1). This high variance hinders learning and slows it down. Furthermore, strong regularization, such as dropout, is not possible as it would further increase the variance which in turn would lead to divergence of the learning process. We believe that this sensitivity to perturbations is the reason that FNNs are less successful than RNNs and CNNs.