# Probabilities in keras (python in R markdown)

I'm struggling with a problem OF class probabilities (binary: 0 and 1). I don't know why but after 100 epochs the probabilities became 0 or 1 (like the class to predict). Maybe something is not working with the code or I missing something?

Train data is 20.000 rows (more or less); test data is 2000 rows. Each row is a match where 0 is lost and 1 is won.

y_train_binary = keras.utils.to_categorical(Y, 2)
y_test_binary = keras.utils.to_categorical(t_Y, 2)
model = Sequential()
model.fit(X, y_train_binary, epochs=300, validation_split=0.2, batch_size=10, verbose=0)
prediction_classes = model.predict_proba(t_X)
numpy.savetxt("C:/Users/Megaport/Desktop/foo.csv", prediction_classes, delimiter=",")


I've tried so many parameters (learning rate, epochs, batch size, epsilon, add layer, less layer, different value of dropout) but the problem is the same: probs are not working.

The probabilities of the class are 1 or 0 and not a value between 1 and 0. A data test with prediction after 100 epoches is like this:

**RESULT**   VALUE_A     VALUE_B     VALUE_C    **PRED_0    PRED_1**
0           4          5          3           1          0
0           7          4          5           0          1
1           6          7          6           0          1
1           2          3          4           0          1


What I'm looking for:

**RESULT**   VALUE_A     VALUE_B     VALUE_C    **PRED_0    PRED_1**
0           4          5          3           0.65          0.35
0           7          4          5           0.25          0.75
1           6          7          6           0.20          0.80
1           2          3          4           0.30          0.70


Plus, someone could please give me some advice looking at accuracy and loss graph?

• I don't understand the problem you're describing. Your plots show the loss decreasing and the accuracy increasing, but not reaching 100% accuracy or zero loss. If the predicted probabilities matched the labels, then you would have 0 loss and 100% accuracy. Can you show us what you mean and clarify your question? – Sycorax says Reinstate Monica Dec 16 '19 at 23:43
• Thanx, I've clarified and edited the question. Let me know if is more clear now. – Mirko Piccolo Dec 17 '19 at 9:00

Some observations

1. This could be a numerical issue. Floating point arithmetic has finite precision, so values $$\epsilon$$ and $$1 - \epsilon$$ for some small $$\epsilon >0$$ may be rounded to 0 and 1.

2. This could be a printing precision issue. When displaying these values, you may only be printing the first several digits, and when the values are very close to 0 and 1, the result is rounded to 1 and 0. Try changing your print statements to print 16 decimal digits.

3. It could be that the absolute values of the weights are so large that the inputs to the softmax activation are saturated very rapidly.

a. You can bound weights away from large values using $$L^2$$ regularization.

b. Smaller learning rates could help, too, because then the weights will not move very quickly from small absolute values to values absolute values.

4. This is tangential to the question you asked, but dropout and batch norm don't work well together.

Günter Klambauer, Thomas Unterthiner, Andreas Mayr and Sepp Hochreiter. "Self-Normalizing Neural Networks"

To robustly train very deep CNNs, batch normalization evolved into a standard to normalize neuron activations to zero mean and unit variance. Layer normalization also ensures zero mean and unit variance, while weight normalization ensures zero mean and unit variance if in the previous layer the activations have zero mean and unit variance. Natural neural networks also aim at normalizing the variance of activations by reparametrization of the weights. However, training with normalization techniques is perturbed by stochastic gradient descent (SGD), stochastic regularization (like dropout), and the estimation of the normalization parameters. Both RNNs and CNNs can stabilize learning via weight sharing, therefore they are less prone to these perturbations. In contrast, FNNs trained with normalization techniques suffer from these perturbations and have high variance in the training error (see Figure 1). This high variance hinders learning and slows it down. Furthermore, strong regularization, such as dropout, is not possible as it would further increase the variance which in turn would lead to divergence of the learning process. We believe that this sensitivity to perturbations is the reason that FNNs are less successful than RNNs and CNNs.

My recommendation is to use either dropout or batch normalization; alternatively, you may wish to implement a network using SELUs and alpha dropout as suggested in this paper.

5. Another tangential observation: I noticed that all of your features are categorical. I've found that entity embeddings can be a powerful strategy for this kind of data.

Cheng Guo, Felix Berkhahn. "Entity Embeddings of Categorical Variables".

We map categorical variables in a function approximation problem into Euclidean spaces, which are the entity embeddings of the categorical variables. The mapping is learned by a neural network during the standard supervised training process. Entity embedding not only reduces memory usage and speeds up neural networks compared with one-hot encoding, but more importantly by mapping similar values close to each other in the embedding space it reveals the intrinsic properties of the categorical variables. We applied it successfully in a recent Kaggle competition and were able to reach the third position with relative simple features. We further demonstrate in this paper that entity embedding helps the neural network to generalize better when the data is sparse and statistics is unknown. Thus it is especially useful for datasets with lots of high cardinality features, where other methods tend to overfit. We also demonstrate that the embeddings obtained from the trained neural network boost the performance of all tested machine learning methods considerably when used as the input features instead. As entity embedding defines a distance measure for categorical variables it can be used for visualizing categorical data and for data clustering.

6. Training a neural network is hard. Here are some general suggestions. What should I do when my neural network doesn't learn?

• Thank you so much! L2 regularization do the work! Now probabilities seems to work. 4. After your thread I've tried to use only dropout, thanks again for your help! 5. I've transformed only the class in categorical, all the features are numerical 6. Priceless! Now I have a lot of material to study! – Mirko Piccolo Dec 17 '19 at 19:26
• Glad that regularization helped. re (5): You have 45 input dimensions. How does that relate to VALUE_A, VALUE_B, etc.? My assumption was that VALUE_* was some categorical feature, and enumerating all of those categories got you to 45 features. – Sycorax says Reinstate Monica Dec 17 '19 at 19:28
• Yep, I have 45 input dimensions. Value_a, b and c was just an example. Do you think the model is underfitted? What would you do? – Mirko Piccolo Dec 17 '19 at 20:00
• It seems that the test loss is consistently lower than the training loss, which fits the definition of under-fitting. I'd follow the advice in my answer. – Sycorax says Reinstate Monica Dec 17 '19 at 20:01