# Return to Answer

 8 Typos, consistent formatting, improvements to English usage. edit approved Oct 14 '18 at 16:39 Matt Wenham 21811 silver badge1010 bronze badges A lot of times you'll see an initial loss of something ridiculous, like 6.5. Conceptually this means that your output is heavily saturated, for example toward 0. For example $$-0.3\ln(0.99)-0.7\ln(0.01) = 3.2$$, so if you're seeing a loss that's bigger than 1, it's likely your model is very skewed. This usually happens when your neural network weights aren't properly balanced, especially closer to the softmax/sigmoid. So this would tell you if you'reyour initialization is bad. You can study this further by making your model predict on a few thousand examples, and then histogramming the outputs. This is especially useful for checking that your data is correctly normalized. As an example, if you expect your output to be heavily skewed toward 0, it might be a good idea to transform your expected outputs (your training data) by taking athe square roots of the expected output. This will avoid gradient issues for saturated sigmoids, at the output. 4) Look at individual layers Tensorboard Tensorboard provides a useful way of visualizing your layer outputs. This can help make sure that inputs/outputs are properly normalized in each layer. It can also catch buggy activations. You can also query layer outputs in keras on a batch of predictions, and then look for layers which have suspiciously skewed activations (either all 0, or all nonzero). 5) Build a simpler model first You've You've decided that the best approach to solve your problem is to use a CNN combined with a bounding box detector, that further processes image crops and then uses an LSTM to combine everything. It takes 10 minutes just for your GPU to initialize your model. Instead, make a batch of fake data (same shape), and break your model down into components. Then make dummy models in place of each component (your "CNN" could just be a single 2x2 20-stride convolution, the LSTM just with just 2 hidden units.). This will help you make sure that your model structure is correct and that there are no extraneous issues. I struggled for a while with such a model, and when I tried a simpler version, I found out that one of the layers wasn't being masked properly due to a keras bug. You can easily (and quickly) query internal model layers and see if you've setup your graph correctly. A lot of times you'll see an initial loss of something ridiculous, like 6.5. Conceptually this means that your output is heavily saturated, for example toward 0. For example $$-0.3\ln(0.99)-0.7\ln(0.01) = 3.2$$, so if you're seeing a loss that's bigger than 1, it's likely your model is very skewed. This usually happens when your neural network weights aren't properly balanced, especially closer to the softmax/sigmoid. So this would tell you if you're initialization is bad. You can study this further by making your model predict on a few thousand examples, and then histogramming the outputs. This is especially useful for checking that your data is correctly normalized. As an example, if you expect your output to be heavily skewed toward 0, it might be a good idea to transform your expected outputs (your training data) by taking a square roots of the expected output. This will avoid gradient issues for saturated sigmoids, at the output. 4) Look at individual layers Tensorboard provides a useful way of visualizing your layer outputs. This can help make sure that inputs/outputs are properly normalized in each layer. It can also catch buggy activations. You can also query layer outputs in keras on a batch of predictions, and then look for layers which have suspiciously skewed activations (either all 0, or all nonzero). 5) Build a simpler model first You've decided that the best approach to solve your problem is to use a CNN combined with a bounding box detector, that further processes image crops and then uses an LSTM to combine everything. It takes 10 minutes just for your GPU to initialize your model. Instead, make a batch of fake data (same shape), and break your model down into components. Then make dummy models in place of each component (your "CNN" could just be a single 2x2 20-stride convolution, the LSTM just with 2 hidden units.). This will help you make sure that your model structure is correct and that there are no extraneous issues. I struggled for a while with such a model, and when I tried a simpler version, I found out that one of the layers wasn't being masked properly due to a keras bug. You can easily (and quickly) query internal model layers and see if you've setup your graph correctly. A lot of times you'll see an initial loss of something ridiculous, like 6.5. Conceptually this means that your output is heavily saturated, for example toward 0. For example $$-0.3\ln(0.99)-0.7\ln(0.01) = 3.2$$, so if you're seeing a loss that's bigger than 1, it's likely your model is very skewed. This usually happens when your neural network weights aren't properly balanced, especially closer to the softmax/sigmoid. So this would tell you if your initialization is bad. You can study this further by making your model predict on a few thousand examples, and then histogramming the outputs. This is especially useful for checking that your data is correctly normalized. As an example, if you expect your output to be heavily skewed toward 0, it might be a good idea to transform your expected outputs (your training data) by taking the square roots of the expected output. This will avoid gradient issues for saturated sigmoids, at the output. 4) Look at individual layers Tensorboard provides a useful way of visualizing your layer outputs. This can help make sure that inputs/outputs are properly normalized in each layer. It can also catch buggy activations. You can also query layer outputs in keras on a batch of predictions, and then look for layers which have suspiciously skewed activations (either all 0, or all nonzero). 5) Build a simpler model first You've decided that the best approach to solve your problem is to use a CNN combined with a bounding box detector, that further processes image crops and then uses an LSTM to combine everything. It takes 10 minutes just for your GPU to initialize your model. Instead, make a batch of fake data (same shape), and break your model down into components. Then make dummy models in place of each component (your "CNN" could just be a single 2x2 20-stride convolution, the LSTM with just 2 hidden units). This will help you make sure that your model structure is correct and that there are no extraneous issues. I struggled for a while with such a model, and when I tried a simpler version, I found out that one of the layers wasn't being masked properly due to a keras bug. You can easily (and quickly) query internal model layers and see if you've setup your graph correctly. 7 deleted 2 characters in body edited Sep 29 '18 at 19:11 Alex R. 11.2k11 gold badge1919 silver badges4141 bronze badges This verifies a few things. First, it quickly shows you that your model is able to learn by checking if your model can overfit your data. In my case, I constantly make sillingsilly mistakes of doing Dense(1,activation='softmax') vs Dense(1,activation='sigmoid') for binary predictions, and the first one gives garbage results. This verifies a few things. First, it quickly shows you that your model is able to learn by checking if your model can overfit your data. In my case, I constantly make silling mistakes of doing Dense(1,activation='softmax') vs Dense(1,activation='sigmoid') for binary predictions, and the first one gives garbage results. This verifies a few things. First, it quickly shows you that your model is able to learn by checking if your model can overfit your data. In my case, I constantly make silly mistakes of doing Dense(1,activation='softmax') vs Dense(1,activation='sigmoid') for binary predictions, and the first one gives garbage results. 6 deleted 10 characters in body edited Jun 20 '18 at 23:49 Alex R. 11.2k11 gold badge1919 silver badges4141 bronze badges If your model is unable to overfit a few data points, then either it's too small (which is unlikely in today's age). Most likely,or something is wrong in its structure or the learning algorithm. If your model is unable to overfit a few data points, then either it's too small (which is unlikely in today's age). Most likely something is wrong in its structure or the learning algorithm. If your model is unable to overfit a few data points, then either it's too small (which is unlikely in today's age),or something is wrong in its structure or the learning algorithm. 5 Fixed numbering edit approved Jun 20 '18 at 5:24 4 added 6 characters in body edited Jun 19 '18 at 19:00 Alex R. 11.2k11 gold badge1919 silver badges4141 bronze badges 3 added 195 characters in body edited Jun 19 '18 at 18:54 Alex R. 11.2k11 gold badge1919 silver badges4141 bronze badges 2 added 195 characters in body edited Jun 19 '18 at 18:47 Alex R. 11.2k11 gold badge1919 silver badges4141 bronze badges 1 answered Jun 19 '18 at 18:45 Alex R. 11.2k11 gold badge1919 silver badges4141 bronze badges