The posted answers are great, and I wanted to add a few "Sanity Checks" which have greatly helped me in the past.


1) **Train your model on a single data point. If this works, train it on two inputs with different outputs.**

This verifies a few things. First, it quickly shows you that your model is able to learn by checking if your model can overfit your data. In my case, I constantly make silly mistakes of doing ```Dense(1,activation='softmax')``` vs ```Dense(1,activation='sigmoid')``` for binary predictions, and the first one gives garbage results.

If your model is unable to overfit a few data points, then either it's too small (which is unlikely in today's age),or something is wrong in its structure or the learning algorithm. 

2) **Pay attention to your initial loss.**

Continuing the binary example, if your data is 30% 0's and 70% 1's, then your intial expected loss around $L=-0.3\ln(0.5)-0.7\ln(0.5)\approx 0.7$. This is because your model should start out close to randomly guessing. 

A lot of times you'll see an initial loss of something ridiculous, like 6.5. Conceptually this means that your output is heavily saturated, for example toward 0. For example $-0.3\ln(0.99)-0.7\ln(0.01) = 3.2$, so if you're seeing a loss that's bigger than 1, it's likely your model is very skewed. This usually happens when your neural network weights aren't properly balanced, especially closer to the softmax/sigmoid. So this would tell you if you're initialization is bad. 

You can study this further by making your model predict on a few thousand examples, and then histogramming the outputs. This is especially useful for checking that your data is correctly normalized. As an example, if you expect your output to be heavily skewed toward 0, it might be a good idea to transform your expected outputs (your training data) by taking a square roots of the expected output. This will avoid gradient issues for saturated sigmoids, at the output. 

3) **Generalize your model outputs to debug**

As an example, imagine you're using an LSTM to make predictions from time-series data. Maybe in your example, you only care about the latest prediction, so your LSTM outputs a single value and not a sequence. Switch the LSTM to return predictions at each step (in keras, this is ```return_sequences=True```). Then you can take a look at your hidden-state outputs after every step and make sure they are actually different. An application of this is to make sure that when you're masking your sequences (i.e. padding them with data to make them equal length), the LSTM is correctly ignoring your masked data. Without generalizing your model *you will never find this issue*. 


4) **Look at individual layers**
Tensorboard provides a useful way of [visualizing your layer outputs][1]. This can help make sure that inputs/outputs are properly normalized in each layer. It can also catch buggy activations. You can also query layer outputs in keras on a batch of predictions, and then look for layers which have suspiciously skewed activations (either all 0, or all nonzero). 


5) **Build a simpler model first**
You've decided that the best approach to solve your problem is to use a CNN combined with a bounding box detector, that further processes image crops and then uses an LSTM to combine everything. It takes 10 minutes just for your GPU to initialize your model.

Instead, make a batch of fake data (same shape), and break your model down into components. Then make dummy models in place of each component (your "CNN" could just be a single 2x2 20-stride convolution, the LSTM just with 2 
hidden units.). This will help you make sure that your model structure is correct and that there are no extraneous issues. I struggled for a while with such a model, and when I tried a simpler version, I found out that one of the layers wasn't being masked properly due to a keras bug. You can easily (and *quickly*) query internal model layers and see if you've setup your graph correctly.

6) **Standardize your Preprocessing and Package Versions**

Neural networks in particular are extremely sensitive to small changes in your data. As an example, two popular image loading packages are ```cv2``` and ```PIL```. Just by virtue of *opening* a JPEG, both these packages will produce **slightly different** images. The differences are usually really small, but you'll occasionally see drops in model performance due to this kind of stuff. Also it makes debugging a nightmare: you got a validation score during training, and then later on you use a different loader and get different accuracy on the same darn dataset. 

So if you're downloading someone's model from github, pay close attention to their preprocessing. What image loaders do they use? What image preprocessing routines do they use? When resizing an image, what interpolation do they use? Do they first resize and then normalize the image? Or the other way around? What's the channel order for RGB images? 

The safest way of standardizing packages is to use a ```requirements.txt``` file that outlines all your packages just like on your training system setup, down to the ```keras==2.1.5``` version numbers. In theory then, using Docker along with the same GPU as on your training system should then produce the same results. 

 
  [1]: https://jhui.github.io/2017/03/12/TensorBoard-visualize-your-learning/