How to verify that an implementation of a neural network works correctly? I wonder what some useful techniques are to check whether an implementation of a neural network works correctly.
Below are some checks I am aware of, I would be interested to know more of them:


*

*Plotting some metrics (F1-score, accuracy, some cost, etc.) on the train/test/valid sets against the batch or epoch number.

*Looking at the evolution of matrix weights across epoch.

*In case of multiple layers, removing some layers and see if it still learns something.

*Using the network on some data sets that it should be able to learn (though I am not aware of any good list of reference data sets to test neural network implementations, which would be a valuable resource, faster than looking for reference points in the literature.) 

*Performing gradient checking.

*Checking that the gradient doesn't explode or vanish.



Some plots exemplifying the previous checks:


*

*plotting train/test/valid set vs epoch number:


Good (quickly overfitting though, but at least the network is able to learn):

Bad:



*Looking at the evolution of matrix weights across epoch:


Epoch 0:

Epoch 50:

Hard to tell whether the changes in the weight matrix are good or bad (without looking at the cost), but at least most of them have changed, and they are not stuck at 0.
 A: First of all, I would like to state that implementation is a dubious word in the context of neural networks (according to me). I guess you listed most of the approaches, but this is how I recently approached testing.
The first interpretation is the actual implementation in some source code. In this case, I think the most important ways of testing this part are:


*

*Check whether the basic operations are correct. Especially when using matrix operations, you might have overseen that * is actually an element-wise multiplication rather than a dot product or you got mismatching dimensions.

*Perform gradient checking. I like to do it for multiple derivatives. It starts with the derivative of the activation function and the total derivative of the cost function w.r.t. weights. If there would be a problem there, you might apply it to the derivative w.r.t. inputs (this should be about the update of the bias) or the outputs (the weighted sum of errors in the consequent layer)

*I also like to create artificial, trivial learning tasks. For instance for a fully connected network, I let a single layer network learn the sum of the inputs or anything for which I know what the weights should be. For multi-layer networks, this this is a bit harder, but you could do something similar without expecting certain weights. This is practically the same idea as using some reference datasets.


The second interpretation for the implementation of a neural network could be the choice of hyperparameters. The most important of these hyperparameters is probably the learning rate. You mentioned some techniques that, in my eyes, rather belong to choosing the right hyperparameters.


*

*Plotting your objective as a function of the epochs for train/valid/test data is a good way to get an idea whether your network is overfitting/underfitting by comparing train/test/valid errors. In case of overfitting, you could reduce the number of layers/neurons and in case of underfitting you could add some. If you don't seem to be learning anything, this is probably due to a learning rate that is too high (Try to minimise the quadratic function with steps of 2, starting from 1).

*If you suffer from exploding or vanishing gradient, you probably need a different activation function or a different learning rate.


I don't think that looking at the weights is that useful. In case of CNNs it helps to tell what kind of kernels you are learning or if you implement some form of weight decay where you really want to force some weights to zero.
It probably does not add much to what you already know and I am forgetting a lot here, but this would be my answer, for what it's worth...
A: You can also plot the learning curve (see section 3.3 in the link below), which is the error (both training error and cross-validating error) vs training set size. From the learning curve, you'll know whether your model suffers from high bias or high variance.
http://www.astroml.org/sklearn_tutorial/practical.html
