First of all, I would like to state that implementation is a dubious word in the context of neural networks (according to me). I guess you listed most of the approaches, but this is how I recently approached testing.
The first interpretation is the actual implementation in some source code. In this case, I think the most important ways of testing this part are:
- Check whether the basic operations are correct. Especially when using matrix operations, you might have overseen that
* is actually an element-wise multiplication rather than a dot product or you got mismatching dimensions.
- Perform gradient checking. I like to do it for multiple derivatives. It starts with the derivative of the activation function and the total derivative of the cost function w.r.t. weights. If there would be a problem there, you might apply it to the derivative w.r.t. inputs (this should be about the update of the bias) or the outputs (the weighted sum of errors in the consequent layer)
- I also like to create artificial, trivial learning tasks. For instance for a fully connected network, I let a single layer network learn the sum of the inputs or anything for which I know what the weights should be. For multi-layer networks, this this is a bit harder, but you could do something similar without expecting certain weights. This is practically the same idea as using some reference datasets.
The second interpretation for the implementation of a neural network could be the choice of hyperparameters. The most important of these hyperparameters is probably the learning rate. You mentioned some techniques that, in my eyes, rather belong to choosing the right hyperparameters.
- Plotting your objective as a function of the epochs for train/valid/test data is a good way to get an idea whether your network is overfitting/underfitting by comparing train/test/valid errors. In case of overfitting, you could reduce the number of layers/neurons and in case of underfitting you could add some. If you don't seem to be learning anything, this is probably due to a learning rate that is too high (Try to minimise the quadratic function with steps of 2, starting from 1).
- If you suffer from exploding or vanishing gradient, you probably need a different activation function or a different learning rate.
I don't think that looking at the weights is that useful. In case of CNNs it helps to tell what kind of kernels you are learning or if you implement some form of weight decay where you really want to force some weights to zero.
It probably does not add much to what you already know and I am forgetting a lot here, but this would be my answer, for what it's worth...