What materials are a starting point for developing deep learning architectures?

Question

So some background on my existing knowledge.

I have my masters in statistics where I spent a good amount of time understanding how machine learning algorithms work, but I was always allowed to use high level packages like Keras to implement them. I can easily make the basic structures like CNNs and RNNs, but lately I've been wanting to experiment with different designs. (For example, I would like to slip a function in between each batch of the training step that effects the updating process, or I would like to develop an activation function with an imaginary component, or I would like to create a forest of neural nets that pass along parts of their layers to newly initialized Nets for time series data, etc)

But for this level of granularity, I will need to deepen my understanding of the lower level side. So my question is this. Those that have went from high level to low level understanding, what was the approach/resources/libraries that you dove into that you found to be a smooth transition?

P.S. More clarification questions, should I dive into tensorflow/Theano/Pytorch, or dive deeper into Keras? Should I start as fundamental as programing tensor operations from scratch, or should I go high-level down? I recognize that this is a difficult question to answer without knowing more about me, but I can clarify as much as possible about my background knowledge.

Edit: So this may be too broad of a question, so I'll narrow it down to the first project that I would like to tackle:

The updating function

I would like to stop the model between each batch of training and see if the validation loss/accuracy improved. If it did, then I do nothing, if it did not, then I would like to not make any adjustments based on the back-propagation, but instead add a small amount of gaussian noise to each layer and try again. Regardless of wether this will lead to improvements or not, it is something I can do to strengthen my understanding of manipulating deep learning architectures.

Would this be something I explore in keras? Or do I need to delve deeper into the lower level platforms? (Bonus if there is a direction I can be pointed in :))

Also, thanks for the welcome into CrossValidated! I can't wait to explore the community!

Answer 1

The most important thing to grasp about most neural-network oriented machine learning frameworks/libraries is that they provide three things:

1. An abstraction called "tensor", with which you can efficiently compute common operations (matrix multiply, relu, etc) on a wide range of different hardware (CPU/GPU/etc).
2. A flexible automatic differentiation system which is necessary for backpropagation / optimization.
3. Lots and lots of convenience code for any common task you might perform -- for example, implementations of SGD, common data augmentation algorithms, common weight initialization strategies.

Keras is an even higher level wrapper over tensorflow, so it only does 3. If you want to explore into the "guts" of things, you might want to ditch it, since it is only an extra level of abstraction you have to work around. Tensorflow and PyTorch provide more direct access to the details imo.

You can't easily (and you shouldn't need to) touch 1 or 2, but since 3 is basically "helper code", you can throw it all out and write your own. Just force yourself not to use any helper code provided by the library that you can write yourself.

I would like to stop the model between each batch of training and see if the validation loss/accuracy improved.

Presumably you're calling some library's "fit" or "train" function. You can just delete this function call and write your own version which you have full control over. It may help to look at the source code of the library's function.