So some background on my existing knowledge.
I have my masters in statistics where I spent a good amount of time understanding how machine learning algorithms work, but I was always allowed to use high level packages like Keras to implement them. I can easily make the basic structures like CNNs and RNNs, but lately I've been wanting to experiment with different designs. (For example, I would like to slip a function in between each batch of the training step that effects the updating process, or I would like to develop an activation function with an imaginary component, or I would like to create a forest of neural nets that pass along parts of their layers to newly initialized Nets for time series data, etc)
But for this level of granularity, I will need to deepen my understanding of the lower level side. So my question is this. Those that have went from high level to low level understanding, what was the approach/resources/libraries that you dove into that you found to be a smooth transition?
P.S. More clarification questions, should I dive into tensorflow/Theano/Pytorch, or dive deeper into Keras? Should I start as fundamental as programing tensor operations from scratch, or should I go high-level down? I recognize that this is a difficult question to answer without knowing more about me, but I can clarify as much as possible about my background knowledge.
Edit: So this may be too broad of a question, so I'll narrow it down to the first project that I would like to tackle:
The updating function
I would like to stop the model between each batch of training and see if the validation loss/accuracy improved. If it did, then I do nothing, if it did not, then I would like to not make any adjustments based on the back-propagation, but instead add a small amount of gaussian noise to each layer and try again. Regardless of wether this will lead to improvements or not, it is something I can do to strengthen my understanding of manipulating deep learning architectures.
Would this be something I explore in keras? Or do I need to delve deeper into the lower level platforms? (Bonus if there is a direction I can be pointed in :))
Also, thanks for the welcome into CrossValidated! I can't wait to explore the community!