I'm looking for resources on the theory behind choosing a loss function for ML---I'm interested in GBDT but for deep learning would work as well. I'd like to get a better understanding of how the loss function affects the model, the difference between validation loss and training loss, etc.
I've googled around and most medium articles I find are too superficial, and I don't know where to start looking for more academic resources.
the difference between validation loss and training loss, etc.
This is addressed to some extent in chapter 7 of Elements of Statistical Learning which is available for free from the authors online.
There are some other general discussions of loss function properties in other chapters of that book (Chapter 3, Chapter 4, Chapter 10 for example).
I'm interested in GBDT
The original paper on Boosting by Schapire might be useful, as well as this later paper by the same author.
Some other general papers on loss functions:
The original paper on L1 regularization by Tibshirani
A comparison of L1 and L2 regularization by Andrew Ng
The original paper on L2 regularization by Tikhonov is in Russian, you might be able to find a translation somewhere.
