Is there a textbook / handbook with full derivations for statistical / machine learning concepts? In particular, I am looking for a textbook which will go over the details of derivations (including all calculus and linear algebra) for learning models and concepts such as logistic regression, Gaussian Discriminant Analysis, with full proofs for variants like Gaussian Naive Bayes.
Books such as "Elements of Statistical Learning" tend to gloss over certain details. For example, when discussing L1 Regularized Logistic Regression (Section 4.4.4), it says that "the score equations ... have the form", and then presents the form without giving the derivation.
 A: Have a look at 'A First Course in Machine Learning,' Simon Rogers and Mark Girolami.
There are many easy to follow step by step derivations of concepts that include calculus and linear algebra. Also, you can look at google book preview to see if it fits your needs.
A: Take it as a fact that no single book can satisfy your criteria. It depends on the depth of the concepts you want to go into. It also depends on what do you mean when you say statistical concepts. 
If by statistical concepts you mean distribution theory, hypothesis testing, ANOVA, Regression and other day to day statistics then Statistical Inference by Casella and Berger should be followed. If you want details of a specific advanced concept say Time Series then you can follow Time Series Analysis by Box and Jenkins for Univariate Time Series and  Introduction to Multivariate Time series by Lutkepohl etc and so on for different topics
Similarly for Machine learning also . Mining of Massive datasets is a comprehensive book with proofs on Machine Learning, but again it all depends on where you are where you want to head to. 
That being said, Forums,Research Papers and Publications are way to go for latest proofs, results etc.
