I have a Ph.D. in statistical physics. I have learned a great deal of basic probability from my physics training, but I have not ever taken a course in either probability or statistics. Like most other theoretical physicists, I know very little statistics.
I often find myself looking up basic statistics concepts when they show up. Examples are: what is unbiased estimator? why do we want the estimator to be unbiased in mean and not median? when should I use 1/uncertainties as weights in a fitting algorithm? If I make a collection of measurements with measurement errors, how do I find an unbiased estimator of the standard deviation of the underlying distribution before measurement errors? what is Bayesian learning? When do I use different statistical tests, and what do those tests really do? when does maximum likelihood give the best estimation of the parameters?
I'm looking for a reference that I could study on my own with insightful examples. I want something complete, from beginner to advanced, and practical. I am more comfortable using techniques if I know exactly what they do, what assumptions they make, and how they are derived.
I have a good math background: I am familiar with basic probability concepts such as conditional probability, independence, central limit theorem, various distributions, generating functions, and a little bit of stochastic differential equations. I'm familiar with basic real analysis and Lebesgue measure.
Containing examples with connections to statistical physics and machine learning is a plus.