Looking for a mathematical book on probability and statistics The Short question: Where can I find a book for the theory of probability and statistics that teaches from scratch in a rigorous (very important condition) way? The book must not be elementary, but it has to start from scratch. (For example, I think the Lang/Hungerford algebra texts begin by defining what a group is: in that sense they start from scratch.)
The long question: I only took an engineering course in probability and statistics. In my opinion, it is very lousy/non-rigorous. You may assume I have no knowledge of probability and statistics. I have to take an independent study statistics course this year. I am allowed to choose a book for the course. It has to be a statistics course. My instructor assumes I know probability because I took the course mentioned above. (I admit I  have a poor understanding of probability and this irritates me a lot.) I'd like to have a book that:
1) Is mathematically oriented and rigorous
2) Has a significant statistics part
3) Teaches the amount of probability needed to do statistics.
 A: As discussed on Meta, I think it's a rather unusual person who could follow a mathematically rigorous treatment of Probability & Statistics without having previously studied them in a shallower, more heuristic fashion. If you're that person ...
For rigour the pair


*

*Lehmann & Romano, Testing Statistical Hypotheses

*Lehmann & Casella, Theory of Point Estimation
would be hard to beat, but would still leave you needing a book on Probability—so three books, not one.


*

*Cox & Hinkley, Theoretical Statistics
is one I'd like to recommend, for clarity of exposition & for their take on foundational issues, but it tends to assume a fair amount of familiarity with the basics, has nothing on Probability; & though it's thorough, I couldn't put my hand on my heart & say "rigorous".
On the whole I'd probably go with


*

*Casella and Berger, Statistical Inference
; its opening chapters cover enough Probability & the later ones on inference are still pretty rigorous. Rather scanty on Bayesian analysis though.
A: The problem is that "rigorous" for probability nowadays means "rigorous measure theory" - and measure theory is a rather tough shell. 
For the econometrics field at least, I still found the books by Aris Spanos, the most useful in terms of containing the essence of the rigorous axiomatic approach to probability, together with the fundamental asymptotics and then from them deriving formally various statistical procedures of estimation and inference. He continuously stresses the probabilistic nature of the statistical models, and that they have to be consistent with their probabilistic foundations (and he does trace the connections). But again, his books have econometrics in mind - for the "statistics" part of your answer it is important to consider what statistical tools are used in your discipline - and then search for a book that contains these and a good treatment of probability itself... I am not sure you will find it in one book though.
A: The book we used in my Statistical Theory classes was Probability and Statistical Inference, by Nitis Mukhopadhyay.  It definitely starts at the beginning with "Notions of Probability" and the classic coin toss, but then quickly moves into statistics and statistical distributions and moments and such.  It definitely seems mathematically-oriented to me, I pretty much never crack it when working on applied statistics projects.
