The Introduction to Statistical Inference by Kiefer, mentioned in a comment by ramiro, is a great candidate for this and my own question. But since ramiro's comment is folded and my question is marked as duplicated, I decided to add an answer here so that more people would know this great book.
The book perfectly fulfills the first three requirements listed in my question. Specifically:
- After a brief description of what is statistical inference in plain English (Chapter 1), it introduces the decision theoretic framework of statistical inference (Chapter 2&3), along with many insightful remarks. Chapter 4 is amazing. It presents most of the principles of statistics: Bayes, Minimax, Admissibility, Unbiasedness, Maximum Likelihood, Method of Moments, along with criticisms and connections.
- Afterwards, it goes into Linear Regression (Chapter 5), Point Estimation (Chapter 7), Hypothesis Testing (Chapter 8), and Confidence Intervals (Chapter 9). Sufficiency (Chapter 6) and Asymptotics (Section 7.6) are also covered. All these are packed concisely within about 300 pages.
- The prerequisites are only Calculus and Linear Algebra, the latter is optional if you omit '5.4 Analysis of the General Linear Model'. Also, it avoids formalization whenever possible. That said, the mathematical maturity required to follow the (often informal) reasoning is actually more than most of introductions to statistics I've tried (e.g. Casella & Berger, which is usually considered as advanced undergraduate or graduate level). You need to devote many thoughts into nuances of statistical ideas that are rarely found in books of this level.
An additional advantage is the good exercises. They are not simple repetitions of the examples, but rather like exam problems. They are broken into several sub-problems to guide the readers, and there are also hints. A unique feature is the the author's suggestions of which problems you should do as a minimum.
- Although the exercises are well-designed and you can solve most of them independently, it would be better if there are solutions.
- Since the book was written in the 70's, the notation is a bit weird and it takes some time to get used to.
Finally, maybe this should not be count as a problem of the book itself, but as mentioned in my another question, I think it is a big plus if one can get to know some computational techniques and tools from the very beginning of learning Statistics.