If you want to learn "from the beginning", I think it depends on what kind of mathematical background you are starting with. If you don't have much background at all, it is recommended that you first start with probability and then move on to statistics.
There are plenty of books out there, and in my opinion the safest choices are in the classical imprints (Wiley Stats, Duxbury Press, Dover) where any edition will do since the material is not "new". I have found/liked "Mathematical Statistics with Applications" by Wackerly, Mendenhall and Shaeffer from Duxbury Press because it has a lot of examples and exercises. If you're looking for something cheaper, any probability book in the Dover Press is good, as I think they only print classics.
If you're looking for discussions about various approaches in their historical (other context), you need to consult a history of statistics book because that is where such matters are explained (not in introductory material). Moreover you need to master the basics of frequentist parametric statistics before being able to meaningfully evaluate which approach is the right one for your particular problem.