What should be taught first: Probability or Statistics? I have newly joined as a faculty member in a math dept. of a reputed institution. I will be teaching the course Probability and Statistics at the undergraduate level. The institution already has a syllabus for this course which I am not very much satisfied with. In that syllabus, statistics is covered first, also estimation part is missing. I always thought basics of probability should be taught before teaching statistics. Can someone give some opinion on this? Also a suggestion for the topics that should be covered in such a course is greatly appreciated.
 A: The plural of anecdote isn't data, but in almost any course I've seen, at least the basics of probability comes before statistics.
On the other hand, historically, ordinary least squares was developed before the normal distribution was discovered! The statistical method came first, the more rigorous, probability based justification of why it works came second!
Stephen Stigler's History of Statistics: Measurement of Uncertainty Before 1900 takes the reader through the historical development:


*

*Mathematicians, astronomers understood basic mechanics and the law of gravity. They could describe the motion of heavenly bodies as a function of several parameters.

*They also had hundreds of observations of the celestial bodies, but how should the observations be combined to recover the parameters?


*

*A hundred observations gives you one hundred equations, but if there
are only three unknowns to solve for, this is an overdetermined system...


*Legendre was first to develop the method of minimizing the sum of the square error. Later this was connected with the work in probability of Gauss and Laplace, that ordinary least squares was in some sense optimal given normally distributed errors.


Why do I bring this up?
There's a certain logical elegance to first build up the mathematical machinery required to derive, understand some method, to lay the foundation before you build the house.
In the reality of science though, the house often comes first, the foundation second :P.
I'd love to see results from the education literature. What's more effective for teaching? What then why? Or why then what?
(I might be a weirdo, but I found the story of how least squares was developed to be an exciting page turner! Stories can make otherwise boring, abstract stuff come alive...)
A: I think it should be an iterative process for most people: you learn a little probability, then a little statistics, then a little more probability, and little more statistics etc.
For instance, take a look at the PhD Stat requirements at GWU. The PhD level Probability course 8257 has the following brief description:
STAT 8257. Probability. 3 Credits.
Probabilistic foundations of statistics, probability distributions, random variables, moments, characteristic functions, modes of convergence, limit theorems, probability bounds. Prerequisite: STAT 6201– STAT 6202, knowledge of calculus through functions of several variables and series.

Note, how it has Master's level statistics courses 6201 and 6202 in the pre-requisites. If you drill down to the lowest level stat or probability course in GWU, you'll get to Introduction to Business and Economic Statistics 1051 or Introduction to Statistics in Social Science 1053. Here's the description to one of them:
STAT 1051. Introduction to Business and Economic Statistics. 3 Credits.
Lecture (3 hours), laboratory (1 hour). Frequency distributions, descriptive measures, probability, probability distributions, sampling, estimation, tests of hypotheses, regression and correlation, with applications to business.

Notice, how the course has "Statistics" title but it teaches a probability within it. For many it's the first encounter with Probability theory after the high school "Stats" course.
This is somewhat similar to how it was taught in my days: the courses and textbooks were usually titled "Probability theory and mathematical statistics", e.g. Gmurman's text.
I can't imagine studying probability theory without any stats whatsoever. The PhD level course above 8257 assumes you already know statistics. So even if you first teach probability there will be some statistics learning involved. It's just for the first course it probably makes a sense to weigh a tad more on statistics, and use it to introduce probability theory too. 
In the end it's an iterative process as I described in the beginning. And as in any good iterative process the first step is not important, whether the very first concept was from stats or probability won't matter after several iterations: you'll get to the same place regardless.
Final note, the teaching approach depends on your field. If you're studying physics, you'll get things like statistical mechanics, Fermi-Dirac statistics, which you're not going to deal with in social sciences. Also, in physics the frequentist approaches are natural, and in fact they're in the basis of some fundamental theories. Hence, it makes a sense to have a stand-alone probability theory taught early on, unlike social sciences where it may not make much sense to spend time on it and instead weigh more on statistics.
