Inference for the skeptical (but not math-averse) reader

I just watched a lecture on statistical inference ("comparing proportions and means"), part of an intro to stats online course. The material made as little sense to me as it always does (by now I must have seen this stuff dozens of times, spread out over the last three decades).

I'm looking for a book on "basic Stats-101" (point estimation, estimate assessment, statistical inference, hypothesis testing, study design) that takes seriously the problem of convincing a skeptical reader...

Below I give some examples of the type of question that the author I'm searching for would take seriously and know how to address convincingly.

But first let me take a minute to stress that in this post I'm not asking these questions. Please, do not answer them! I give them just as examples, and by way of "litmus test" (for the type of author of searching for).

1. If a "proportion" is simply the mean of a Boolean variable (i.e. one that takes only the values 0 and 1), why are different procedures taught for doing statistical inference with "proportions" and with "means"?

2. If the normal distribution is so robust that assuming normality gives good results even in cases when that data is not quite normally distributed, and if the t-distribution is so normal-looking, why all the fuss about using the t-distribution instead of the normal?

3. What exactly are "degrees of freedom", and why do we worry about them?

4. What does it mean to speak of the "true" value of a parameter, considering that we are just using distributions that happen to look similar to the data?

5. How come "exploratory data analysis" is a good thing, while "data snooping" is an evil thing?

As I've said, I'm put off by the attitude that's implied by a neglect of such questions. It's not the "epistemological stance" that I want to see in someone who's teaching me something. I'm looking for authors who respect the reader's skepticism and rationality, and who know how to address them (without necessarily going off into pages and pages of formalisms and technicalities).

I realize that this is a tall order, and maybe especially so when it comes to statistics. Therefore, I don't expect that many authors will have succeeded at it. But at the moment I'd be content with finding just one.

Let me add that I'm not math-averse. On the contrary, I love math. (I'm comfortable with analysis [aka "advanced calculus"], linear algebra, probability theory, even basic measure theory.)

That said, my interest at the moment is in "applied", "practical", "everyday", "real-world" statistics (as opposed to theoretical niceties). (But I don't want a cookbook either!)

FWIW, I have read the first few chapters of Data analysis using regression and multilevel/hierarchical models by Gelman and Hill, and I like the authors' tone. Their focus is practical, but go into the theory when needed. They also often step back, and assess standard practices critically, and offer candid opinions that appeal to a skeptical reader's commonsense. Unfortunately, these authors have not written a book devoted to the subject I'm asking about in this post ("Stats 101" stuff, as described above). I'm also aware that one of these authors (Gelman) co-authored the highly-regarded Bayesian data analysis, but, again, this is not what I'm looking for at the moment.

EDIT:

Dikran Marsupial raises the following objection:

I don't think there is necessarily anything wrong with neglecting questions, there comes a point where addressing every question detracts from the exposition of the basic concepts which is often more important (especially in a stats 101 book!).

I agree with that. It would be more accurate for me to say that I'm looking for a "second look at basic stats." In fact, with this as my motivation, I looked at the textbooks used at graduate courses on inference (say), and found that they too neglected questions like the ones I've listed. If anything, they seemed even less inclined to delve into such questions (so that they can focus on matters like the conditions for some convergence-or-other of this-or-that...).

The problem is that the more advanced books are addressed to a radically different population of readers, one where the "skepticism of the outsider" has been drastically depleted. IOW, those who are taking graduate-level statistics are past the point of being bothered by the questions that bother me. They're not skeptical about any of this stuff anymore. (How did they get over the skepticism hump? Maybe some were never too critical in the first place, especially if they learned their stats fairly early on--I know I was not a particularly critical freshman myself, for example, though I did not take stats then. Others may have had teachers who filled in where their textbooks fell short. Some may have been clever enough to figure out the answers to such questions for themselves. Who knows.)

• Most of those questions - the ones we aren't to answer - have good answers already here on CV. Some of those questions even have fairly trivial answers. But there's a thousand questions people come up with during any topic -- you're never going to find an explanation that covers all possible questions, neither will you find one that just anticipates just the specific questions that you have while you're watching or reading. Well you might hit one or two just by chance, but overall it's an impossible standard. Why not just ask the questions as they arise? On a site that answers questions? Jul 10 '13 at 0:28
• I will say similar questions do come up when people do statistics (and some books answer some of them), but most of them aren't actual questions relating to skepticism - mostly they're simple comprehension questions. e.g. "what are degrees of freedom and why do we care" isn't skeptical at all, it's asking for clarity of explanation and motivation. The reasons why advanced users of statistics don't worry about that is because they already understand why they're so valuable. Jul 10 '13 at 0:31

You have already got some good suggestions. Here are some more. First, two blogs that I read sporadically, and where questions such as you ask yourself are sometimes discussed. As they are blogs, you could even ask questions and get some very good answers! Here they come:

http://andrewgelman.com/ (Andrew Gelman)

http://errorstatistics.com/ (Deborah Mayo)

And a few books I think will help you: Box, Hunter & Hunter: Statistics for experimenters.

As the title says, this is a ("first", but really, really ... second) course for people which would like to design their own experiments, and so analyze them. Very high on the "why" part.

Then: D R Cox: Principles of Statistical Inference, another very good book about the "why" not the "how".

And, since you ask why means and proportions are treated differently, here is a book which does not do that: http://www.amazon.com/Statistics-4th-David-Freedman/dp/0393929728/ref=sr_1_1?s=books&ie=UTF8&qid=1373395118&sr=1-1&keywords=freedman+statistics

Low on maths, high on principles.

Try a radically different path to the subject: get "A History of Mathematical Statistics (From 1750 to 1930)", by Anders Hald, and learn about the history of our subject. Once you grasp the slow emergence of the concept of a statistical model, your questions will look trivial. The two pieces of a statistical model must be clearly understood: the observable data $X$ and the non-observable parameter $\Theta$. The sampling distribution of $X\mid \Theta$ is postulated, and our goal is to learn about $\Theta$ given some values of $X$. Looking at some of your questions: 1) Different models; 2) The $t$ distribution is the sampling distribution of a specific statistic (a function of data $X$) when the data are supposed to be normal; 3) The degrees of freedom characterize the sampling distribution of a statistic supposing that the values of $\Theta$ are constrained (by a so-called null hypothesis), and so on. Also, pick some simple inference problem (like normal data with known variance) and solve it in both the classical and Bayesian ways. Contrast the differences. That may be illuminating.

• +1 a large part of the reason why statistical procedures appear questionable is because they are rather counter-intuitive and a historical appreciation of how statistics ended up the way it is, is probably a good approach for the skeptic. Jul 9 '13 at 14:28
• @DikranMarsupial: In fact, I have read Stigler's "The history of statistics" precisely to get at the bottom of my difficulties with stats. And it did help some. Unfortunately, it leaves out most of the Fisher vs Neyman era, when (I suspect) much of the perverse bizarreness arose. I attempted about 50 pages of Hald's book, but found his derivations impossible to follow: they were clearly addressed to readers with significant statistical expertise... Come to think of it, I think that a book addressed to the educated layman and titled "Why is statistics so weird?" would sell well... :)
– kjo
Jul 9 '13 at 14:56
• IMHO the counter-intuitiveness of statistics pre-dates the Fisher vs Neyman era (although intuitiveness does not imply usefulness or vice-versa). Ian Hacking's books may be worth a try, I enjoyed "The Emergence of Probability". I find Bayesian stats much easier to understand conceptually than frequentist stats, but much harder to perform in practice. Understanding the differences between Bayesian and frequentist approaches helped me a lot in understanding the frequentist approach. Jul 9 '13 at 15:10
• Thanks, I'll check out Hacking. His books having in my sights for a long time, though I've put off reading him thinking he was more centered around the philosophy of probability theory (a subject I find fascinating in its own right). That said, I recognize that at least a few of my problems with statistics impinge on the interpretation of probability, so maybe reading Hacking will take care of at least some of my objections.
– kjo
Jul 9 '13 at 16:50
• @kjo I read Hacking's Logic of Statistical Inference & found it very 'philosophical' - he was up on the idea of likelihood as 'support' and down on Neyman-Pearson but kept on with simple examples - didn't discuss nuisance parameters, multi-parameter inference, or anything tricky (unless I've forgotten). Worth reading, certainly, but I wouldn't recommend it to start with. Jul 9 '13 at 17:48

I rather doubt there will be a single book that suits you as individual people tend to be skeptical about different things, and books are written for a target audience, rather than for individuals. This is one of the good things about being taught by a person, rather than just a book, which is that you can ask questions as you go. This is a pretty difficult thing to do in a linear text.

I don't think there is necessarily anything wrong with neglecting questions, there comes a point where addressing every question detracts from the exposition of the basic concepts which is often more important (especially in a stats 101 book!).

I suspect the best approach is to get a good book and then look up the answer to unanswered questions elsewhere. I've got a bookshelf full of statistics texts in front of me, simply because none of them in isolation is all that I need (not even Jaynes' book ;o).

For the absolute beginner, I think Grant Foster's book "Understanding Statistics" is a good place to start, but I suspect it is rather too basic in this case.

• "I don't think there is necessarily anything wrong with neglecting questions..." I think this is a fair objection; I've added an EDIT to my post to address it.
– kjo
Jul 9 '13 at 14:49

Abelson (1995), Statistics as Principled Argument is introductory & has an interesting take on some of the questions that often confuse learners.

But perhaps you just need to read some books on theoretical statistics (skipping all the stuff about convergence, metric spaces, &c.) & then even if they don't answer specifically questions like your examples, you'll be able to answer most of them yourself, & look up the rest, as @Dikran suggests.

I suggested in another thread reading Cox & Hinkley, Theoretical Statistics or Cox, Principles of Statistical Inference together with Casella & Berger, Statistical Inference to get an understanding of the different perspectives there are.

• I found it interesting but I don't think it is at the level expected by the OP at all.
– Gala
Jul 9 '13 at 15:31
• @Gael You're probably right, especially in view of the edit. Jul 9 '13 at 16:56