There are several threads on this site for book recommendations on introductory statistics and machine learning but I am looking for a text on advanced statistics including, in order of priority: maximum likelihood, generalized linear models, principal component analysis, non-linear models. I've tried Statistical Models by A.C. Davison but frankly I had to put it down after 2 chapters. The text is encyclopedic in its coverage and mathematical treats but, as a practitioner, I like to approach subjects by understanding the intuition first, and then delve into the mathematical background.

These are some texts that I consider outstanding for their pedagogical value. I would like to find an equivalent for the more advanced subjects I mentioned.

• I wonder... how does the Hyndman et al. book treat the topics you list above? The normal treatment of these issues in forecasting is quite specific to the field, so I would not expect one to learn a lot for application to general statistics from a forecasting book. – Stephan Kolassa Jul 27 '12 at 20:28
• @StephanKolassa The books I listed are just examples of introductory statistics that I mentioned for the pedagogical value. – Robert Kubrick Jul 27 '12 at 20:54

Maximum likelihood: In all Likelihood (Pawitan). Moderately clear book and the most clear (IMO) with respect to books dealing with likelihood only. Also has R code.

GLMs: Categorical Data Analysis (Agresti, 2002) is one of the best written stat books I have read (also has R code available). This text will also help with maximum likelihood. The third edition is coming out in a few months.

Second on my list for the above two is Collett's Modelling Binary Data.

PCA: I find Rencher's writing clear in Methods of multivariate analysis. This is a graduate level text, but it is introductory.

• I agree. I may be prejudice because I think he references my bootstrap bppk and not many people do. – Michael R. Chernick Jul 28 '12 at 1:45
• Your books are great. :) If the question had asked for a bootstrap book, they would have been on my list. – julieth Jul 28 '12 at 1:53
• +1 for Collet's book. It contains lots of useful data sets. – user10525 Jul 28 '12 at 15:31
• Thanks to all for the excellent selection. I chose this answer simply because is short in recs and offers more breadth in terms of subjects treated (PCA, likelihood, multivariate analysis). I will only know which books are best after reading them, of course. Agresti is not included here but it has strong support by the other posters. – Robert Kubrick Jul 28 '12 at 19:32

Some books on Likelihood Estimation

• * Amari, Barndorff-Nielsen, Kass, Lauritzen and Rao, Differential geometry in statistical inference. $-\small{\text{Geometrical approach for proving existence, uniqueness and other properties of MLE.}}$

• * Butler, Saddlepoint Approximations with Applications.
$-\small{\text{Saddlepoint approximations to the MLE on complicated models.}}$

• * Cox, Principles of Statistical Inference.
$-\small{\text{A basic reference on MLE.}}$

• * Cox and Barndorff-Nielsen, Inference and Asymptotics. $-\small{\text{Likelihood, pseudo-likelihood, approximation theorems and asymptotics explained by}}$ $\small{\text{two exponents in this area.}}$

• * Edwards, Likelihood.
$-\small{\text{A reference for a general discussion on this concept.}}$

• * Ferguson, A Course in Large Sample Theory. $-\small{\text{Contains classical results on asymptotic properties of point estimators.}}$

• * Kalbfleisch, Probability and Statistical Inference II. $\spadesuit$
$-\small{\text{Introductory book containing interesting basic results such as the continuous }}$ $\small{\text{approximation to the likelihood which is not always explained.}}$

• * Lehmann and Casella, Theory of Point Estimation.
$-\small{\text{Classical results on point estimation, an essential reference.}}$

• * Pace and Salvan, Principles of Statistical Inference: From a Neo-Fisherian Perspective. $-\small{\text{A good reference on a school of thought becoming more and more popular:}}$ $\small{\text{the Neo-Fisherian.}}$

• * Pawittan, In All Likelihood: Statistical Modelling and Inference Using Likelihood.

• * Serfling, Approximation Theorems of Mathematical Statistics. $-\small{\text{More rigorous book, here you can find the mystical "regularity conditions".}}$

• * Severini, Likelihood Methods in Statistics.

• * Shao, Mathematical Statistics.
$-\small{\text{Classical results, good as a textbook.}}$

• * Sprott, Statistical Inference in Science. $\spadesuit$ $-\small{\text{Basic reference on likelihood, profile likelihood and classical statistical modelling.}}$

• * van der Vaart, Asymptotic Statistics.
$-\small{\text{A general reference on: modes of convergence, properties of MLE, delta method,}}$ $\small{\text{ moment estimators, efficiency and tests.}}$

• * Young and Smith, Essentials of Statistical Inference. $-\small{\text{A more recent book on: Likelihood, pseudolikelihood, saddlepoint approximations,}}$ $\small{p^*\text{ formula, modified profile likelihoods and more.}}$

$\spadesuit$ Suggestion for the OP

• Are these listed in a particular order (e.g., your favorite to your least favorite) or not? – Jake Westfall Jul 27 '12 at 16:46
• @Jake As my memory brought them back. I will include more as soon as I manage to recall them and then I will organise them in alphabetical order. – user10525 Jul 27 '12 at 16:48
• @Procrastinator Thanks for the exhaustive list, but I am more interested in specific recommendations focused on the criteria I described rather than a large list. – Robert Kubrick Jul 27 '12 at 19:02
• @RobertKubrick I do not quite understand your comment, but of course, it is a matter of taste. I am focused on books dealing with some aspects of likelihood estimation. I can tell you some particular aspects if you wish. This topic is too extensive to be summarised in a couple of books ... and believe me, I am not choosing books randomly. – user10525 Jul 27 '12 at 19:06
• I am familiar with some of the books and for the ones I know I think Procrastinator has identified books that fit the criteria. But Robert Kubrick what do you expect of us. This is a difficult question and we serve you best by giving you a good list. in the end you have personal choices to make and we don't know you well enough to pick for you. We are not saying buy every book on the list . But you can go to amazon and take a peek inside. Read the customer reviews and the publisher descriptions. – Michael R. Chernick Jul 27 '12 at 23:21

My guess is that, for your requirements, the best book on generalized linear models is probably:

There are other books that might be considered better, but I suspect would be less appealing to a practitioner who would prefer to avoid dense mathematics:

As for your other topics, I'm afraid I don't know of books for them, but maybe others can make some recommendations.

• McCullagh & Nelder certainly demands some mathematical sophistication, but I think "completely impenetrable for all but very advanced mathematical statisticians" overdoes it. I think it's less mathematically demanding than, say, Hogg & Craik. – Peter Flom Jul 27 '12 at 17:40
• Sorry about the hyperbole, @Peter, I've edited the comment. (However, that was in line w/ what I've heard; I note that I haven't actually read it.) – gung - Reinstate Monica Jul 27 '12 at 17:55
• Good suggestions @Gung. – Michael R. Chernick Jul 27 '12 at 23:23
• Peter did you mean Hogg & Craig. Bob Hogg has a new coauthor in the recent editins Elliot Tanis. – Michael R. Chernick Jul 27 '12 at 23:26
• I would recommend anything by Agresti. He's very high on the clarity/sophistication index. That is, at any given level of mathematical sophistication, Agresti writes clearly, compared to others. – Peter Flom Jul 28 '12 at 1:34

Not sure if these are at the level you're looking for, but some books I've found useful-

GLMs - McCullagh and Nelder is the canonical book

PCA - A User's Guide to Principal Components - despite the title it does go into some degree of depth on the topic

The Nonlinear Models books that I like and rely on are (1) Bates and Watts and (2) Gallant. Both are published by Wiley.

• Oops, @Gung and I were editing at the same time to fix the links. Now neither edit is there! OK, now I think it's right – Peter Flom Jul 28 '12 at 1:30
• +1, thanks Michael. I'm sure that will be helpful from the OP. BTW, the way I do links is to copy the link on the other page, double-click / highlight the word or phrase that I want to serve as the hyperlink here, & then click on the button above the text window next to the quotation mark that looks like 3 links from a chain. That opens a wizard where you can paste the web address. Cheers – gung - Reinstate Monica Jul 28 '12 at 1:34
• I'm sure the book is good but \$202! – Glen Jul 28 '12 at 13:37
• Wiley books are a little more expensive than other publishers but this is not far from the going rate for advanced statistics books. I got mine long ago when they were much cheaper. But the answer is then to borrow it from a technical library or get it used. Used copies of texts like these often sell for considerably less evenin good condition, much like buying a used car. – Michael R. Chernick Jul 28 '12 at 19:54

I really like Larry Wasserman's books "All of Statistics" and "All of Nonparametric Statistics". They are very readable, and cover a lot of ground quickly.

• (+1) I had not realized these (well-received) books included advanced topics as well as introductory material. – whuber May 28 '14 at 17:08
• The only problem is that the book is too brief. – LaTeXFan Dec 12 '14 at 1:53

For Bayesian analysis (including imprecise analysis), I'm going to put in big plugs for:

That last book, by the brilliant Peter Walley, is an eye-opener on different ways of doing sensitivity analysis, and the fact that this can be built into probability theory at an axiomatic level.

Mehta (2014) Statistical Topics (ISBN: 978-1499273533) is good intermediate level statistics story telling. Doesn't cover much of you topics you noted above though.

One really simple introductory statistics book is Andy Field's "Discovering Statistics using R" - also available for SPSS. It contains a lot of nice examples and is even fun to read. Less precise, though compared to other books, but with very little mathematical formulations and lots of text. I found it easy for a basic start, and am still using it from time to time.

• That's a reasonable suggestion, but I think it isn't quite what the OP is asking for. – gung - Reinstate Monica Dec 2 '13 at 19:50
• +1. Andy Field's books are not at all what this question is about so far as I can tell. – Nick Cox Jun 9 '14 at 15:14