Taleb and the Black Swan Taleb's book "The Black Swan" was a New York Times best seller when it came out several years ago.  The book is now in its second edition.  After meeting with statisticians at a JSM (an annual statistical conference), Taleb toned down his criticism of statistics somewhat.  But the thrust of the book is that statistics is not very useful because it relies on the normal distribution and very rare events: "Black Swans" don't have normal distributions.  
Do you think this is valid criticism? Is Taleb missing some important aspects of statistical modeling?  Can rare events be predicted at least in the sense that probabilities of occurrences can be estimated?
 A: I also have not read the book, but there is no way that his point can be as simplistic as saying that there are distributions with fatter tails than the normal distribution.  This would be a comment to the other answers, but I have not accumulated enough accolades on this website.
From Wikipedia:
"He states that statistics is fundamentally incomplete as a field as it cannot predict the risk of rare events..."
This question is also quite similar to What is the community's take on the Fourth Quadrant?
A: I read the Black Swan a couple of years ago.  The Black Swan idea is good and the attack on the ludic fallacy (seeing things as though they are dice games, with knowable probabilities) is good but statistics is outrageously misrepresented, with the central problem being the wrong claim that all statistics falls apart if variables are not normally distributed.  I was sufficiently annoyed by this aspect to write Taleb the letter below:
Dear Dr Taleb
I recently read "The Black Swan".  Like you, I am a fan of Karl Popper, and I found myself agreeing with much that is in it.  I think your exposition of the ludic fallacy is basically sound, and draws attention to a real and common problem.  However, I think that much of Part III lets your overall argument down badly, even to the point of possibly discrediting the rest of the book.  This is a shame, as I think the arguments with regard to Black Swans and "unknown unknowns" stand on their merits without relying on some of the errors in Part III.
The main issue I wish to point out - and seek your response on, particularly if I have misunderstood issues - is your misrepresentation of the field of applied statistics.  In my judgement, chapters 14, 15 and 16 depend largely upon a straw man argument, misrepresenting statistics and econometrics.  The field of econometrics that you describe is not the one that I was taught when I studied applied statistics, econometrics, and actuarial risk theory (at the Australian National University, but using texts that seemed pretty standard).  The issues that you raise (such as the limitations of Gaussian distributions) are well and truly understood and taught, even at the undergraduate level.
For example, you go to some lengths to show how income distribution does not follow a normal distribution, and present this as an argument against statistical practice in general.  No competent statistician would ever claim that it does, and ways of dealing with this issue are well established.  Just using techniques from the very most basic "first year econometrics" level, for example, transforming the variable by taking its logarithm would make your numerical examples look much less convincing.  Such a transformation would in fact invalidate much of what you say, because then the variance of the original variable does increase as its mean increases.
I am sure there are some incompetent econometricians who do OLS regressions etc with an untransformed response variable the way you say, but that just makes them incompetent and using techniques which are well established to be inappropriate.  They would certainly have been failed even in undergraduate courses, which spend much time looking for more appropriate ways of modelling variables such as income, reflecting the actual observed (non-Gaussian) distribution.
The family of Generalized Linear Models is one set of techniques developed in part to get around the problems you raise.  Many of the exponential family of distributions (eg Gamma, Exponential, and Poisson distributions) are asymmetrical and have variance that increases as the centre of the distribution increases, getting around the problem you point out with using the Gaussian distribution.  If this is still too limiting, it is possible to drop a pre-existing "shape" altogether and simply specify a relationship between the mean of a distribution and its variance (eg allowing the variance to increase proportionately to the square of the mean), using the "quasi-likelihood" method of estimation.
Of course, you could argue that this form of modelling is still too simplistic and an intellectual trap that lulls us into thinking the future will be like the past.  You may be correct, and I think the strength of your book is to make people like me consider this.  But you need different arguments to those that you use in chapters 14-16.  The great weight you place on the fact that the variance of the Gaussian distribution is constant regardless of its mean (which causes problems with scalability), for instance, is invalid.  So is your emphasis on the fact that real-life distributions tend to be asymmetric rather than bell-curves.
Basically, you have taken one over-simplification of the most basic approach to statistics (naïve modelling of raw variables as having Gaussian distributions) and shown, at great length, (correctly) the shortcomings of such an oversimplified approach.  You then use this to make the gap to discredit the whole field.  This is either a serious lapse in logic, or a propaganda technique.  It is unfortunate because it detracts from your overall argument, much of which (as I said) I found valid and persuasive.
I would be interested to hear what you say in response.  I doubt I am the first to have raised this issue.
Yours sincerely
PE
A: I don't think Taleb would actually say that statistical techniques relying on the Gaussian distribution are not useful. His point in the book was that they are highly useful for many (but not all) physical or biological processes and modeling. He makes some good points and some bad (The Black Swan and Linked were the beginning of the "everything is a power law!" plague that still haunts us today), but it's important to remember that the book is a collection of literary and philosophical essays meant for the lay person. 
That said I think Taleb likes to aggravate people. You can see this in his battle with Myron Scholes. In this case it may have been useful as statistical education at the undergrad level, and sometimes at the graduate level, sort of flits over the assumption of Gaussian distributions.  I imagine during his years in finance he came across a large number of quants with a great knowledge of Black-Scholes and other techniques but who did not consider underlying assumptions like the distribution. I suspect Taleb was poking at the educational establishment for a failure to properly educate.
A: Those of you who have not read the book are way off base. He makes a LARGE distinction between the scalable and unscalable. For unscalable matters traditional stats will serve one well enough. He is not critiquing that whatsoever. Black Swans originate in the scalable and are hard to predict given past empirical data. The book is about how these events can have enormous impact and are generally only explained after the fact. The epistemology is excellent.
A: I did read "The Black Swan", I did enjoy it, and I am a statistician. I didn't find its "criticism of statistics" unbearable, at all. Point by point: 


*

*Taleb did not invent the concept of the black swan. It had been a favored example in philosophical thought for quite a while! 

*Taleb is not so much criticizing "statistics", as certain (bad) applications of it. 

*The book was a bestseller. It was not directed toward statisticians, but to the general public. It did very well in teaching that public about things statisticians knew very well, but many of the other readers (the majority!) did not. So we could learn a lot from that book about how to "sell" statistics. 

*Most important (for me), Taleb included a lot of references to ancient Greek skeptical philosophy. Nobody else has mentioned that point here, but I think that inclusion was the real selling point of the book! 

*The book is a literary work, not a technical work. If you want to criticize Taleb for his technical work, go to his homepage and download some of his technical papers.


For those which doesn' t like this answer, or dislike the book, can have a look at Taleb's technical arguments in the new https://fernandonogueiracosta.files.wordpress.com/2014/07/taleb-nassim-silent-risk.pdf  "Silent Risk", which is technical.   
A: I've not read the book, but as stated the criticism seems pretty unreasonable to me.  If extreme events are important, then statistics has appropriate tools in the toolbox, such as extreme value theory, and a good statistician will know how to use them (or at least find out how to use them and will be sufficiently engaged with the purpose of the analysis to look).  The criticism seems to be "statistics is bad because there are bad statisticians that only know about normal distributions".
A: Saying that " the thrust of the book is that statistics is not very useful " is inaccurate, I think. Having read the book, what he appears to be saying is that things like quantitative finance or any sort of securities trading that assumes a normal distribution is fundamentally flawed (actually, in the book, he calls people who claim to use these models to make predictions, "charlatans"). According to Taleb, while the normal distribution does a great job of modelling the values of tangible/physical things (eg. height, weight, life span etc.), systems like the markets are often driven by human emotion and thus, are prone to large swings that normal distributions cannot accurately predict.
I don't understand statistics well, and until reading the answers here, I'd never heard of things like extreme value theory. Regardless, The Black Swan and Fooled By Randomness seem to have similar premises, which is "normal distribution not always OK". I don't recall him defaming the entire field of statistics. 
A: I strongly recommend Dennis Lindley's review of this book. It contains a number of devastating arguments against the poor and arrogant exposition of ideas in the book:
http://onlinelibrary.wiley.com/doi/10.1111/j.1740-9713.2008.00281.x/abstract
The Black Swan is another example where being a "Best-seller" does not guarantee high quality content.
A: I haven't read the Black Swan, but if his criticism of statistics is really as simple as you say, then it's ridiculous. Obviously some statistics relies on the Normal distribution, but much does not. 
Can rare events be modeled? Of course they can. The real question is how well they can be modeled. And that question will have different answers in different fields, based on how much we know about the rare events and their antecedents. 
In today's NY Times Magazine there's an interesting article by Nate Silver on how weather forecasting has improved in the last decade or so. This includes better modeling of rare events such as hurricanes.
Is the book worth reading?
