# What is the hardest statistical concept to grasp?

This is a similar question to the one here, but different enough I think to be worthwhile asking.

I thought I'd put as a starter, what I think one of the hardest to grasp is.

Mine is the difference between probability and frequency. One is at the level of "knowledge of reality" (probability), while the other is at the level "reality itself" (frequency). This almost always makes me confused if I think about it too much.

Edwin Jaynes Coined a term called the "mind projection fallacy" to describe getting these things mixed up.

Any thoughts on any other tough concepts to grasp?

• (I don't know enough to put this as an answer, hence adding a comment.) I always thought it was strange that PI crops up in statistical equations. I mean - what's PI got to do with statistics? :) – Reinstate Monica - Goodbye SE Jan 27 '11 at 9:16
• I'd agree (In my surprisal) - I think its that $\pi$ pops up in many mathematical analysis. Just a note you can write $\pi$ by with Latex commands as $\text{\pi}$ enclosed within $signs. I use the wiki page to get the syntax en.wikibooks.org/wiki/LaTeX/Mathematics . Another trick is to "right click" on an equation you see on this site, and select "show source" to get the commands that were used. – probabilityislogic Jan 27 '11 at 10:59 • @Wiki If you accept that$\pi$crops up when you go from measuring the length of a straigh piece of line to the length of a piece of circle, I don't see why it would not appear while going from measuring a probability to fall down on a segment to measuring the probability to fall down in a piece of circle ? – robin girard Jan 27 '11 at 12:19 • @Wiki Whenever you have trigonometric funcions (sine, cosine, tangent etc.) you risk having$\pi$pop up. And remember that whenever you derive a function you're actually finding a tangent. What is surprising is that$\pi$doesn't appear more often. – Carlos Accioly Jan 28 '11 at 18:36 • @Carlos I suspect the prevalence of$2\pi$is mostly due to the use of the$\ell^2$metric, leading to n-spheres. In the same vein, I would expect it's$e$whose prevalence is due to analysis. – sesqu Jan 29 '11 at 22:00 ## 12 Answers for some reason, people have difficulty grasping what a p-value really is. • @shabbychef: Most of the people grasp it in the worst possible way i.e. probability of making Type I error. – suncoolsu Jan 27 '11 at 7:19 • I think that's mostly related to how p-values are explained in classes (i.e.: just by giving a quick definition and without specifying what p-values are NOT) – nico Jan 27 '11 at 7:34 • I think this is mainly to do with how it is introduced. For me, it was an "add-on" to the classical hypothesis test - so it appears as though its just another way to do a hypothesis test. The other problem is that it is usually only taught with respect to a normal distribution, where everything "works nice" (e.g. p-value is a measure of evidence in testing a normal mean). Generalising the p-value is not easy as there is no specific principles to guide the generalisation (e.g. there is no general agreement on how a p-value should vary with the sample size & multiple comparisons) – probabilityislogic Jan 27 '11 at 11:12 • @shabbychef +1 though student often have difficulties with p-values (roughly because the concept in testing is a bit more subtle than a binary decision process and be cause "inverting a function" is not easy to aprehend). When you say "for some reason" do you mean it is unclear for you why people have difficulties ? PS: If I could, I would try to make statistics on this site about the relation between "being a top answer" and "talking about p-value" :) . I also even ask myself if the hardest statistical concept to grasp can have the most upvote (if it is difficult to grasp ... :) ) – robin girard Jan 27 '11 at 12:28 • @eduardo - yes a small enough p-value is sufficient to cast doubt on the null hypothesis: but it is calculated in complete isolation to an alternative. Using p-values alone, you can never formally "reject"$H_0$, because no alternative has been specified. If you formally reject$H_0$, then you must also reject the calculations which was based on the assumption of$H_0$being true, which means you must reject the calculation of the p-value that was derived under this assumption (it messes with your head, but it is the only way to reason consistently). – probabilityislogic Jan 30 '11 at 5:14 Similar to shabbychef's answer, it is difficult to understand the meaning of a confidence interval in frequentist statistics. I think the biggest obstacle is that a confidence interval doesn't answer the question that we would like to answer. We'd like to know, "what's the chance that the true value is inside this particular interval?" Instead, we can only answer, "what's the chance that a randomly chosen interval created in this way contains the true parameter?" The latter is obviously less satisfying. • The more I think about confidence intervals, the harder it is for me to think of what kind of question they can answer at a conceptual level that cannot be answered by asking for "the chance a true value is within an interval, given one's state of knowledge". If I were to ask "what is the chance (conditional on my information) that the average income in 2010 was between 10,000 and 50,000?" I don't think the theory of confidence intervals can give an answer to this question. – probabilityislogic Jan 27 '11 at 11:28 What is the meaning of "degrees of freedom"? How about df that are not whole numbers? Conditional probability probably leads to most mistakes in everyday experience. There are many harder concepts to grasp, of course, but people usually don't have to worry about them--this one they can't get away from & is a source of rampant misadventure. • +1; could you add an example or two, favourite or current ? – denis Jan 31 '11 at 14:53 • For starters: P(you have the disease|test is positive) != P(test is positive|you have the disease). – xmjx Sep 23 '11 at 6:39 I think that very few scientists understand this basic point: It is only possible to interpret results of statistical analyses at face value, if every step was planned in advance. Specifically: • Sample size has to be picked in advance. It is not ok to keep analyzing the data as more subjects are added, stopping when the results looks good. • Any methods used to normalize the data or exclude outliers must also be decided in advance. It isn't ok to analyze various subsets of the data until you find results you like. • And finally, of course, the statistical methods must be decided in advance. Is it not ok to analyze the data via parametric and nonparametric methods, and pick the results you like. Exploratory methods can be useful to, well, explore. But then you can't turn around and run regular statistical tests and interpret the results in the usual way. • I think John Tukey might disagree en.wikipedia.org/wiki/Exploratory_data_analysis ;o) – Dikran Marsupial Jan 27 '11 at 17:32 • I would partially disagree here. I think the caveat that people miss is that the appropriate conditioning operations are easy to ignore for these kinds of issues. Each of these operations change the conditions of the inference, and hence, they change the conditions of it applicability (and therefore to its generality). These is definitely only applicable to "confirmatory analysis", where a well defined model and question have been constructed. In exploratory phase, not looking to answer definite questions - more looking to build a model and come up with hypothesis for the data. – probabilityislogic Jan 27 '11 at 18:05 • I edited my answer a bit to take into account the comments of Dikran and probabilityislogic. Thanks. – Harvey Motulsky Jan 28 '11 at 14:52 • For me, the "excluding outliers" is not as clearly wrong as your answer implies. For example, you may only be interested in the relationships at a certain range of responses, and excluding outliers actually helps this kind of analysis. For example, if you want to model "middle class" income, then excluding the super rich and impoverished outliers is a good idea. It is only the outliers within your frame of inference (e.g. "strange" middle class observations) were your comments apply – probabilityislogic Jan 29 '11 at 6:10 • Ultimately the real problem with the issues raised in the initial answer is that they (at least partially) invalidate p-values. If you are interested in quantifying an observed effect, one should be able to do any and all of the above with impunity. – russellpierce Jan 29 '11 at 19:24 Tongue firmly in cheek: For frequentists, the Bayesian concept of probability; for Bayesians, the frequentist concept of probability. ;o) Both have merit of course, but it can be very difficult to understand why one framework is interesting/useful/valid if your grasp of the other is too firm. Cross-validated is a good remedy as asking questions and listening to answers is a good way to learn. • I rule I use to remember: Use probabilities to predict frequencies. Once the frequencies have been observed, use them to evaluate the probabilities you assigned. The unfortunately confusing thing is that, often the probability you assign is equal to a frequency you have observed. One thing I have always found odd is why do frequentists even use the word probability? wouldn't it make their concepts easier to understand if the phrase "the frequency of an event" was used instead of "the probability of an event"? – probabilityislogic Jan 28 '11 at 16:58 • Interestingly, cross validation can be seen as a Monte Carlo approximation to the integral of a loss function in Decision Theory. You have an integral$\int p(x) L(\textbf{x}_{n},x) dx$and you approximate it by$\sum_{i=1}^{i=n} L(\textbf{x}_{[n-i]},x_i)$Where$\textbf{x}_{n}$is data vector, and$\textbf{x}_{[n-i]}$is the data vector with the ith observation$x_i\$ removed – probabilityislogic Jan 30 '11 at 1:30

From my personal experience the concept of likelihood can also cause quite a lot of stir, especially for non-statisticians. As wikipedia says, it is very often mixed up with the concept of probability, which is not exactly correct.

Fiducial inference. Even Fisher admitted he didn't understand what it does, and he invented it.

What do the different distributions really represent, besides than how they are used.

• This was the question I found most distracting after statistics 101. I would encounter many distributions with no motivation for them beyond "properties" that were relevant to topics at hand. It took unacceptably long to find out what any represented. – sesqu Jan 29 '11 at 22:12
• Maximum entropy "thinking" is one method which helps understand what a distribution is, namely a state of knowledge (or a description of uncertainty about something). This is the only definition that has made sense to me in all situations – probabilityislogic Jan 30 '11 at 4:57
• Ben Bolker provides a good overview of this in the 'beastiary of distributions' section of Ecological Models and Data in R – David LeBauer Sep 23 '11 at 5:04

I think the question is interpretable in two ways, which will give very different answers:

1) For people studying statistics, particularly at a relatively advanced level, what is the hardest concept to grasp?

2) Which statistical concept is misunderstood by the most people?

For 1) I don't know the answer at all. Something from measure theory, maybe? Some type of integration? I don't know.

For 2) p-value, hands down.

• Measure theory is neither a field of statistics nor hard. Some types of integration are hard, but, once again, that isn't statistics. – pyon Jan 29 '11 at 22:12

Confidence interval in non-Bayesian tradition is a difficult one.

I think people miss the boat on pretty much everything the first time around. I think what most students don't understand is that they're usually estimating parameters based on samples. They don't know the difference between a sample statistic and a population parameter. If you beat these ideas into their head, the other stuff should follow a little bit easier. I'm sure most students don't understand the crux of the CLT either.