Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am an econ/stat major. I am aware that economists have tried to modify their assumptions about human behavior and rationality by identifying situations in which people don't behave rationally. For example, suppose I offer you a 100% chance of a \$1000 loss or a 50% chance at a \$2500 loss, people choose the \$2500 option even though the expected value of the latter is a greater loss than a \$1000 guaranteed loss. This is known as "loss aversion". Behavioral economists now study these patterns and try to identify ways humans deviate from those axioms normally assumed to constitute "rational" behavior. Here, I assume it is rational to prefer the least expected loss.

I was wondering if statisticians have identified common patterns in data collection that yield biased results in how people interpret data. If there was essentially a "rational" way to collect data, I assume there are examples where humans deviate from this and exhibit "bias". If so, what are the most common biases humans make when collecting or interpreting data?

share|improve this question
4  
There is a great paper by Podsakoff et al. that reviews the common method biases and proposes both statistical and procedural remedies: ln.edu.hk/mkt/staff/gcui/CommonMethodBias.pdf Have a look at Table 2. – ayhan Feb 28 at 11:17
    
Related: stats.stackexchange.com/questions/4551/… – CL. Feb 28 at 11:51
9  
You have an irrational presumption of irrationality. You are failing to apply a utility function to the outcome. In your stated example, presume that the person has \$1000 and must use it to repay a loan to a gangster a minute from now, or will be killed by the gangster. The 100% chance of \$1000 loss results in 100% chance of being killed, whereas the 50% chance of \$2500 loss only results in 50% chance of being killed. As an econ major, you should be attuned to utility as a starting point before declaring irrationality. – Mark L. Stone Feb 28 at 13:48
2  
Statisticians don't typically do that kind of research. I wonder if this Q is more appropriate for the Cognitive Sciences SE site. – gung Feb 28 at 14:39
3  
I think streetlight effect - looking for lost keys (data) under the lightpost because that's where the light is at night - is extremely common, especially now, with so much easy data on tap. // There is no "rational way" to collect data because you - the researcher collecting it - is not randomized. – A.S. Feb 28 at 23:43
up vote 23 down vote accepted

I think in academia, p-values are very commonly misinterpreted. People tend to forget that the p-value expresses a conditional probability. Even if an experiment has been perfectly conducted and all requisites of the chosen statistical test are met, the false discovery rate is typically much higher than the significance level alpha. The false discovery rate increases with a decrease in statistical power and prevalence of true positives (Colquhoun, 2014; Nuzzo, 2014).

In addition people tend to regard their estimates as the truth and the parameter they estimate as random (Haller & Kraus, 2002). For example when they say that in “95% of the cases this identified confidence interval covers the parameter”...

Confusion of correlation and causation is probably also a very common error in data interpretation.

In terms of data collection, I think a common error is to take the most easily accessible rather than the most representative sample.

Colquhoun, D. (2014). An investigation of the false discovery rate and the misinterpretation of P values. Royal Society Open Science, 1–15.

Nuzzo, R. (2014). Statistical errors: P values, the “gold standard” of statistical validity are not as reliable as many scientists assume. Nature, 506, 150–152.

Haller, H. & Kraus, S. (2002): Misinterpretations of Significance: A Problem Students Share with Their Teachers? Methods of Psychological Research Online, Vol.7, No.1

share|improve this answer
1  
Thanks for adding links to my references! :-) – bee guy Mar 3 at 10:48

I would say a general inability to appreciate what true randomness looks like. People seem to expect too few spurious patterns than actually occur in sequences of random events. This also shows up when we try to simulate randomness on our own.

Another fairly common one is not understanding independence, as in the gambler's fallacy. We sometimes think that prior events can affect future ones even when it's clearly impossible, like the previous deal of a shuffled deck of cards impacting a future one.

share|improve this answer

It has already been pointed out that many of the behaviors and thought processes labeled "irrational" or "biased" by (behavioral) economists are actually highly adaptive and efficient in the real world. Nonetheless, OP's question is interesting. I think, however, that it may be profitable to refer to more fundamental, descriptive knowledge about our cognitive processes, rather than go looking for specific "biases" that correspond to those discussed in the economic literature (e.g., loss aversion, endowment effect, baserate neglect etc.).

For instance, evaluability is certainly an issue in data analysis. Evaluability theory states that we overweight information that we find easy to interpret or evaluate. Consider the case of a regression coefficient. Evaluating the "real-world" consequences of a coefficient can be hard work. We need to consider the units of the independent and the dependent variable as well was the distributions of our independent and dependent variable to understand whether a coefficient has practical relevance. Evaluating the significance of a coefficient, on the other hand, is easy: I merely compare its p-value to my alpha level. Given the greater evaluability of the p-value compared the coefficient itself, it is scarcely surprising that so much is made of p-values.

(Standardizing increases the evaluability of a coefficient, but it may increase ambiguity: the sense that relevant information is unavailable or withheld, because the "original" form of the data we are processing is not available to us.)

A related cognitive "bias" is the concreteness principle, the tendency to overweight information that is "right there" in a decision context, and does not require retrieval from memory. (The concreteness principle also states that we are likely to use information in the format in which it is given and tend to avoid performing transformations.) Interpreting a p-value can be done by merely looking at the regression output; it does not require me to retrieve any substantive knowledge about the thing that I am modeling.

I expect that many biases in the interpretation of statistical data can be traced to the general understanding that we are likely to take the easy route when solving a problem or forming a judgment (see "cognitive miser", "bounded rationality" and so on). Relatedly, doing something "with ease" usually increases the confidence with which we hold the resulting beliefs (fluency theory). (One might also consider the possibility that data which are easier to articulate - to ourselves or to others - are overweighted in our analyses.) I think this becomes particularly interesting when we consider possible exceptions. Some psychological research suggests, for instance, that if we believe that a problem should be difficult to solve, then we may favor approaches and solutions which are less concrete and more difficult, e.g., choose a more arcane method over a simple one.

share|improve this answer

Linearity.

I think a common bias during data interpretation/analysis is that people usually are quick to assume linear relations. Mathematically, a regression model assumes that its deterministic component is a linear function of the predictors; unfortunately that is not always true. I recently went to an undergraduate poster conference and the amount of bluntly quadratic or non-linear trends I saw being fitted with a linear model was worrying to say the least.

(This is in addition to the mentions of gambler's fallacy, $p$-value misinterpretation and true randomness; +1 to all relevant posts.)

share|improve this answer

The biggest single factor I can think of is broadly known as "confirmation bias". Having settled upon what I think my study will show, I uncritically accept data that lead to that conclusion, while making excuses for all data points that appear to refute it. I may unconsciously reject as "obvious instrument error" (or some equivalent) any data points that don't fit my conclusion. In some cases, it won't be quite as blatant; rather than throwing out those data points entirely, I'll concoct some formula to remove the "error", which will conveniently steer the results toward confirming my preordained conclusion.

There's nothing particularly nefarious about this; it's just how our brains work. It takes a great deal of effort to filter out such bias, and it's one of the reasons why scientists like to concoct double-blind studies, such that the person performing the measurements does not know what the experiment is expected to prove. It then requires enormous discipline not to adjust away what he has faithfully measured.

share|improve this answer
    
I think this is the most dangerous bias actually because it can already occur in the data collection stage, e.g. collecting data in a tiny subsample that is most likely to confirm your expectations or using leading survey questions. – stijn Mar 1 at 5:59
    
The confirmation biases can be really bad between disciplines, where even the alleged fundamental basis of the disciplines are different, leading assertions that "X is impossible in (using) your discipline (with its sensing methods), but is obvious in mine (we can sense X)". e.g. Apples are meant to hang in trees, or lie on the ground; they can't 'fall' of their own accord. Often, in the physical sciences, there is a change of mathematical basis that hides the confusion. – Philip Oakley Mar 2 at 10:43

An intersting case is the discussions of the Gamblers Fallacy.

Should the existing data be included or exluded? If I am already ahead with 6 sixes, are these to be included in my run of a dozen tries? Be clear about prior data.

When should I change from absolute numbers to ratio's? It takes a long time for the advantage gained during an intial winning streak to return to zero (a random walk).

0.1% of a million dollars may not be much to a big company, but to loose $1000 could be life and death to a sole trader (which is why investors want 'driven' people to invest in). Being able to shift to percentages can be a bias.

Even statisticians have biases.

share|improve this answer

I would recommend "Thinking, Fast and Slow" by Daniel Kahneman, which explains many cognitive biases in lucid language.

You may also refer to "http://www.burns-stat.com/review-thinking-fast-slow-daniel-kahneman/" which summarizes some of the biases in the above book.

For more detailed chapter wise summary you may want to read "https://erikreads.files.wordpress.com/2014/04/thinking-fast-and-slow-book-summary.pdf".

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.