198
votes
21answers
63k views

Python as a statistics workbench

Lots of people use a main tool like Excel or another spreadsheet, SPSS, Stata, or R for their statistics needs. They might turn to some specific package for very special needs, but a lot of things can ...
189
votes
25answers
70k views

Making sense of principal component analysis, eigenvectors & eigenvalues

In today's pattern recognition class my professor talked about PCA, eigenvectors & eigenvalues. I got the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like a ...
161
votes
56answers
69k views

What is your favorite “data analysis” cartoon?

This is one of my favorites: One entry per answer. This is in the vein of the Stack Overflow question What’s your favorite “programmer” cartoon?. P.S. Do not hotlink the cartoon without the site's ...
157
votes
15answers
32k views

The Two Cultures: statistics vs. machine learning?

Last year, I read a blog post from Brendan O'Connor entitled "Statistics vs. Machine Learning, fight!" that discussed some of the differences between the two fields. Andrew Gelman responded favorably ...
125
votes
136answers
36k views

Famous statistician quotes

What is your favorite statistician quote? This is community wiki, so please one quote per answer.
125
votes
20answers
43k views

Why square the difference instead of taking the absolute value in standard deviation?

In the definition of standard deviation, why do we have to square the difference from the mean to get the mean (E) and take the square root back at the end? Can't we just simply take the absolute ...
122
votes
42answers
11k views

What are common statistical sins?

I'm a grad student in psychology, and as I pursue more and more independent studies in statistics, I am increasingly amazed by the inadequacy of my formal training. Both personal and second hand ...
110
votes
8answers
26k views

Is Facebook coming to an end?

Recently, this paper has received a lot of attention (e.g. from WSJ). Basically, the authors conclude that Facebook will lose 80% of its members by 2017. They base their claims on an extrapolation ...
108
votes
6answers
9k views

Is $R^2$ useful or dangerous?

I was skimming through some lecture notes by Cosma Shalizi (in particular, section 2.1.1 of the second lecture), and was reminded that you can get very low $R^2$ even when you have a completely linear ...
105
votes
15answers
24k views

Bayesian and frequentist reasoning in plain English

How would you describe in plain English the characteristics that distinguish Bayesian from Frequentist reasoning?
100
votes
14answers
31k views

Amazon interview question—probability of 2nd interview

I got this question during an interview with Amazon: 50% of all people who receive a first interview receive a second interview 95% of your friends that got a second interview felt they had a good ...
98
votes
7answers
40k views

What is the difference between “likelihood” and “probability”?

The wikipedia page claims that likelihood and probability are distinct concepts. In non-technical parlance, "likelihood" is usually a synonym for "probability," but in statistical usage there is a ...
97
votes
13answers
23k views

Is normality testing 'essentially useless'?

A former colleague once argued to me as follows: "we usually apply normality tests to the results of processes that, under the null, generate random variables that are only asymptotically or ...
90
votes
5answers
61k views

Difference between logit and probit models

What is the difference between the logit and the probit model? I'm more interested here in knowing when to use logistic regression, and when to use probit. If there's any literature which define it ...
90
votes
6answers
13k views

Explaining to laypeople why bootstrapping works

I recently used bootstrapping to estimate confidence intervals for a project. Someone who doesn't know much about statistics recently asked me to explain why bootstrapping works, i.e., why is it that ...
90
votes
8answers
8k views

Why does a 95% CI not imply a 95% chance of containing the mean?

It seems that through various related questions here, there is consensus that the "95%" part of what we call a "95% confidence interval" refers to the fact that if we were to exactly replicate our ...
89
votes
5answers
19k views

What is the intuition behind beta distribution?

Disclaimer: I'm not a statistician but a software engineer. Most of my knowledge in statistics comes from self-education, thus I still have many gaps in understanding concepts that may seem trivial ...
89
votes
8answers
13k views

Detecting a given face in a database of facial images

I'm working on a little project involving the faces of twitter users via their profile pictures. A problem I've encountered is that after I filter out all but the images that are clear portrait ...
81
votes
10answers
23k views

Is there any reason to prefer the AIC or BIC over the other?

The AIC and BIC are both methods of assessing model fit penalized for the number of estimated parameters. As I understand it, BIC penalizes models more for free parameters than does AIC. Beyond a ...
80
votes
16answers
28k views

Does Julia have any hope of sticking in the statistical community?

I recently read a post from R-Bloggers, that linked to this blog post from John Myles White about a new language called Julia. Julia takes advantage of a just-in-time compiler that gives it wicked ...
78
votes
27answers
31k views

What is the best introductory Bayesian statistics textbook?

Which is the best introductory textbook for Bayesian statistics? One book per answer, please.
77
votes
1answer
33k views

Interpretation of R's lm() output

the help pages in R assume I know what those numbers mean. I don't :) I'm trying to really intuitively understand every number here. I will just post the output and comment on what I found out. There ...
76
votes
25answers
7k views

Locating freely available data samples

I've been working on a new method for analyzing and parsing datasets to identify and isolate subgroups of a population without foreknowledge of any subgroup's characteristics. While the method works ...
74
votes
7answers
26k views

What's the difference between a confidence interval and a credible interval?

Joris and Srikant's exchange here got me wondering (again) if my internal explanations for the the difference between confidence intervals and credible intervals were the correct ones. How you would ...
73
votes
16answers
5k views

How to annoy a statistical referee?

I recently asked a question regarding general principles around reviewing statistics in papers. What I would now like to ask, is what particularly irritates you when reviewing a paper, i.e. what's the ...
71
votes
58answers
71k views

Statistics Jokes

Well we've got favourite statistics quotes. What about statistics jokes? So, what's your favourite statistics joke?
70
votes
12answers
39k views

What is the difference between data mining, statistics, machine learning and AI?

What is the difference between data mining, statistics, machine learning and AI? Would it be accurate to say that they are 4 fields attempting to solve very similar problems but with different ...
69
votes
9answers
97k views

How to summarize data by group in R?

I have R data frame like this: ...
68
votes
9answers
31k views

How should I transform non-negative data including zeros?

If I have highly skewed positive data I often take logs. But what should I do with highly skewed non-negative data that include zeros? I have seen two transformations used: log(x+1) which has the ...
68
votes
10answers
184k views

What is the meaning of p values and t values in statistical tests?

After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
67
votes
21answers
35k views

R vs SAS, why is SAS preferred by private companies?

I learned R but it seems that companies are much more interested in SAS experience. What are the advantages of SAS over R?
67
votes
21answers
15k views

Resources for learning R [duplicate]

I'm interested in learning R on the cheap. What's the best free resource/book/tutorial for learning R?
67
votes
4answers
12k views

Which “mean” to use and when?

So we have arithmetic mean (AM), geometric mean (GM) and harmonic mean (HM). Their mathematical formulation is also well known along with their associated stereotypical examples (e.g., Harmonic mean ...
66
votes
5answers
6k views

What skills are required to perform large scale statistical analyses?

Many statistical jobs ask for experience with large scale data. What are the sorts of statistical and computational skills that would be need for working with large data sets. For example, how about ...
65
votes
9answers
37k views

What are the differences between Factor Analysis and Principal Component Analysis

It seems that a number of the statistical packages that I use wrap these two concepts together. However, I'm wondering if there are different assumptions or data 'formalities' that must be true to use ...
64
votes
6answers
32k views

How to understand degrees of freedom?

From Wikipedia, there are three interpretations of the degrees of freedom of a statistic: In statistics, the number of degrees of freedom is the number of values in the final calculation of a ...
63
votes
6answers
15k views

Does causation imply correlation?

Correlation does not imply causation, as there could be many explanations for the correlation. But does causation imply correlation? Intuitively, I would think that the presence of causation means ...
62
votes
6answers
6k views

Why is Euclidean distance not a good metric in high dimensions?

I read that 'Euclidean distance is not a good distance in high dimensions'. I guess this statement has something to do with the curse of dimensionality, but what exactly? Besides, what is 'high ...
62
votes
5answers
2k views

Essential data checking tests

In my job role I often work with other people's datasets, non-experts bring me clinical data and I help them to summarise it and perform statistical tests. The problem I am having is that the ...
62
votes
6answers
8k views

How to efficiently manage a statistical analysis project?

We often hear of project management and design patterns in computer science, but less frequently in statistical analysis. However, it seems that a decisive step toward designing an effective and ...
61
votes
16answers
16k views

Most interesting statistical paradoxes

Because I find them fascinating, I'd like to hear what folks in this community find as the most interesting statistical paradox and why.
60
votes
11answers
10k views

What's wrong with XKCD's Frequentists vs. Bayesians comic?

This xkcd comic (Frequentists vs. Bayesians) makes fun of a frequentist statistician who derives an obviously wrong result. However it seems to me that his reasoning is actually correct in the ...
59
votes
8answers
26k views
58
votes
6answers
6k views

Model for predicting number of Youtube views of Gangnam Style

PSY's music video "Gangnam style" is popular, after a little more than 2 months it has about 540 million viewers. I learned this from my preteen children at dinner last week and soon the discussion ...
57
votes
19answers
8k views

Free statistical textbooks

Are there any free statistical textbooks available?
55
votes
19answers
4k views

What are some valuable Statistical Analysis open source projects?

What are some valuable Statistical Analysis open source projects available right now? Edit: as pointed out by Sharpie, valuable could mean helping you get things done faster or more cheaply.
55
votes
8answers
37k views

When should you center your data & when should you standardize?

In some literature, I have read that a regression with multiple explanatory variables, if in different units, needed to be standardized. (Standardizing consists in subtracting the mean and dividing ...
55
votes
6answers
72k views

In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?

Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
55
votes
11answers
10k views

Are large data sets inappropriate for hypothesis testing?

In a recent article of Amstat News, the authors (Mark van der Laan and Sherri Rose) stated that "We know that for large enough sample sizes, every study—including ones in which the null hypothesis of ...
55
votes
3answers
13k views

When should I use lasso vs ridge?

Say I want to estimate a large number of parameters, and I want to penalize some of them because I believe they should have little effect compared to the others. How do I decide what penalization ...

15 30 50 per page