# All Questions

152k views

### Making sense of principal component analysis, eigenvectors & eigenvalues

In today's pattern recognition class my professor talked about PCA, eigenvectors & eigenvalues. I got the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like a ...
102k views

### Python as a statistics workbench

Lots of people use a main tool like Excel or another spreadsheet, SPSS, Stata, or R for their statistics needs. They might turn to some specific package for very special needs, but a lot of things can ...
76k views

### The Two Cultures: statistics vs. machine learning?

Last year, I read a blog post from Brendan O'Connor entitled "Statistics vs. Machine Learning, fight!" that discussed some of the differences between the two fields. Andrew Gelman responded favorably ...
111k views

### What is your favorite “data analysis” cartoon?

This is one of my favorites: One entry per answer. This is in the vein of the Stack Overflow question What’s your favorite “programmer” cartoon?. P.S. Do not hotlink the cartoon without the site's ...
98k views

### Why square the difference instead of taking the absolute value in standard deviation?

In the definition of standard deviation, why do we have to square the difference from the mean to get the mean (E) and take the square root back at the end? Can't we just simply take the absolute ...
62k views

### What is the intuition behind beta distribution?

Disclaimer: I'm not a statistician but a software engineer. Most of my knowledge in statistics comes from self-education, thus I still have many gaps in understanding concepts that may seem trivial ...
90k views

### What is the difference between “likelihood” and “probability”?

The wikipedia page claims that likelihood and probability are distinct concepts. In non-technical parlance, "likelihood" is usually a synonym for "probability," but in statistical usage there is a ...
35k views

### How to understand the drawbacks of K-means

K-means is a widely used method in cluster analysis. In my understanding, this method does NOT require ANY assumptions, i.e., give me a dataset and a pre-specified number of clusters, k, and I just ...
71k views

### Bayesian and frequentist reasoning in plain English

How would you describe in plain English the characteristics that distinguish Bayesian from Frequentist reasoning?
170k views

### Difference between logit and probit models

What is the difference between Logit and Probit model? I'm more interested here in knowing when to use logistic regression, and when to use Probit. If there is any literature which defines it using ...
15k views

### What are common statistical sins?

I'm a grad student in psychology, and as I pursue more and more independent studies in statistics, I am increasingly amazed by the inadequacy of my formal training. Both personal and second hand ...
59k views

51k views

### Is normality testing 'essentially useless'?

A former colleague once argued to me as follows: We usually apply normality tests to the results of processes that, under the null, generate random variables that are only asymptotically or ...
16k views

### Is $R^2$ useful or dangerous?

I was skimming through some lecture notes by Cosma Shalizi (in particular, section 2.1.1 of the second lecture), and was reminded that you can get very low $R^2$ even when you have a completely linear ...
55k views

### Explaining to laypeople why bootstrapping works

I recently used bootstrapping to estimate confidence intervals for a project. Someone who doesn't know much about statistics recently asked me to explain why bootstrapping works, i.e., why is it that ...
89k views

### How would you explain Markov Chain Monte Carlo (MCMC) to a layperson?

Maybe the concept, why it's used, and an example.
86k views

### Interpretation of R's lm() output

the help pages in R assume I know what those numbers mean. I don't :) I'm trying to really intuitively understand every number here. I will just post the output and comment on what I found out. There ...
88k views

### How to understand degrees of freedom?

From Wikipedia, there are three interpretations of the degrees of freedom of a statistic: In statistics, the number of degrees of freedom is the number of values in the final calculation of a ...
95k views

### What is the difference between data mining, statistics, machine learning and AI?

What is the difference between data mining, statistics, machine learning and AI? Would it be accurate to say that they are 4 fields attempting to solve very similar problems but with different ...
114k views

### How to choose the number of hidden layers and nodes in a feedforward neural network?

Is there a standard and accepted method for selecting the number of layers, and the number of nodes in each layer, in a FF NN? I'm interested in automated ways of building neural networks.
25k views

Suppose we have data set $(X_i,Y_i$) with $n$ points. We want to perform a linear regression, but first we sort the $X_i$ values and the $Y_i$ values independently of each other, forming data set $(... 9answers 24k views ### Why does a 95% CI not imply a 95% chance of containing the mean? It seems that through various related questions here, there is consensus that the "95%" part of what we call a "95% confidence interval" refers to the fact that if we were to exactly replicate our ... 33answers 62k views ### What is the best introductory Bayesian statistics textbook? Which is the best introductory textbook for Bayesian statistics? One book per answer, please. 8answers 51k views ### What's the difference between a confidence interval and a credible interval? Joris and Srikant's exchange here got me wondering (again) if my internal explanations for the the difference between confidence intervals and credible intervals were the correct ones. How you would ... 14answers 309k views ### What is the meaning of p values and t values in statistical tests? After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ... 9answers 123k views ### What is the difference between test set and validation set? I found this confusing when I use the neural network toolbox in Matlab. It divided the raw data set into three parts: training set validation set test set I notice in many training or learning ... 7answers 108k views ### When conducting multiple regression, when should you center your predictor variables & when should you standardize them? In some literature, I have read that a regression with multiple explanatory variables, if in different units, needed to be standardized. (Standardizing consists in subtracting the mean and dividing ... 11answers 12k views ### What is a data scientist? Having recently graduated from my PhD program in statistics, I had for the last couple of months began searching for work in the field of statistics. Almost every company I considered had a job ... 10answers 56k views ### Is there any reason to prefer the AIC or BIC over the other? The AIC and BIC are both methods of assessing model fit penalized for the number of estimated parameters. As I understand it, BIC penalizes models more for free parameters than does AIC. Beyond a ... 7answers 16k views ### Why is Euclidean distance not a good metric in high dimensions? I read that 'Euclidean distance is not a good distance in high dimensions'. I guess this statement has something to do with the curse of dimensionality, but what exactly? Besides, what is 'high ... 18answers 52k views ### Does Julia have any hope of sticking in the statistical community? I recently read a post from R-Bloggers, that linked to this blog post from John Myles White about a new language called Julia. Julia takes advantage of a just-in-time compiler that gives it wicked ... 8answers 27k views ### Is Facebook coming to an end? Recently, this paper has received a lot of attention (e.g. from WSJ). Basically, the authors conclude that Facebook will lose 80% of its members by 2017. They base their claims on an extrapolation ... 14answers 49k views ### Amazon interview question—probability of 2nd interview I got this question during an interview with Amazon: 50% of all people who receive a first interview receive a second interview 95% of your friends that got a second interview felt they had a good ... 9answers 246k views ### How to summarize data by group in R? [closed] I have R data frame like this: ... 10answers 94k views ### What are the differences between Factor Analysis and Principal Component Analysis? It seems that a number of the statistical packages that I use wrap these two concepts together. However, I'm wondering if there are different assumptions or data 'formalities' that must be true to use ... 5answers 34k views ### Which “mean” to use and when? So we have arithmetic mean (AM), geometric mean (GM) and harmonic mean (HM). Their mathematical formulation is also well known along with their associated stereotypical examples (e.g., Harmonic mean ... 9answers 68k views ### How should I transform non-negative data including zeros? If I have highly skewed positive data I often take logs. But what should I do with highly skewed non-negative data that include zeros? I have seen two transformations used:$\log(x+1)$which has the ... 64answers 131k views ### Statistics Jokes Well, we've got favourite statistics quotes. What about statistics jokes? So, what's your favourite statistics joke? 22answers 79k views ### R vs SAS, why is SAS preferred by private companies? I learned R but it seems that companies are much more interested in SAS experience. What are the advantages of SAS over R? 7answers 148k views ### What is the difference between fixed effect, random effect and mixed effect models? In simple terms, how would you explain (perhaps with simple examples) the difference between fixed effect, random effect and mixed effect models? 8answers 14k views ### Detecting a given face in a database of facial images I'm working on a little project involving the faces of twitter users via their profile pictures. A problem I've encountered is that after I filter out all but the images that are clear portrait ... 7answers 37k views ### Algorithms for automatic model selection I would like to implement an algorithm for automatic model selection. I am thinking of doing stepwise regression but anything will do (it has to be based on linear regressions though). My problem ... 8answers 57k views ### How would you explain covariance to someone who understands only the mean? ...assuming that I'm able to augment their knowledge about variance in an intuitive fashion ( Understanding "variance" intuitively ) or by saying: It's the average distance of the data ... 8answers 160k views ### In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values? Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else? 19answers 6k views ### How to annoy a statistical referee? I recently asked a question regarding general principles around reviewing statistics in papers. What I would now like to ask, is what particularly irritates you when reviewing a paper, i.e. what's the ... 26answers 21k views ### Free statistical textbooks Are there any free statistical textbooks available? 26answers 28k views ### Locating freely available data samples I've been working on a new method for analyzing and parsing datasets to identify and isolate subgroups of a population without foreknowledge of any subgroup's characteristics. While the method works ... 2answers 269k views ### How do I get the number of rows of a data.frame in R? [closed] After reading a dataset: dataset <- read.csv("forR.csv") How can I get R to give me the number of cases it contains? Also, will the returned value include of ... 19answers 22k views ### The Sleeping Beauty Paradox The situation Some researchers would like to put you to sleep. Depending on the secret toss of a fair coin, they will briefly awaken you either once (Heads) or twice (Tails). After each waking, ... 5answers 29k views ### Can a probability distribution value exceeding 1 be OK? On the Wikipedia page about naive Bayes classifiers, there is this line:$p(\mathrm{height}|\mathrm{male}) = 1.5789\$ (A probability distribution over 1 is OK. It is the area under the bell curve ...

15 30 50 per page