72
votes
7answers
24k views

Algorithms for automatic model selection

I would like to implement an algorithm for automatic model selection. I am thinking of doing stepwise regression but anything will do (it has to be based on linear regressions though). My problem ...
138
votes
15answers
36k views

Is normality testing 'essentially useless'?

A former colleague once argued to me as follows: "we usually apply normality tests to the results of processes that, under the null, generate random variables that are only asymptotically or ...
98
votes
14answers
259k views

What is the meaning of p values and t values in statistical tests?

After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
69
votes
5answers
18k views

Can a probability distribution value exceeding 1 be OK?

On the Wikipedia page about naive Bayes classifiers, there is this line: $p(\mathrm{height}|\mathrm{male}) = 1.5789$ (A probability distribution over 1 is OK. It is the area under the bell curve ...
133
votes
7answers
114k views

Difference between logit and probit models

What is the difference between Logit and Probit model? I'm more interested here in knowing when to use logistic regression, and when to use Probit. If there is any literature which defines it using ...
57
votes
2answers
12k views

What if residuals are normally distributed, but y is not?

I've got a weird question. Assume that you have a small sample where the dependent variable that you're going to analyze with a simple linear model is highly left skewed. Thus you assume that $u$ is ...
275
votes
27answers
109k views

Making sense of principal component analysis, eigenvectors & eigenvalues

In today's pattern recognition class my professor talked about PCA, eigenvectors & eigenvalues. I got the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like a ...
95
votes
7answers
70k views

When conducting multiple regression, when should you center your predictor variables & when should you standardize them?

In some literature, I have read that a regression with multiple explanatory variables, if in different units, needed to be standardized. (Standardizing consists in subtracting the mean and dividing ...
55
votes
6answers
34k views

How can a regression be significant yet all predictors be non-significant?

My multiple regression analysis model has a statistically significant F value however all beta values are statistically non-significant. All the regression assumptions are met. No multicollinearity ...
138
votes
6answers
12k views

Is $R^2$ useful or dangerous?

I was skimming through some lecture notes by Cosma Shalizi (in particular, section 2.1.1 of the second lecture), and was reminded that you can get very low $R^2$ even when you have a completely linear ...
105
votes
7answers
58k views

How to understand degrees of freedom?

From Wikipedia, there are three interpretations of the degrees of freedom of a statistic: In statistics, the number of degrees of freedom is the number of values in the final calculation of a ...
75
votes
6answers
116k views

In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?

Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
19
votes
2answers
2k views

Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?

The coefficient of an explanatory variable in a multiple regression tells us the relationship of that explanatory variable with the dependent variable. All this, while 'controlling' for the other ...
65
votes
3answers
15k views

Assessing approximate distribution of data based on a histogram

Suppose I want to see whether my data is exponential based on a histogram (i.e. skewed to the right). Depending on how I group or bin the data, I can get wildly different histograms. One set of ...
114
votes
2answers
57k views

Interpretation of R's lm() output

the help pages in R assume I know what those numbers mean. I don't :) I'm trying to really intuitively understand every number here. I will just post the output and comment on what I found out. There ...
34
votes
4answers
7k views

How can adding a 2nd IV make the 1st IV significant?

I have what is probably a simple question, but it is baffling me right now, so I am hoping you can help me out. I have a least squares regression model, with one independent variable and one ...
54
votes
4answers
24k views

How to deal with perfect separation in logistic regression?

If you have a variable which perfectly separates zeroes and ones in target variable, R will yield the following "perfect or quasi perfect separation" warning message: ...
19
votes
2answers
15k views

Interpretation of log transformed predictor

I'm wondering if it makes a difference in interpretation whether only the dependent, both the dependent and independent, or only the independent variables are log transformed. In the case of ...
32
votes
4answers
22k views

Best practice when analysing pre-post treatment-control designs

Imagine the following common design: 100 participants are randomly allocated to either a treatment or a control group the dependent variable is numeric and measured pre- and post- treatment Three ...
93
votes
10answers
62k views

What are the differences between Factor Analysis and Principal Component Analysis?

It seems that a number of the statistical packages that I use wrap these two concepts together. However, I'm wondering if there are different assumptions or data 'formalities' that must be true to use ...
47
votes
12answers
24k views

Including the interaction but not the main effects in a model

Is it ever valid to include a two-way interaction in a model without including the main effects? What if your hypothesis is only about the interaction, do you still need to include the main effects?
22
votes
5answers
4k views

What is the benefit of breaking up a continuous predictor variable?

I'm wondering what the value is in taking a continuous predictor variable and breaking it up (e.g., into quintiles), before using it in a model. It seems to me that by binning the variable we lose ...
92
votes
9answers
48k views

How should I transform non-negative data including zeros?

If I have highly skewed positive data I often take logs. But what should I do with highly skewed non-negative data that include zeros? I have seen two transformations used: $\log(x+1)$ which has the ...
150
votes
137answers
47k views

Famous statistical quotations

What is your favorite statistical quote? This is community wiki, so please one quote per answer.
44
votes
3answers
2k views

Is there an intuitive interpretation of $A^TA$?

For a given data matrix $A$, it seems like $A^TA$ plays an important role in statistics. For example, it is an important part of the analytical solution of ordinary least squares. Or, for PCA, its ...
174
votes
20answers
68k views

Why square the difference instead of taking the absolute value in standard deviation?

In the definition of standard deviation, why do we have to square the difference from the mean to get the mean (E) and take the square root back at the end? Can't we just simply take the absolute ...
41
votes
7answers
24k views

When is it ok to remove the intercept in lm()?

I am running linear regression models and wondering what the conditions are for removing the intercept term of lm()? In comparing results from two different lm ...
14
votes
2answers
4k views

Interpretation of simple predictions to odds ratios in logistic regression

I'm somewhat new to using logistic regression, and a bit confused by a discrepancy between my interpretations of the following values which I thought would be the same: exponentiated beta values ...
28
votes
5answers
52k views

What is the difference between linear regression on y with x and x with y?

The Pearson correlation coefficient of x and y is the same, whether you compute pearson(x, y) or pearson(y, x). This suggests that doing a linear regression of y given x or x given y should be the ...
21
votes
2answers
5k views

Simulation of logistic regression power analysis - designed experiments

This question is in response to an answer given by @Greg Snow in regards to a question I asked concerning power analysis with logistic regression and SAS ...
5
votes
1answer
4k views

Converting similarity matrix to (euclidean) distance matrix

In Random forest algorithm, Breiman (author) constructs similarity matrix as follows: Send all learning examples down each tree in the forest If two examples land in the same leaf increment ...
24
votes
7answers
3k views

What is meant by a “random variable”?

What do they mean when they say "random variable"?
11
votes
3answers
10k views

Using principal component analysis (PCA) for feature selection

I'm new to feature selection and I was wondering how you would use PCA to perform feature selection. Does PCA compute a relative score for each input variable that you can use to filter out ...
16
votes
4answers
5k views

Why do statisticians say a non-significant result means “you can't reject the null” as opposed to accepting the null hypothesis?

Traditional statistical tests, like the two sample t-test, focus on trying to eliminate the hypothesis that there is no difference between a function of two independent samples. Then, we choose a ...
64
votes
1answer
13k views

R's lmer cheat-sheet

There's a lot of discussion going on on this forum about the proper way to specify various hierarchical models using lmer. I thought it would be great to have all ...
112
votes
8answers
15k views

Why does a 95% CI not imply a 95% chance of containing the mean?

It seems that through various related questions here, there is consensus that the "95%" part of what we call a "95% confidence interval" refers to the fact that if we were to exactly replicate our ...
31
votes
8answers
34k views

Which pseudo-$R^2$ measure is the one to report for logistic regression (Cox & Snell or Nagelkerke)?

I have SPSS output for a logistic regression model. The output reports two measures for the model fit, Cox & Snell and ...
28
votes
6answers
4k views

How to choose between t-test or non-parametric test e.g. Wilcoxon in small samples

Certain hypotheses can be tested using Student's t-test (maybe using Welch's correction for unequal variances in the two-sample case), or by a non-parametric test like the Wilcoxon paired signed rank ...
28
votes
2answers
15k views

Does an unbalanced sample matter when doing logistic regression?

Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables. My problem is the following: ...
28
votes
4answers
30k views

Under what conditions should Likert scales be used as ordinal or interval data?

Many studies in the social sciences use Likert scales. When is it appropriate to use Likert data as ordinal and when is it appropriate to use it as interval data?
102
votes
10answers
36k views

Is there any reason to prefer the AIC or BIC over the other?

The AIC and BIC are both methods of assessing model fit penalized for the number of estimated parameters. As I understand it, BIC penalizes models more for free parameters than does AIC. Beyond a ...
30
votes
1answer
3k views

Relationship between SVD and PCA. How to use SVD to perform PCA?

Principal component analysis (PCA) is usually explained via an eigen-decomposition of the covariance matrix. However, it can also be performed via singular value decomposition (SVD) of the data matrix ...
22
votes
4answers
8k views

How to generate correlated random numbers (given means, variances and degree of correlation)?

I'm sorry if this seems a bit too basic, but I guess I'm just looking to confirm understanding here. I get the sense I'd have to do this in two steps, and I've started trying to grok correlation ...
8
votes
1answer
7k views

How to interpret type I (sequential) ANOVA and MANOVA?

My primary question is how to interpret the output (coefficients, F, P) when conducting a Type I (sequential) ANOVA? My specific research problem is a bit more complex, so I will break my example into ...
51
votes
4answers
71k views

How exactly does one “control for other variables”?

Here is the article that motivated this question: Does impatience make us fat? I liked this article, and it nicely demonstrates the concept of “controlling for other variables” (IQ, career, income, ...
31
votes
6answers
10k views

What is a complete list of the usual assumptions for linear regression?

What are the usual assumptions for linear regression? Do they include: a linear relationship between the independent and dependent variable independent errors normal distribution of errors ...
49
votes
7answers
5k views

What, precisely, is a confidence interval?

I know roughly and informally what a confidence interval is. However, I can't seem to wrap my head around one rather important detail: According to Wikipedia: A confidence interval does not ...
37
votes
2answers
5k views

Is it possible to have a pair of Gaussian random variables for which the joint distribution is not Gaussian?

Somebody asked me this question in a job interview and I replied that their joint distribution is always Gaussian. I thought that I can always write a bivariate Gaussian with their means and variance ...
19
votes
4answers
11k views

What if interaction wipes out my direct effects in regression?

In a regression, the interaction term wipes out both related direct effects. Do I drop the interaction or report the outcome? The interaction was not part of the original hypothesis.
62
votes
13answers
18k views

What is the best way to identify outliers in multivariate data?

Suppose I have a large set of multivariate data with at least three variables. How can I find the outliers? Pairwise scatterplots won't work as it is possible for an outlier to exist in 3 dimensions ...

15 30 50 per page