91
votes
12answers
21k views

Is normality testing 'essentially useless'?

A former colleague once argued to me as follows: "we usually apply normality tests to the results of processes that, under the null, generate random variables that are only asymptotically or ...
39
votes
5answers
11k views

Algorithms for automatic model selection

I would like to implement an algorithm for automatic model selection. I am thinking of doing stepwise regression but anything will do (it has to be based on linear regressions though). My problem ...
43
votes
3answers
8k views

A Probability distribution value exceeding 1 is OK?

On the Wikipedia page about naive Bayes classifiers, there is this line: $p(\mathrm{height}|\mathrm{male}) = 1.5789$ (A probability distribution over 1 is OK. It is the area under the bell curve ...
86
votes
5answers
54k views

Difference between logit and probit models

What is the difference between the logit and the probit model? I'm more interested here in knowing when to use logistic regression, and when to use probit. If there's any literature which define it ...
41
votes
2answers
7k views

What if residuals are normally distributed, but y is not?

I've got a weird question. Assume that you have a small sample where the dependent variable that you're going to analyze with a simple linear model is highly left skewed. Thus you assume that $u$ is ...
184
votes
25answers
65k views

Making sense of principal component analysis, eigenvectors & eigenvalues

In today's pattern recognition class my professor talked about PCA, eigenvectors & eigenvalues. I got the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like a ...
65
votes
10answers
173k views

What is the meaning of p values and t values in statistical tests?

After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
42
votes
6answers
17k views

How can a regression be significant yet all predictors be non-significant?

My multiple regression analysis model has a statistically significant F value however all beta values are statistically non-significant. All the regression assumptions are met. No multicollinearity ...
43
votes
3answers
12k views

Assessing approximate distribution of data based on a histogram

Suppose I want to see whether my data is exponential based on a histogram (i.e. skewed to the right). Depending on how I group or bin the data, I can get wildly different histograms. One set of ...
23
votes
2answers
11k views

Best practice when analysing pre-post treatment-control designs

Imagine the following common design: 100 participants are randomly allocated to either a treatment or a control group the dependent variable is numeric and measured pre- and post- treatment Three ...
105
votes
6answers
9k views

Is $R^2$ useful or dangerous?

I was skimming through some lecture notes by Cosma Shalizi (in particular, section 2.1.1 of the second lecture), and was reminded that you can get very low $R^2$ even when you have a completely linear ...
55
votes
6answers
66k views

In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?

Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
51
votes
7answers
33k views

When should you center your data & when should you standardize?

In some literature, I have read that a regression with multiple explanatory variables, if in different units, needed to be standardized. (Standardizing consists in subtracting the mean and dividing ...
69
votes
1answer
30k views

Interpretation of R's lm() output

the help pages in R assume I know what those numbers mean. I don't :) I'm trying to really intuitively understand every number here. I will just post the output and comment on what I found out. There ...
66
votes
9answers
28k views

How should I transform non-negative data including zeros?

If I have highly skewed positive data I often take logs. But what should I do with highly skewed non-negative data that include zeros? I have seen two transformations used: log(x+1) which has the ...
64
votes
6answers
29k views

How to understand degrees of freedom?

From Wikipedia, there are three interpretations of the degrees of freedom of a statistic: In statistics, the number of degrees of freedom is the number of values in the final calculation of a ...
37
votes
12answers
14k views

Including the interaction but not the main effects in a model

Is it ever valid to include a two-way interaction in a model without including the main effects? What if your hypothesis is only about the interaction, do you still need to include the main effects?
17
votes
5answers
23k views

What is the difference between linear regression on y with x and x with y?

The Pearson correlation coefficient of x and y is the same, whether you compute pearson(x, y) or pearson(y, x). This suggests that doing a linear regression of y given x or x given y should be the ...
15
votes
2answers
6k views

Interpretation of log transformed predictor

I'm wondering if it makes a difference in interpretation whether only the dependent, both the dependent and independent, or only the independent variables are log transformed. In the case of ...
65
votes
9answers
34k views

What are the differences between Factor Analysis and Principal Component Analysis

It seems that a number of the statistical packages that I use wrap these two concepts together. However, I'm wondering if there are different assumptions or data 'formalities' that must be true to use ...
19
votes
3answers
3k views

How can adding a 2nd IV make the 1st IV significant?

I have what is probably a simple question, but it is baffling me right now, so I am hoping you can help me out. I have a least squares regression model, with one independent variable and one ...
122
votes
135answers
34k views

Famous statistician quotes

What is your favorite statistician quote? This is community wiki, so please one quote per answer.
28
votes
4answers
8k views

How to deal with perfect separation in logistic regression?

If you have a variable which perfectly separates zeroes and ones in target variable, R will yield the following "perfect or quasi perfect separation" warning message: ...
4
votes
1answer
2k views

Converting similarity matrix to (euclidean) distance matrix

In Random forest algorithm, Breiman (author) constructs similarity matrix as follows: Send all learning examples down each tree in the forest If two examples land in the same leaf increment ...
23
votes
4answers
22k views

Under what conditions should Likert scales be used as ordinal or interval data?

Many studies in the social sciences use Likert scales. When is it appropriate to use Likert data as ordinal and when is it appropriate to use it as interval data?
78
votes
10answers
22k views

Is there any reason to prefer the AIC or BIC over the other?

The AIC and BIC are both methods of assessing model fit penalized for the number of estimated parameters. As I understand it, BIC penalizes models more for free parameters than does AIC. Beyond a ...
25
votes
8answers
26k views

Which pseudo-$R^2$ measure is the one to report for logistic regression (Cox & Snell or Nagelkerke)?

I have a SPSS Output for a logistic regression. This output reports two measure for the model fit, Cox & Snell and ...
42
votes
7answers
4k views

What, precisely, is a confidence interval?

I know roughly and informally what a confidence interval is. However, I can't seem to wrap my head around one rather important detail: According to Wikipedia: A confidence interval does not ...
8
votes
1answer
2k views

How to choose between ANOVA and ANCOVA in a designed experiment?

I am conducting an experiment which has the following: DV: Slice consumption (continuous or could be categorical) IV: Healthy message, unhealthy message, no message (control) (3 groups in which ...
121
votes
19answers
38k views

Why square the difference instead of taking the absolute value in standard deviation?

In the definition of standard deviation, why do we have to square the difference from the mean to get the mean (E) and take the square root back at the end? Can't we just simply take the absolute ...
28
votes
3answers
989 views

Is there an intuitive interpretation of $A^TA$?

For a given data matrix $A$, it seems like $A^TA$ plays an important role in statistics. For example, it is an important part of the analytical solution of ordinary least squares. Or, for PCA, its ...
12
votes
2answers
3k views

Simulation of Logistic Regression Power Analysis - Designed Experiments

This question is in response to an answer given by @Greg Snow in regards to a question I asked concerning power analysis with logistic regression and SAS Proc GLMPOWER. If I am designing an ...
21
votes
2answers
8k views

Does an unbalanced sample matter when doing logistic regression?

Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables. My problem is the following: ...
16
votes
4answers
6k views

What if interaction wipes out my direct effects in regression?

In a regression, the interaction term wipes out both related direct effects. Do I drop the interaction or report the outcome? The interaction was not part of the original hypothesis.
10
votes
3answers
3k views

How does the Goodman-Kruskal gamma test and the Kendall tau or Spearman rho test compare?

In my work, we are comparing predicted rankings versus true rankings for some sets of data. Up until recently, we've been using Kendall-Tau alone. A group working on a similar project suggested we try ...
52
votes
28answers
11k views

What book would you recommend for non-statistician scientists?

What book would you recommend for scientists who are not statisticians? Clear delivery is most appreciated. As well as the explanation of the appropriate techniques and methods for typical tasks: ...
47
votes
4answers
49k views

Pearson's or Spearman's correlation with non-normal data

I get this question frequently enough in my statistics consulting work, that I thought I'd post it here. I have an answer, which is posted below, but I was keen to hear what others have to say. ...
15
votes
3answers
3k views

How to generate correlated random numbers (given means, variances and degree of correlation)?

I'm sorry if this seems a bit too basic, but I guess I'm just looking to confirm understanding here. I get the sense I'd have to do this in two steps, and I've started trying to grok correlation ...
18
votes
2answers
675 views

Intuitive explanation for density of transformed variable?

Suppose $X$ is a random variable with pdf $f_X(x)$. Then the random variable $Y=X^2$ has the pdf $f_Y(y)=\left\{\begin{array}{ll}\frac{1}{2\sqrt{y}}\left(f_X(\sqrt{y})+f_X(-\sqrt{y})\right) & y ...
9
votes
4answers
1k views

Obtaining a formula for prediction limits in a linear model

Let's take the following example: set.seed(342) x1 <- runif(100) x2 <- runif(100) y <- x1+x2 + 2*x1*x2 + rnorm(100) fit <- lm(y~x1*x2) This creates a ...
90
votes
8answers
7k views

Why does a 95% CI not imply a 95% chance of containing the mean?

It seems that through various related questions here, there is consensus that the "95%" part of what we call a "95% confidence interval" refers to the fact that if we were to exactly replicate our ...
50
votes
13answers
10k views

What is the best way to identify outliers in multivariate data?

Suppose I have a large set of multivariate data with at least three variables. How can I find the outliers? Pairwise scatterplots won't work as it is possible for an outlier to exist in 3 dimensions ...
34
votes
11answers
11k views

Simple algorithm for online outlier detection of a generic time series

I am working with a large amount of time series. These time series are basically network measurements coming every 10 minutes, and some of them are periodic (i.e. the bandwidth), while some other ...
28
votes
6answers
11k views

When is it ok to remove the intercept in lm()?

I am running linear regression models and wondering what the conditions are for removing the intercept term of lm()? In comparing results from two different lm ...
11
votes
3answers
4k views

Factor analysis of questionnaires composed of Likert items

I used to analyse items from a psychometric point of view. But now I am trying to analyse other types of questions on motivation and other topics. These questions are all on Likert scales. My initial ...
8
votes
1answer
4k views

How to interpret type I (sequential) ANOVA and MANOVA?

My primary question is how to interpret the output (coefficients, F, P) when conducting a Type I (sequential) ANOVA? My specific research problem is a bit more complex, so I will break my example into ...
15
votes
3answers
15k views

Significance of coefficients in linear regression: significant t-test vs non-significant F-statistic

I'm fitting a multiple linear regression model between 4 categorical variables (with 4 levels each) and a numerical output. My dataset has 43 observations. R gives me the following $p$-values from ...
12
votes
4answers
5k views

Creating an index of quality from multiple variables to enable rank ordering

I have four numeric variables. All of them are measures of soil quality. Higher the variable, higher the quality. The range for all of them is different: Var1 from 1 to 10 Var2 from 1000 to 2000 ...
9
votes
1answer
2k views

Interpretation of simple predictions to odds ratios in logistic regression

I'm somewhat new to using logistic regression, and a bit confused by a discrepancy between my interpretations of the following values which I thought would be the same: exponentiated beta values ...
150
votes
15answers
28k views

The Two Cultures: statistics vs. machine learning?

Last year, I read a blog post from Brendan O'Connor entitled "Statistics vs. Machine Learning, fight!" that discussed some of the differences between the two fields. Andrew Gelman responded favorably ...

15 30 50 per page