Questions tagged [canonical-question]

Applied by moderators to exceptional threads likely to answer many subsequent questions.

Filter by
Sorted by
Tagged with
4 votes
1 answer
206 views

What can I validly conclude about a variable that becomes significant and switches sign when other variables are included in the model?

I have a dataset dat where each row represents a soil sample, with independent variables chemical measurements a, ...
user avatar
4 votes
2 answers
7k views

How to determine $p$ and $q$ in my ARIMA model from these ACF and PACF plots?

I have converted stock price index time series data into stationary series by differencing once, so $d=1$. I also have removed the seasonal component of the data. I want to develop a model for ...
user avatar
  • 41
24 votes
1 answer
8k views

How do you deal with "nested" variables in a regression model?

Consider a statistical problem where you have a response variable that you want to describe conditional on an explanatory ...
user avatar
  • 94.2k
58 votes
4 answers
38k views

What should I do when my neural network doesn't generalize well?

I'm training a neural network and the training loss decreases, but the validation loss doesn't, or it decreases much less than what I would expect, based on references or experiments with very similar ...
user avatar
  • 15.9k
305 votes
8 answers
255k views

What should I do when my neural network doesn't learn?

I'm training a neural network but the training loss doesn't decrease. How can I fix this? I'm not asking about overfitting or regularization. I'm asking about how to solve the problem where my ...
user avatar
  • 78.1k
196 votes
10 answers
100k views

Why is accuracy not the best measure for assessing classification models?

This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference. ...
user avatar
  • 113k
33 votes
4 answers
15k views

Independent variable = Random variable?

I'm slightly confused if an independent variable (also called predictor or feature) in a statistical model, for example the $X$ in linear regression $Y=\beta_0+\beta_1 X$, is a random variable ?
user avatar
  • 1,075
11 votes
5 answers
3k views

Interpretation of Bayes Theorem applied to positive mammography results

I'm trying to wrap my head around the result of Bayes Theorem applied to the classic mammogram example, with the twist of the mammogram being perfect. That is, Incidence of cancer: $.01$ ...
user avatar
39 votes
3 answers
13k views

Do we need gradient descent to find the coefficients of a linear regression model?

I was trying to learn machine learning using the Coursera material. In this lecture, Andrew Ng uses gradient descent algorithm to find the coefficients of the linear regression model that will ...
user avatar
  • 6,005
95 votes
6 answers
32k views

Principled way of collapsing categorical variables with many levels?

What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model? Consider a variable like college ...
user avatar
  • 11.5k
20 votes
5 answers
10k views

$P[X=x]=0$ when $X$ is a continuous variable

I know that for any continuous variable $P[X=x]=0$. But I can't visualize that if $P[X=x]=0$, there is an infinite number of possible $x$'s. And also why do their probabilities get infinitely small ?
user avatar
  • 1,187
9 votes
2 answers
51k views

What does the notation like 8.6e-28 mean? What is the 'e' for?

I have a problem with the interpretation of a test result in which the p-value is 8.6e-28. How should it be interpreted? What is the ...
user avatar
526 votes
3 answers
388k views

Relationship between SVD and PCA. How to use SVD to perform PCA?

Principal component analysis (PCA) is usually explained via an eigen-decomposition of the covariance matrix. However, it can also be performed via singular value decomposition (SVD) of the data matrix ...
user avatar
  • 94.4k
7 votes
5 answers
16k views

Coefficient changes sign when adding a variable in logistic regression

In my logistic regression the sign of coefficients of a variable (location distance of an amenity) changes based on other variables (with time -ve, with travel distance +ve) in the model. When the ...
user avatar
  • 173
219 votes
4 answers
410k views

How to interpret a QQ plot

I am working with a small dataset (21 observations) and have the following normal QQ plot in R: Seeing that the plot does not support normality, what could I infer about the underlying distribution? ...
user avatar
  • 18.5k
49 votes
4 answers
44k views

Does the sign of scores or of loadings in PCA or FA have a meaning? May I reverse the sign?

I performed principal component analysis (PCA) with R using two different functions (prcomp and princomp) and observed that the ...
user avatar
100 votes
8 answers
45k views

What is the benefit of breaking up a continuous predictor variable?

I'm wondering what the value is in taking a continuous predictor variable and breaking it up (e.g., into quintiles), before using it in a model. It seems to me that by binning the variable we lose ...
user avatar
  • 1,531
44 votes
4 answers
53k views

Should covariates that are not statistically significant be 'kept in' when creating a model?

I have several covariates in my calculation for a model, and not all of them are statistically significant. Should I remove those that are not? This question discusses the phenomenon, but does not ...
user avatar
  • 669
50 votes
5 answers
36k views

Normality of dependent variable = normality of residuals?

This issue seems to rear its ugly head all the time, and I'm trying to decapitate it for my own understanding of statistics (and sanity!). The assumptions of general linear models (t-test, ANOVA, ...
user avatar
  • 861
41 votes
5 answers
115k views

How to derive the least square estimator for multiple linear regression?

In the simple linear regression case $y=\beta_0+\beta_1x$, you can derive the least square estimator $\hat\beta_1=\frac{\sum(x_i-\bar x)(y_i-\bar y)}{\sum(x_i-\bar x)^2}$ such that you don't have to ...
user avatar
  • 779
69 votes
5 answers
24k views

How small a quantity should be added to x to avoid taking the log of zero?

I have analysed my data as they are. Now I want to look at my analyses after taking the log of all variables. Many variables contain many zeros. Therefore I add a small quantity to avoid taking the ...
user avatar
  • 3,414
30 votes
6 answers
11k views

t-test for partially paired and partially unpaired data

An investigator wishes to produce a combined analysis of several datasets. In some datasets there are paired observations for treatment A and B. In others there are unpaired A and/or B data. I am ...
user avatar
230 votes
8 answers
111k views

Algorithms for automatic model selection

I would like to implement an algorithm for automatic model selection. I am thinking of doing stepwise regression but anything will do (it has to be based on linear regressions though). My problem ...
user avatar
  • 2,450
59 votes
7 answers
76k views

Effect of switching response and explanatory variable in simple linear regression

Let's say that there exists some "true" relationship between $y$ and $x$ such that $y = ax + b + \epsilon$, where $a$ and $b$ are constants and $\epsilon$ is i.i.d normal noise. When I randomly ...
user avatar
17 votes
2 answers
5k views

Explaining two-tailed tests

I am looking for various ways of explaining to my students (in an elementary statistics course) what is a two tailed test, and how its P value is calculated. How do you explain to your students the ...
user avatar
57 votes
3 answers
242k views

ANOVA assumption normality/normal distribution of residuals

The Wikipedia page on ANOVA lists three assumptions, namely: Independence of cases – this is an assumption of the model that simplifies the statistical analysis. Normality – the distributions of the ...
user avatar
174 votes
6 answers
108k views

Can a probability distribution value exceeding 1 be OK?

On the Wikipedia page about naive Bayes classifiers, there is this line: $p(\mathrm{height}|\mathrm{male}) = 1.5789$ (A probability distribution over 1 is OK. It is the area under the bell curve ...
user avatar
207 votes
17 answers
202k views

Intuitive explanation for dividing by $n-1$ when calculating standard deviation?

I was asked today in class why you divide the sum of square error by $n-1$ instead of with $n$, when calculating the standard deviation. I said I am not going to answer it in class (since I didn't ...
user avatar
84 votes
9 answers
73k views

Why is it possible to get significant F statistic (p<.001) but non-significant regressor t-tests?

In a multiple linear regression, why is it possible to have a highly significant F statistic (p<.001) but have very high p-values on all the regressor's t tests? In my model, there are 10 ...
user avatar
  • 1,439
60 votes
5 answers
66k views

Best practice when analysing pre-post treatment-control designs

Imagine the following common design: 100 participants are randomly allocated to either a treatment or a control group the dependent variable is numeric and measured pre- and post- treatment Three ...
user avatar
1240 votes
27 answers
822k views

Making sense of principal component analysis, eigenvectors & eigenvalues

In today's pattern recognition class my professor talked about PCA, eigenvectors and eigenvalues. I understood the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like ...
user avatar
  • 12.7k
100 votes
9 answers
46k views

Is there an intuitive explanation why multicollinearity is a problem in linear regression?

The wiki discusses the problems that arise when multicollinearity is an issue in linear regression. The basic problem is multicollinearity results in unstable parameter estimates which makes it very ...
user avatar
205 votes
8 answers
454k views

In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?

Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
user avatar
  • 2,201
525 votes
23 answers
280k views

Why square the difference instead of taking the absolute value in standard deviation?

In the definition of standard deviation, why do we have to square the difference from the mean to get the mean (E) and take the square root back at the end? Can't we just simply take the absolute ...
user avatar
  • 5,495
69 votes
8 answers
66k views

What are good basic statistics to use for ordinal data?

I have some ordinal data gained from survey questions. In my case they are Likert style responses (Strongly Disagree-Disagree-Neutral-Agree-Strongly Agree). In my data they are coded as 1-5. I don'...
user avatar
  • 1,549
283 votes
16 answers
527k views

What is the meaning of p values and t values in statistical tests?

After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
user avatar
  • 4,176