All Questions
26,044
questions
225
votes
10
answers
118k
views
Why is accuracy not the best measure for assessing classification models?
This is a general question that was asked indirectly multiple times in here, but it lacks a single authoritative answer. It would be great to have a detailed answer to this for the reference.
...
Tim♦
- 128k
240
votes
8
answers
118k
views
Algorithms for automatic model selection
I would like to implement an algorithm for automatic model selection.
I am thinking of doing stepwise regression but anything will do (it has to be based on linear regressions though).
My problem ...
- 2,550
370
votes
15
answers
138k
views
Is normality testing 'essentially useless'?
A former colleague once argued to me as follows:
We usually apply normality tests to the results of processes that,
under the null, generate random variables that are only
asymptotically or ...
- 14.1k
101
votes
4
answers
11k
views
Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?
TL;DR
See title.
Motivation
I am hoping for a canonical answer along the lines of "(1) No, (2) Not applicable, because (1)", which we can use to close many wrong questions about unbalanced ...
- 110k
339
votes
9
answers
309k
views
What should I do when my neural network doesn't learn?
I'm training a neural network but the training loss doesn't decrease. How can I fix this?
I'm not asking about overfitting or regularization. I'm asking about how to solve the problem where my ...
- 85.2k
286
votes
15
answers
543k
views
What is the meaning of p values and t values in statistical tests?
After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
- 4,236
179
votes
6
answers
116k
views
Can a probability distribution value exceeding 1 be OK?
On the Wikipedia page about naive Bayes classifiers, there is this line:
$p(\mathrm{height}|\mathrm{male}) = 1.5789$ (A probability distribution over 1 is OK. It is the area under the bell curve ...
- 4,689
203
votes
10
answers
220k
views
How to deal with perfect separation in logistic regression?
If you have a variable which perfectly separates zeroes and ones in target variable, R will yield the following "perfect or quasi perfect separation" warning message:
...
- 6,981
1308
votes
27
answers
885k
views
Making sense of principal component analysis, eigenvectors & eigenvalues
In today's pattern recognition class my professor talked about PCA, eigenvectors and eigenvalues.
I understood the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like ...
- 13.4k
82
votes
4
answers
54k
views
Reduce Classification Probability Threshold
I have a question regarding classification in general. Let $f$ be a classifier, which outputs a set of probabilities given some data D. Normally, one would say: well, if $P(c|D) > 0.5$, we will ...
- 1,369
271
votes
3
answers
29k
views
How to know that your machine learning problem is hopeless?
Imagine a standard machine-learning scenario:
You are confronted with a large multivariate dataset and you have a
pretty blurry understanding of it. What you need to do is to make
predictions ...
Tim♦
- 128k
106
votes
8
answers
50k
views
What is the benefit of breaking up a continuous predictor variable?
I'm wondering what the value is in taking a continuous predictor variable and breaking it up (e.g., into quintiles), before using it in a model.
It seems to me that by binning the variable we lose ...
- 1,641
332
votes
13
answers
191k
views
How to understand degrees of freedom?
From Wikipedia, there are three interpretations of the degrees of freedom of a statistic:
In statistics, the number of degrees of freedom is the number of
values in the final calculation of a ...
- 18.5k
381
votes
12
answers
367k
views
Difference between logit and probit models
What is the difference between Logit and Probit model?
I'm more interested here in knowing when to use logistic regression, and when to use Probit.
If there is any literature which defines it using ...
- 6,184
176
votes
10
answers
207k
views
When is it ok to remove the intercept in a linear regression model?
I am running linear regression models and wondering what the conditions are for removing the intercept term.
In comparing results from two different regressions where one has the intercept and the ...
- 1,961
103
votes
10
answers
24k
views
104
votes
6
answers
37k
views
Principled way of collapsing categorical variables with many levels?
What techniques are available for collapsing (or pooling) many categories to a few, for the purpose of using them as an input (predictor) in a statistical model?
Consider a variable like college ...
- 12.1k
26
votes
1
answer
7k
views
Is accuracy an improper scoring rule in a binary classification setting?
I have recently been learning about proper scoring rules for probabilistic classifiers. Several threads on this website have made a point of emphasizing that accuracy is an improper scoring rule and ...
- 261
280
votes
6
answers
44k
views
Is $R^2$ useful or dangerous?
I was skimming through some lecture notes by Cosma Shalizi (in particular, section 2.1.1 of the second lecture), and was reminded that you can get very low $R^2$ even when you have a completely linear ...
- 9,490
401
votes
7
answers
389k
views
When conducting multiple regression, when should you center your predictor variables & when should you standardize them?
In some literature, I have read that a regression with multiple explanatory variables, if in different units, needed to be standardized. (Standardizing consists in subtracting the mean and dividing ...
- 4,421
295
votes
16
answers
104k
views
Why does a 95% Confidence Interval (CI) not imply a 95% chance of containing the mean?
It seems that through various related questions here, there is consensus that the "95%" part of what we call a "95% confidence interval" refers to the fact that if we were to exactly replicate our ...
- 13k
63
votes
5
answers
69k
views
Best practice when analysing pre-post treatment-control designs
Imagine the following common design:
100 participants are randomly allocated to either a treatment or a control group
the dependent variable is numeric and measured pre- and post- treatment
Three ...
- 43.2k
101
votes
8
answers
35k
views
When is unbalanced data really a problem in Machine Learning?
We already had multiple questions about unbalanced data when using logistic regression, SVM, decision trees, bagging and a number of other similar questions, what makes it a very popular topic! ...
Tim♦
- 128k
88
votes
1
answer
70k
views
What are the shortcomings of the Mean Absolute Percentage Error (MAPE)?
The Mean Absolute Percentage Error (mape) is a common accuracy or error measure for time series or other predictions,
$$ \text{MAPE} = \frac{100}{n}\sum_{t=1}^n\frac{|A_t-F_t|}{A_t}\%,$$
where $A_t$ ...
- 110k
120
votes
3
answers
126k
views
Does an unbalanced sample matter when doing logistic regression?
Okay, so I think I have a decent enough sample, taking into account the 20:1 rule of thumb: a fairly large sample (N=374) for a total of 7 candidate predictor variables.
My problem is the following: ...
- 1,303
113
votes
18
answers
106k
views
Including the interaction but not the main effects in a model
Is it ever valid to include a two-way interaction in a model without including the main effects? What if your hypothesis is only about the interaction, do you still need to include the main effects?
- 6,570
563
votes
3
answers
441k
views
Relationship between SVD and PCA. How to use SVD to perform PCA?
Principal component analysis (PCA) is usually explained via an eigen-decomposition of the covariance matrix. However, it can also be performed via singular value decomposition (SVD) of the data matrix ...
- 99.2k
409
votes
9
answers
818k
views
What is the difference between fixed effect, random effect and mixed effect models?
In simple terms, how would you explain (perhaps with simple examples) the difference between fixed effect, random effect and mixed effect models?
- 5,908
138
votes
3
answers
47k
views
What if residuals are normally distributed, but y is not?
I've got a weird question. Assume that you have a small sample where the dependent variable that you're going to analyze with a simple linear model is highly left skewed. Thus you assume that $u$ is ...
- 5,765
126
votes
14
answers
76k
views
Maximum Likelihood Estimation (MLE) in layman terms
Could anyone explain to me in detail about maximum likelihood estimation (MLE) in layman's terms? I would like to know the underlying concept before going into mathematical derivation or equation.
- 1,739
24
votes
2
answers
6k
views
Proper scoring rule when there is a decision to make (e.g. spam vs ham email)
Among others on here, Frank Harrell is adamant about using proper scoring rules to assess classifiers. This makes sense. If we have 500 $0$s with $P(1)\in[0.45, 0.49]$ and 500 $1$s with $P(1)\in[0.51, ...
- 47.2k
205
votes
5
answers
257k
views
How exactly does one “control for other variables”?
Here is the article that motivated this question: Does impatience make us fat?
I liked this article, and it nicely demonstrates the concept of “controlling for other variables” (IQ, career, income, ...
- 2,747
237
votes
4
answers
348k
views
When (and why) should you take the log of a distribution (of numbers)?
Say I have some historical data e.g., past stock prices, airline ticket price fluctuations, past financial data of the company...
Now someone (or some formula) comes along and says "let's take/use ...
- 14.1k
132
votes
4
answers
53k
views
Is it possible to have a pair of Gaussian random variables for which the joint distribution is not Gaussian?
Somebody asked me this question in a job interview and I replied that their joint distribution is always Gaussian. I thought that I can always write a bivariate Gaussian with their means and variance ...
- 2,689
213
votes
8
answers
492k
views
In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?
Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
- 2,281
96
votes
10
answers
45k
views
What is a complete list of the usual assumptions for linear regression?
What are the usual assumptions for linear regression?
Do they include:
a linear relationship between the independent and dependent variable
independent errors
normal distribution of errors
...
- 969
207
votes
7
answers
190k
views
PCA on correlation or covariance?
What are the main differences between performing principal component analysis (PCA) on the correlation matrix and on the covariance matrix? Do they give the same results?
- 2,250
25
votes
1
answer
11k
views
Goodness of fit and which model to choose linear regression or Poisson
I need some advice regarding two main dilemmas in my research, which is a case study of 3 big pharmaceuticals and innovation. Number of patents per year is the dependent variable.
My questions are
...
- 331
545
votes
23
answers
304k
views
Why square the difference instead of taking the absolute value in standard deviation?
In the definition of standard deviation, why do we have to square the difference from the mean to get the mean (E) and take the square root back at the end? Can't we just simply take the absolute ...
- 5,705
166
votes
1
answer
122k
views
Crossed vs nested random effects: how do they differ and how are they specified correctly in lme4?
Here is how I have understood nested vs. crossed random effects:
Nested random effects occur when a lower level factor appears only within a particular level of an upper level factor.
For ...
- 3,334
54
votes
5
answers
32k
views
Generic sum of Gamma random variables
I have read that the sum of Gamma random variables with the same scale parameter is another Gamma random variable. I've also seen the paper by Moschopoulos describing a method for the summation of a ...
- 1,117
218
votes
3
answers
158k
views
R's lmer cheat sheet
There's a lot of discussion going on on this forum about the proper way to specify various hierarchical models using lmer.
I thought it would be great to have all ...
70
votes
2
answers
55k
views
How to interpret type I, type II, and type III ANOVA and MANOVA?
My primary question is how to interpret the output (coefficients, F, P) when conducting a Type I (sequential) ANOVA?
My specific research problem is a bit more complex, so I will break my example ...
- 1,881
29
votes
1
answer
12k
views
Omitted variable bias in logistic regression vs. omitted variable bias in ordinary least squares regression
I have a question about omitted variable bias in logistic and linear regression.
Say I omit some variables from a linear regression model. Pretend that those omitted variables are uncorrelated with ...
132
votes
6
answers
26k
views
Is there an intuitive interpretation of $A^TA$ for a data matrix $A$?
For a given data matrix $A$ (with variables in columns and data points in rows), it seems like $A^TA$ plays an important role in statistics. For example, it is an important part of the analytical ...
- 2,315
68
votes
2
answers
24k
views
Is there a difference between 'controlling for' and 'ignoring' other variables in multiple regression?
The coefficient of an explanatory variable in a multiple regression tells us the relationship of that explanatory variable with the dependent variable. All this, while 'controlling' for the other ...
- 1,515
62
votes
3
answers
22k
views
What is the intuition behind conditional Gaussian distributions?
Suppose that $\mathbf{X} \sim N_{2}(\mathbf{\mu}, \mathbf{\Sigma})$. Then the conditional distribution of $X_1$ given that $X_2 = x_2$ is multivariate normally distributed with mean:
$$ E[P(X_1 | ...
- 621
59
votes
4
answers
18k
views
Why do statisticians say a non-significant result means "you can't reject the null" as opposed to accepting the null hypothesis?
Traditional statistical tests, like the two sample t-test, focus on trying to eliminate the hypothesis that there is no difference between a function of two independent samples. Then, we choose a ...
- 2,400
116
votes
4
answers
45k
views
Why isn't Logistic Regression called Logistic Classification?
Since Logistic Regression is a statistical classification model dealing with categorical dependent variables, why isn't it called Logistic Classification? Shouldn't the "Regression" name be reserved ...
- 2,136
43
votes
3
answers
16k
views
Relation between confidence interval and testing statistical hypothesis for t-test
It is well known that confidence intervals and testing statistical hypothesis are strongly related. My questions is focused on comparison of means for two groups based on a numerical variable. Let's ...
- 1,371