Greatest Hits

1247 votes
27 answers

Making sense of principal component analysis, eigenvectors & eigenvalues

In today's pattern recognition class my professor talked about PCA, eigenvectors and eigenvalues. I understood the mathematics of it. If I'm asked to find eigenvalues etc. I'll do it correctly like ...
user avatar
  • 12.8k
73 votes
10 answers

What is the difference between discrete data and continuous data?

What is the difference between discrete data and continuous data?
user avatar
  • 881
370 votes
7 answers

How to normalize data to 0-1 range?

I am lost in normalizing, could anyone guide me please. I have a minimum and maximum values, say -23.89 and 7.54990767, respectively. If I get a value of 5.6878 how can I scale this value on a scale ...
user avatar
  • 4,071
749 votes
11 answers

How to choose the number of hidden layers and nodes in a feedforward neural network?

Is there a standard and accepted method for selecting the number of layers, and the number of nodes in each layer, in a feed-forward neural network? I'm interested in automated ways of building neural ...
user avatar
  • 52.6k
173 votes
7 answers

What's the difference between variance and standard deviation?

I was wondering what the difference between the variance and the standard deviation is. If you calculate the two values, it is clear that you get the standard deviation out of the variance, but what ...
user avatar
  • 3,559
384 votes
9 answers

What is the difference between fixed effect, random effect and mixed effect models?

In simple terms, how would you explain (perhaps with simple examples) the difference between fixed effect, random effect and mixed effect models?
user avatar
  • 5,608
543 votes
11 answers

What is the difference between test set and validation set?

I found this confusing when I use the neural network toolbox in Matlab. It divided the raw data set into three parts: training set validation set test set I notice in many training or learning ...
user avatar
  • 6,949
277 votes
6 answers

What is batch size in neural network?

I'm using Python Keras package for neural network. This is the link. Is batch_size equals to number of test samples? From ...
user avatar
  • 3,701
529 votes
3 answers

Relationship between SVD and PCA. How to use SVD to perform PCA?

Principal component analysis (PCA) is usually explained via an eigen-decomposition of the covariance matrix. However, it can also be performed via singular value decomposition (SVD) of the data matrix ...
user avatar
  • 95k
619 votes
12 answers

What is the difference between "likelihood" and "probability"?

The wikipedia page claims that likelihood and probability are distinct concepts. In non-technical parlance, "likelihood" is usually a synonym for "probability," but in statistical usage there is a ...
user avatar
532 votes
15 answers

What is the intuition behind beta distribution?

Disclaimer: I'm not a statistician but a software engineer. Most of my knowledge in statistics comes from self-education, thus I still have many gaps in understanding concepts that may seem trivial ...
user avatar
  • 9,460
45 votes
5 answers

What is the difference between a population and a sample?

What is the difference between a population and a sample? What common variables and statistics are used for each one, and how do those relate to each other?
user avatar
  • 2,078
361 votes
12 answers

Difference between logit and probit models

What is the difference between Logit and Probit model? I'm more interested here in knowing when to use logistic regression, and when to use Probit. If there is any literature which defines it using ...
user avatar
  • 5,904
83 votes
28 answers

Examples for teaching: Correlation does not mean causation

There is an old saying: "Correlation does not mean causation". When I teach, I tend to use the following standard examples to illustrate this point: number of storks and birth rate in Denmark; number ...
283 votes
16 answers

What is the meaning of p values and t values in statistical tests?

After taking a statistics course and then trying to help fellow students, I noticed one subject that inspires much head-desk banging is interpreting the results of statistical hypothesis tests. It ...
user avatar
  • 4,186
311 votes
8 answers

What should I do when my neural network doesn't learn?

I'm training a neural network but the training loss doesn't decrease. How can I fix this? I'm not asking about overfitting or regularization. I'm asking about how to solve the problem where my ...
user avatar
  • 79.5k
146 votes
5 answers

How to choose between Pearson and Spearman correlation?

How do I know when to choose between Spearman's $\rho$ and Pearson's $r$? My variable includes satisfaction and the scores were interpreted using the sum of the scores. However, these scores could ...
user avatar
206 votes
8 answers

In linear regression, when is it appropriate to use the log of an independent variable instead of the actual values?

Am I looking for a better behaved distribution for the independent variable in question, or to reduce the effect of outliers, or something else?
user avatar
  • 2,211
179 votes
78 answers

Statistics Jokes

Well, we've got favourite statistics quotes. What about statistics jokes?
275 votes
6 answers

What does AUC stand for and what is it?

Searched high and low and have not been able to find out what AUC, as in related to prediction, stands for or means.
user avatar
  • 3,129
179 votes
8 answers

What is the influence of C in SVMs with linear kernel?

I am currently using an SVM with a linear kernel to classify my data. There is no error on the training set. I tried several values for the parameter $C$ ($10^{-5}, \dots, 10^2$). This did not ...
user avatar
  • 2,535
222 votes
4 answers

How to interpret a QQ plot

I am working with a small dataset (21 observations) and have the following normal QQ plot in R: Seeing that the plot does not support normality, what could I infer about the underlying distribution? ...
user avatar
  • 18.6k
277 votes
153 answers

Famous statistical quotations

What is your favorite statistical quote? This is community wiki, so please one quote per answer.
371 votes
80 answers

What is your favorite "data analysis" cartoon?

Data analysis cartoons can be useful for many reasons: they help communicate; they show that quantitative people have a sense of humor too; they can instigate good teaching moments; and they can help ...
213 votes
17 answers

Intuitive explanation for dividing by $n-1$ when calculating standard deviation?

I was asked today in class why you divide the sum of square error by $n-1$ instead of with $n$, when calculating the standard deviation. I said I am not going to answer it in class (since I didn't ...
user avatar
  • 20.1k
250 votes
7 answers

How to choose a predictive model after k-fold cross-validation?

I am wondering how to choose a predictive model after doing K-fold cross-validation. This may be awkwardly phrased, so let me explain in more detail: whenever I run K-fold cross-validation, I use K ...
user avatar
  • 4,355
203 votes
4 answers

What does the hidden layer in a neural network compute?

I'm sure many people will respond with links to 'let me google that for you', so I want to say that I've tried to figure this out so please forgive my lack of understanding here, but I cannot figure ...
user avatar
  • 2,167
45 votes
3 answers

How to interpret F- and p-value in ANOVA?

I am new to statistics and I currently deal with ANOVA. I carry out an ANOVA test in R using aov(dependendVar ~ IndependendVar) I get – among others – an F-value ...
user avatar
  • 561
342 votes
5 answers

What is the trade-off between batch size and number of iterations to train a neural network?

When training a neural network, what difference does it make to set: batch size to $a$ and number of iterations to $b$ vs. batch size to $c$ and number of iterations to $d$ where $ ab = cd $? To ...
user avatar
529 votes
23 answers

Why square the difference instead of taking the absolute value in standard deviation?

In the definition of standard deviation, why do we have to square the difference from the mean to get the mean (E) and take the square root back at the end? Can't we just simply take the absolute ...
user avatar
  • 5,535
226 votes
4 answers

When (and why) should you take the log of a distribution (of numbers)?

Say I have some historical data e.g., past stock prices, airline ticket price fluctuations, past financial data of the company... Now someone (or some formula) comes along and says "let's take/use ...
user avatar
  • 13.6k
110 votes
10 answers

Validation Error less than training error?

I found two questions here and here about this issue but there is no obvious answer or explanation yet.I enforce the same problem where the validation error is less than training error in my ...
user avatar
  • 1,203
267 votes
10 answers

How would you explain covariance to someone who understands only the mean?

...assuming that I'm able to augment their knowledge about variance in an intuitive fashion ( Understanding "variance" intuitively ) or by saying: It's the average distance of the data ...
user avatar
  • 13.6k
429 votes
5 answers

How to understand the drawbacks of K-means

K-means is a widely used method in cluster analysis. In my understanding, this method does NOT require ANY assumptions, i.e., give me a dataset and a pre-specified number of clusters, k, and I just ...
user avatar
  • 6,487
382 votes
7 answers

When conducting multiple regression, when should you center your predictor variables & when should you standardize them?

In some literature, I have read that a regression with multiple explanatory variables, if in different units, needed to be standardized. (Standardizing consists in subtracting the mean and dividing ...
user avatar
  • 4,231
186 votes
2 answers

How to determine which distribution fits my data best?

I have a dataset and would like to figure out which distribution fits my data best. I used the fitdistr() function to estimate the necessary parameters to ...
user avatar
  • 1,965
390 votes
18 answers

What happens if the explanatory and response variables are sorted independently before regression?

Suppose we have data set $(X_i,Y_i)$ with $n$ points. We want to perform a linear regression, but first we sort the $X_i$ values and the $Y_i$ values independently of each other, forming data set $(...
user avatar
265 votes
11 answers

How would you explain Markov Chain Monte Carlo (MCMC) to a layperson?

Maybe the concept, why it's used, and an example.
user avatar
221 votes
13 answers

What is the difference between data mining, statistics, machine learning and AI?

What is the difference between data mining, statistics, machine learning and AI? Would it be accurate to say that they are 4 fields attempting to solve very similar problems but with different ...
118 votes
4 answers

Softmax vs Sigmoid function in Logistic classifier?

What decides the choice of function ( Softmax vs Sigmoid ) in a Logistic classifier ? Suppose there are 4 output classes . Each of the above function gives the probabilities of each class being the ...
user avatar
  • 1,635
416 votes
14 answers

Bayesian and frequentist reasoning in plain English

How would you describe in plain English the characteristics that distinguish Bayesian from Frequentist reasoning?
user avatar
149 votes
5 answers

Percentile vs quantile vs quartile

What is the difference between the three terms below? percentile quantile quartile
user avatar
  • 12.5k
147 votes
8 answers

What is the difference between off-policy and on-policy learning?

Artificial intelligence website defines off-policy and on-policy learning as follows: "An off-policy learner learns the value of the optimal policy independently of the agent's actions. Q-learning ...
user avatar
  • 7,845
177 votes
9 answers

What exactly are keys, queries, and values in attention mechanisms?

How should one understand the keys, queries, and values that are often mentioned in attention mechanisms? I've tried searching online, but all the resources I find only speak of them as if the reader ...
user avatar
  • 2,441
148 votes
3 answers

Help me understand Bayesian prior and posterior distributions

In a group of students, there are 2 out of 18 that are left-handed. Find the posterior distribution of left-handed students in the population assuming uninformative prior. Summarize the results. ...
user avatar
  • 1,481
178 votes
3 answers

What is the difference between a consistent estimator and an unbiased estimator?

What is the difference between a consistent estimator and an unbiased estimator? The precise technical definitions of these terms are fairly complicated, and it's difficult to get an intuitive feel ...
user avatar
101 votes
10 answers

What's the difference between correlation and simple linear regression?

In particular, I am referring to the Pearson product-moment correlation coefficient.
user avatar
138 votes
3 answers

What is the difference between linear regression and logistic regression?

What is the difference between linear regression and logistic regression? When would you use each?
user avatar
  • 2,903
91 votes
10 answers

Is there a minimum sample size required for the t-test to be valid?

I'm currently working on a quasi-experimental research paper. I only have a sample size of 15 due to low population within the chosen area and that only 15 fit my criteria. Is 15 the minimum sample ...
user avatar
151 votes
3 answers

When is R squared negative?

My understanding is that $R^2$ cannot be negative as it is the square of R. However I ran a simple linear regression in SPSS with a single independent variable and a dependent variable. My SPSS output ...
user avatar
  • 2,007