Questions tagged [mathematical-statistics]
Mathematical theory of statistics, concerned with formal definitions and general results.
7,216
questions
173
votes
7
answers
862k
views
What's the difference between variance and standard deviation?
I was wondering what the difference between the variance and the standard deviation is.
If you calculate the two values, it is clear that you get the standard deviation out of the variance, but what ...
- 3,559
161
votes
9
answers
105k
views
Bottom to top explanation of the Mahalanobis distance?
I'm studying pattern recognition and statistics and almost every book I open on the subject I bump into the concept of Mahalanobis distance. The books give sort of intuitive explanations, but still ...
- 5,257
142
votes
21
answers
102k
views
What's the difference between probability and statistics?
What's the difference between probability and statistics, and why are they studied together?
- 1,557
131
votes
10
answers
97k
views
Why does the Cauchy distribution have no mean?
From the distribution density function we could identify a mean (=0) for Cauchy distribution just like the graph below shows. But why do we say Cauchy distribution has no mean?
- 5,849
126
votes
9
answers
97k
views
Numerical example to understand Expectation-Maximization
I am trying to get a good grasp on the EM algorithm, to be able to implement and use it. I spent a full day reading the theory and a paper where EM is used to track an aircraft using the position ...
- 2,473
119
votes
14
answers
72k
views
Maximum Likelihood Estimation (MLE) in layman terms
Could anyone explain to me in detail about maximum likelihood estimation (MLE) in layman's terms? I would like to know the underlying concept before going into mathematical derivation or equation.
- 1,549
110
votes
10
answers
182k
views
Validation Error less than training error?
I found two questions here and here about this issue but there is no obvious answer or explanation yet.I enforce the same problem where the validation error is less than training error in my ...
- 1,203
104
votes
13
answers
80k
views
Simple algorithm for online outlier detection of a generic time series
I am working with a large amount of time series. These time series are basically network measurements coming every 10 minutes, and some of them are periodic (i.e. the bandwidth), while some other aren'...
- 1,921
103
votes
7
answers
73k
views
How does the reparameterization trick for VAEs work and why is it important?
How does the reparameterization trick for variational autoencoders (VAE) work? Is there an intuitive and easy explanation without simplifying the underlying math? And why do we need the 'trick'?
- 2,514
96
votes
8
answers
32k
views
If mean is so sensitive, why use it in the first place?
It is a known fact that median is resistant to outliers. If that is the case, when and why would we use the mean in the first place?
One thing I can think of perhaps is to understand the presence of ...
- 4,262
95
votes
12
answers
11k
views
Who Are The Bayesians?
As one becomes interested in statistics, the dichotomy "Frequentist" vs. "Bayesian" soon becomes commonplace (and who hasn't read Nate Silver's The Signal and the Noise, anyway?). In talks and ...
- 23.7k
93
votes
10
answers
21k
views
88
votes
5
answers
44k
views
Mutual information versus correlation
Why and when we should use Mutual Information over statistical correlation measurements such as "Pearson", "spearman", or "Kendall's tau" ?
- 1,025
80
votes
5
answers
47k
views
How exactly did statisticians agree to using (n-1) as the unbiased estimator for population variance without simulation?
The formula for computing variance has $(n-1)$ in the denominator:
$s^2 = \frac{\sum_{i=1}^N (x_i - \bar{x})^2}{n-1}$
I've always wondered why. However, reading and watching a few good videos about "...
- 13.7k
79
votes
12
answers
8k
views
Famous statistical wins and horror stories for teaching purposes
I am designing a one year program in data analysis with a local community college. The program aims to prepare students to handle basic tasks in data analysis, visualization and summarization, ...
78
votes
5
answers
29k
views
Central limit theorem for sample medians
If I calculate the median of a sufficiently large number of observations drawn from the same distribution, does the central limit theorem state that the distribution of medians will approximate a ...
- 977
75
votes
6
answers
105k
views
Why is it that natural log changes are percentage changes? What is about logs that makes this so?
Can somebody explain how the properties of logs make it so you can do log linear regressions where the coefficients are interpreted as percentage changes?
- 957
74
votes
15
answers
11k
views
Why would parametric statistics ever be preferred over nonparametric?
Can someone explain to me why would anyone choose a parametric over a nonparametric statistical method for hypothesis testing or regression analysis?
In my mind, it's like going for rafting and ...
- 907
70
votes
1
answer
72k
views
KL divergence between two multivariate Gaussians
I'm having trouble deriving the KL divergence formula assuming two multivariate normal distributions. I've done the univariate case fairly easily. However, it's been quite a while since I took math ...
- 3,115
67
votes
3
answers
32k
views
Variables are often adjusted (e.g. standardised) before making a model - when is this a good idea, and when is it a bad one?
In what circumstances would you want to, or not want to scale or standardize a variable prior to model fitting? And what are the advantages / disadvantages of scaling a variable?
- 5,618
65
votes
14
answers
10k
views
What is the most surprising characterization of the Gaussian (normal) distribution?
A standardized Gaussian distribution on $\mathbb{R}$ can be defined by giving explicitly its density:
$$ \frac{1}{\sqrt{2\pi}}e^{-x^2/2}$$
or its characteristic function.
As recalled in this ...
64
votes
8
answers
6k
views
Are bayesians slaves of the likelihood function?
In his book "All of Statistics", Prof. Larry Wasserman presents the following Example (11.10, page 188). Suppose that we have a density $f$ such that $f(x)=c\,g(x)$, where $g$ is a known (nonnegative, ...
- 22.2k
64
votes
4
answers
18k
views
What is so cool about de Finetti's representation theorem?
From Theory of Statistics by Mark J. Schervish (page 12):
Although DeFinetti's representation theorem 1.49 is central to motivating parametric models, it is not actually used in their ...
- 13.5k
56
votes
13
answers
5k
views
What are the breakthroughs in Statistics of the past 15 years?
I still remember the Annals of Statistics paper on Boosting by Friedman-Hastie-Tibshirani, and the comments on that same issues by other authors (including Freund and Schapire). At that time, clearly ...
56
votes
9
answers
3k
views
Are we exaggerating importance of model assumption and evaluation in an era when analyses are often carried out by laymen
Bottom line, the more I learn about statistics, the less I trust published papers in my field; I simply believe that researchers are not doing their statistics well enough.
I'm a layman, so to speak. ...
- 2,093
55
votes
6
answers
7k
views
What are the main theorems in Machine (Deep) Learning?
Al Rahimi has recently given a very provocative talk in NIPS 2017 comparing current Machine Learning to Alchemy. One of his claims is that we need to get back to theoretical developments, to have ...
55
votes
19
answers
9k
views
Mathematical Statistics Videos
A question previously sought recommendations for textbooks on mathematical statistics
Does anyone know of any good online video lectures on mathematical statistics?
The closest that I've found are:
...
54
votes
5
answers
17k
views
When is a biased estimator preferable to unbiased one?
It's obvious many times why one prefers an unbiased estimator. But, are there any circumstances under which we might actually prefer a biased estimator over an unbiased one?
- 3,733
53
votes
3
answers
14k
views
Consider the sum of $n$ uniform distributions on $[0,1]$, or $Z_n$. Why does the cusp in the PDF of $Z_n$ disappear for $n \geq 3$?
I've been wondering about this one for a while; I find it a little weird how abruptly it happens. Basically, why do we need just three uniforms for $Z_n$ to smooth out like it does? And why does the ...
- 1,356
52
votes
2
answers
50k
views
What is the distribution of the sum of non i.i.d. gaussian variates?
If $X$ is distributed $N(\mu_X, \sigma^2_X)$,
$Y$ is distributed $N(\mu_Y, \sigma^2_Y)$
and $Z = X + Y$, I know that $Z$ is distributed $N(\mu_X + \mu_Y, \sigma^2_X + \sigma^2_Y)$ if X and Y are ...
- 1,432
51
votes
3
answers
9k
views
How can I calculate $\int^{\infty}_{-\infty}\Phi\left(\frac{w-a}{b}\right)\phi(w)\,\mathrm dw$
Suppose $\phi(\cdot)$ and $\Phi(\cdot)$ are density function and distribution function of the standard normal distribution.
How can one calculate the integral:
$$\int^{\infty}_{-\infty}\Phi\left(\...
- 853
50
votes
3
answers
10k
views
How does saddlepoint approximation work?
How does saddlepoint approximation work? What sort of problem is it good for?
(Feel free to use a particular example or examples by way of illustration)
Are there any drawbacks, difficulties, things ...
- 261k
50
votes
6
answers
14k
views
Motivation for Kolmogorov distance between distributions
There are many ways to measure how similar two probability distributions are. Among methods which are popular (in different circles) are:
the Kolmogorov distance: the sup-distance between the ...
- 2,976
49
votes
3
answers
97k
views
Derive Variance of regression coefficient in simple linear regression
In simple linear regression, we have $y = \beta_0 + \beta_1 x + u$, where $u \sim iid\;\mathcal N(0,\sigma^2)$. I derived the estimator:
$$
\hat{\beta_1} = \frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}...
- 1,603
47
votes
4
answers
10k
views
Taking the expectation of Taylor series (especially the remainder)
My question concerns trying to justify a widely-used method, namely taking the expected value of Taylor Series. Assume we have a random variable $X$ with positive mean $\mu$ and variance $\sigma^2$. ...
- 655
46
votes
6
answers
10k
views
Why don't linear regression assumptions matter in machine learning?
When I learned linear regression in my statistics class, we are asked to check for a few assumptions which need to be true for linear regression to make sense. I won't delve deep into those ...
- 571
46
votes
3
answers
20k
views
Empirical relationship between mean, median and mode
For a unimodal distribution that is moderately skewed, we have the following empirical relationship between the mean, median and mode:
$$
\text{(Mean - Mode)}\sim 3\,\text{(Mean - Median)}
$$
How ...
- 1,367
44
votes
5
answers
15k
views
What is the purpose of characteristic functions?
I'm hoping that someone can explain, in layman's terms, what a characteristic function is and how it is used in practice. I've read that it is the Fourier transform of the pdf, so I guess I know what ...
- 3,377
44
votes
2
answers
16k
views
Differences between Bhattacharyya distance and KL divergence
I'm looking for an intuitive explanation for the following questions:
In statistics and information theory, what's the difference between Bhattacharyya distance and KL divergence, as measures of the ...
- 543
43
votes
7
answers
21k
views
Are there any examples of where the central limit theorem does not hold?
Wikipedia says -
In probability theory, the central limit theorem (CLT) establishes that, in most situations, when independent random variables are added, their properly normalized sum tends ...
- 563
43
votes
9
answers
7k
views
Correlation does not imply causation; but what about when one of the variables is time?
I know this question has been asked a billion times, so, after looking online, I am fully convinced that correlation between 2 variables does not imply causation. In one of my stats lectures today, we ...
- 1,405
41
votes
8
answers
99k
views
Looking for a good and complete probability and statistics book
I never had the opportunity to visit a stats course from a math faculty. I am looking for a probability theory and statistics book that is complete and self-sufficient. By complete I mean that it ...
39
votes
6
answers
37k
views
Why does the variance of the Random walk increase?
The random walk that is defined as $Y_{t} = Y_{t-1} + e_t$, where $e_t$ is white noise. Denotes that the current position is the sum of the previous position + an unpredicted term.
You can prove that ...
- 679
39
votes
4
answers
51k
views
How do you Interpret RMSLE (Root Mean Squared Logarithmic Error)?
I've been doing a machine learning competition where they use RMSLE (Root Mean Squared Logarithmic Error) to evaluate the performance predicting the sale price of a category of equipment. The problem ...
- 391
39
votes
3
answers
17k
views
Distribution of scalar products of two random unit vectors in $D$ dimensions
If $\mathbf{x}$ and $\mathbf{y}$ are two independent random unit vectors in $\mathbb{R}^D$ (uniformly distributed on a unit sphere), what is the distribution of their scalar product (dot product) $\...
- 95.1k
38
votes
8
answers
22k
views
Is it possible to prove a null hypothesis?
As the question states - Is it possible to prove the null hypothesis? From my (limited) understanding of hypothesis, the answer is no but I can't come up with a rigorous explanation for it. Does the ...
- 491
38
votes
4
answers
27k
views
Is a strong background in maths a total requisite for ML?
I'm starting to want to advance my own skillset and I've always been fascinated by machine learning. However, six years ago instead of pursuing this I decided to take a completely unrelated degree to ...
- 503
37
votes
8
answers
15k
views
What is Bayes' theorem all about?
What are the main ideas, that is, concepts related to Bayes' theorem?
I am not asking for any derivations of complex mathematical notation.
- 6,731
37
votes
2
answers
3k
views
Probability inequalities
I am looking for some probability inequalities for sums of unbounded random variables. I would really appreciate it if anyone can provide me some thoughts.
My problem is to find an exponential upper ...
- 575
36
votes
3
answers
5k
views
I know the 95% confidence interval for ln(x), do I also know the 95% confidence interval of x?
Suppose the 95% confidence interval for $\ln(x)$ is $[l,u]$. Is it true that the 95% CI for $x$ is simply $[e^l, e^u]$?
I have the intuition the answer is yes, because $\ln$ is a continuous function. ...
- 485