Questions tagged [mathematical-statistics]

Mathematical theory of statistics, concerned with formal definitions and general results.

Filter by
Sorted by
Tagged with
79
votes
8answers
13k views

What is meant by a “random variable”?

What do they mean when they say "random variable"?
97
votes
12answers
60k views

Maximum Likelihood Estimation (MLE) in layman terms

Could anyone explain to me in detail about maximum likelihood estimation (MLE) in layman's terms? I would like to know the underlying concept before going into mathematical derivation or equation.
140
votes
9answers
90k views

Bottom to top explanation of the Mahalanobis distance?

I'm studying pattern recognition and statistics and almost every book I open on the subject I bump into the concept of Mahalanobis distance. The books give sort of intuitive explanations, but still ...
48
votes
3answers
8k views

Consider the sum of $n$ uniform distributions on $[0,1]$, or $Z_n$. Why does the cusp in the PDF of $Z_n$ disappear for $n \geq 3$?

I've been wondering about this one for a while; I find it a little weird how abruptly it happens. Basically, why do we need just three uniforms for $Z_n$ to smooth out like it does? And why does the ...
57
votes
5answers
17k views

Central limit theorem for sample medians

If I calculate the median of a sufficiently large number of observations drawn from the same distribution, does the central limit theorem state that the distribution of medians will approximate a ...
93
votes
14answers
69k views

Simple algorithm for online outlier detection of a generic time series

I am working with a large amount of time series. These time series are basically network measurements coming every 10 minutes, and some of them are periodic (i.e. the bandwidth), while some other aren'...
41
votes
3answers
6k views

How does saddlepoint approximation work?

How does saddlepoint approximation work? What sort of problem is it good for? (Feel free to use a particular example or examples by way of illustration) Are there any drawbacks, difficulties, things ...
119
votes
10answers
77k views

Why does the Cauchy distribution have no mean?

From the distribution density function we could identify a mean (=0) for Cauchy distribution just like the graph below shows. But why do we say Cauchy distribution has no mean?
58
votes
14answers
8k views

What is the most surprising characterization of the Gaussian (normal) distribution?

A standardized Gaussian distribution on $\mathbb{R}$ can be defined by giving explicitly its density: $$ \frac{1}{\sqrt{2\pi}}e^{-x^2/2}$$ or its characteristic function. As recalled in this ...
31
votes
3answers
12k views

Distribution of scalar products of two random unit vectors in $D$ dimensions

If $\mathbf{x}$ and $\mathbf{y}$ are two independent random unit vectors in $\mathbb{R}^D$ (uniformly distributed on a unit sphere), what is the distribution of their scalar product (dot product) $\...
43
votes
3answers
7k views

How can I calculate $\int^{\infty}_{-\infty}\Phi\left(\frac{w-a}{b}\right)\phi(w)\,\mathrm dw$

Suppose $\phi(\cdot)$ and $\Phi(\cdot)$ are density function and distribution function of the standard normal distribution. How can one calculate the integral: $$\int^{\infty}_{-\infty}\Phi\left(\...
20
votes
5answers
5k views

What does “likelihood is only defined up to a multiplicative constant of proportionality” mean in practice?

I'm reading a paper where the authors are leading from a discussion of maximum likelihood estimation to Bayes' Theorem, ostensibly as an introduction for beginners. As a likelihood example, they ...
37
votes
8answers
19k views

Is it possible to prove a null hypothesis?

As the question states - Is it possible to prove the null hypothesis? From my (limited) understanding of hypothesis, the answer is no but I can't come up with a rigorous explanation for it. Does the ...
90
votes
8answers
30k views

If mean is so sensitive, why use it in the first place?

It is a known fact that median is resistant to outliers. If that is the case, when and why would we use the mean in the first place? One thing I can think of perhaps is to understand the presence of ...
12
votes
4answers
1k views

Maximum likelihood function for mixed type distribution

In general we maximize a function $$ L(\theta; x_1, \ldots, x_n) = \prod_{i=1}^n f(x_i \mid \theta) $$ where $f$ is probability density function if the underlying distribution is continuous, and a ...
62
votes
15answers
9k views

Why would parametric statistics ever be preferred over nonparametric?

Can someone explain to me why would anyone choose a parametric over a nonparametric statistical method for hypothesis testing or regression analysis? In my mind, it's like going for rafting and ...
121
votes
9answers
88k views

Numerical example to understand Expectation-Maximization

I am trying to get a good grasp on the EM algorithm, to be able to implement and use it. I spent a full day reading the theory and a paper where EM is used to track an aircraft using the position ...
70
votes
7answers
43k views

How does the reparameterization trick for VAEs work and why is it important?

How does the reparameterization trick for variational autoencoders (VAE) work? Is there an intuitive and easy explanation without simplifying the underlying math? And why do we need the 'trick'?
72
votes
5answers
39k views

How exactly did statisticians agree to using (n-1) as the unbiased estimator for population variance without simulation?

The formula for computing variance has $(n-1)$ in the denominator: $s^2 = \frac{\sum_{i=1}^N (x_i - \bar{x})^2}{n-1}$ I've always wondered why. However, reading and watching a few good videos about "...
40
votes
3answers
74k views

Derive Variance of regression coefficient in simple linear regression

In simple linear regression, we have $y = \beta_0 + \beta_1 x + u$, where $u \sim iid\;\mathcal N(0,\sigma^2)$. I derived the estimator: $$ \hat{\beta_1} = \frac{\sum_i (x_i - \bar{x})(y_i - \bar{y})}...
42
votes
2answers
37k views

What is the distribution of the sum of non i.i.d. gaussian variates?

If $X$ is distributed $N(\mu_X, \sigma^2_X)$, $Y$ is distributed $N(\mu_Y, \sigma^2_Y)$ and $Z = X + Y$, I know that $Z$ is distributed $N(\mu_X + \mu_Y, \sigma^2_X + \sigma^2_Y)$ if X and Y are ...
22
votes
3answers
20k views

Asymptotic distribution of sample variance of non-normal sample

This is a more general treatment of the issue posed by this question. After deriving the asymptotic distribution of the sample variance, we can apply the Delta method to arrive at the corresponding ...
7
votes
1answer
3k views

Distribution of a quadratic form, non-central chi-squared distribution

Definition. Suppose $\mathbf{y} \sim \mathcal{N}(\boldsymbol{\mu}, I_{n \times n})$. Then $$w = \mathbf{y}^{T}\mathbf{y} = \|\mathbf{y}\|^2 \sim \chi^{2}_{n}\left(\theta = \|\boldsymbol{\mu}\|^2/...
10
votes
1answer
2k views

the relationship between maximizing the likelihood and minimizing the cross-entropy

There is a statement that maximizing the likelihood is equivalent to minimizing the cross-entropy. Are there any proof for this statement?
6
votes
3answers
7k views

Tool for generating correlated data sets

Does anyone know of a tool that I can use to generate a set of data with known correlations (and to put the icing on the cake - output this in json,csv,txt or some common format)? I am working on ...
79
votes
10answers
119k views

Validation Error less than training error?

I found two questions here and here about this issue but there is no obvious answer or explanation yet.I enforce the same problem where the validation error is less than training error in my ...
56
votes
3answers
15k views

What is so cool about de Finetti's representation theorem?

From Theory of Statistics by Mark J. Schervish (page 12): Although DeFinetti's representation theorem 1.49 is central to motivating parametric models, it is not actually used in their ...
56
votes
1answer
49k views

KL divergence between two multivariate Gaussians

I'm having trouble deriving the KL divergence formula assuming two multivariate normal distributions. I've done the univariate case fairly easily. However, it's been quite a while since I took math ...
18
votes
4answers
3k views

Good resources (online or book) on the mathematical foundations of statistics

Before I ask my question, let me give you a bit of background about what I know about statistics so that you have a better sense of the types of resources that I'm looking for. I'm a graduate student ...
8
votes
1answer
1k views

Estimating parameters for a binomial

First of all I'd like to precise that I'm not an expert of the subject. Suppose to have two random variables $X$ and $Y$ that are binomial, respectively $X\sim B(n_1,p)$ and $Y\sim B(n_2,p),$ note ...
9
votes
1answer
4k views

Moment generating function of the inner product of two gaussian random vectors

Can anybody please suggest how I can compute the moment generating function of the inner product of two Gaussian random vectors, each distributed as $\mathcal N(0,\sigma^2)$, independent of each other?...
26
votes
6answers
5k views

What is the mathematical difference between random- and fixed-effects?

I have found a lot on the internet regarding the interpretation of random- and fixed-effects. However I could not get a source pinning down the following: What is the mathematical difference between ...
23
votes
2answers
4k views

What is the intuition behind defining completeness in a statistic as being impossible to form an unbiased estimator of $0$ from it?

In classical statistics, there is a definition that a statistic $T$ of a set of data $y_1, \ldots, y_n$ is defined to be complete for a parameter $\theta$ it is impossible to form an unbiased ...
28
votes
2answers
8k views

What is the distribution of $R^2$ in linear regression under the null hypothesis? Why is its mode not at zero when $k>3$?

What is the distribution of the coefficient of determination, or R squared, $R^2$, in linear univariate multiple regression under the null hypothesis $H_0:\beta=0$? How does it depend on the number ...
23
votes
3answers
2k views

Having a conjugate prior: Deep property or mathematical accident?

Some distributions have conjugate priors and some do not. Is this distinction just an accident? That is, you do the math, and it works out one way or the other, but it does not really tell you ...
19
votes
2answers
1k views

Constructing a discrete r.v. having as support all the rationals in $[0,1]$

This is the constructivist sequel of this question. If we can't have a discrete uniform random variable having as support all the rationals in the interval $[0,1]$, then the next best thing is: ...
10
votes
2answers
764 views

Is there an elegant/insightful way to understand this linear regression identity for multiple $R^2$?

In linear regression I have come across a delightful result that if we fit the model $$E[Y] = \beta_1 X_1 + \beta_2 X_2 + c,$$ then, if we standardize and centre the $Y$, $X_1$ and $X_2$ data, $$R^...
8
votes
2answers
7k views

Transformation Chi-squared to Normal distribution

The relationship between the standard normal and the chi-squared distributions is well known. I was wondering though, is there a transformation that can lead from a $\chi^2 (1)$ back to a standard ...
2
votes
1answer
2k views

What is meant by using a probability distribution to model the output data for a regression problem?

Often a theoretical text will say something like, 'a probability distribution may be used to model the data' or, 'assume a probability distribution such as normal or Lognormal for the outputs'. ...
8
votes
1answer
284 views

Prove that $E(X^n)^{1/n}$ is non-decreasing for non-negative random variables

For a nonnegative random variable $X$, how to prove that $E(X^n)^{\frac1n}$ is nondecreasing in $n$?
93
votes
12answers
10k views

Who Are The Bayesians?

As one becomes interested in statistics, the dichotomy "Frequentist" vs. "Bayesian" soon becomes commonplace (and who hasn't read Nate Silver's The Signal and the Noise, anyway?). In talks and ...
30
votes
1answer
24k views

How are the standard errors computed for the fitted values from a logistic regression?

When you predict a fitted value from a logistic regression model, how are standard errors computed? I mean for the fitted values, not for the coefficients (which involves Fishers information matrix). ...
31
votes
3answers
3k views

How to rigorously define the likelihood?

The likelihood could be defined by several ways, for instance : the function $L$ from $\Theta\times{\cal X}$ which maps $(\theta,x)$ to $L(\theta \mid x)$ i.e. $L:\Theta\times{\cal X} \rightarrow \...
43
votes
3answers
17k views

Empirical relationship between mean, median and mode

For a unimodal distribution that is moderately skewed, we have the following empirical relationship between the mean, median and mode: $$ \text{(Mean - Mode)}\sim 3\,\text{(Mean - Median)} $$ How ...
30
votes
1answer
13k views

Maximum likelihood estimators for a truncated distribution

Consider $N$ independent samples $S$ obtained from a random variable $X$ that is assumed to follow a truncated distribution (e.g. a truncated normal distribution) of known (finite) minimum and maximum ...
12
votes
1answer
12k views

Distribution of sum of squares error for linear regression?

I know that distribution of sample variance $$ \sum\frac{(X_i-\bar{X})^2}{\sigma^2}\sim \chi^2_{(n-1)} $$ $$ \sum\frac{(X_i-\bar{X})^2}{n-1}\sim \frac{\sigma^2}{n-1}\chi^2_{(n-1)} $$ It's from the ...
15
votes
2answers
10k views

Deriving the bivariate Poisson distribution

I've recently encountered the bivariate Poisson distribution, but I'm a little confused as to how it can be derived. The distribution is given by: $P(X = x, Y = y) = e^{-(\theta_{1}+\theta_{2}+\...
5
votes
3answers
6k views

Why is prewhitening important?

I am writing code, geophysical time series processing. First step is to prewhiten values in time domain. Why is this step important? For example, I have found this on sas.com If, as is usually ...
8
votes
3answers
36k views

Prove F test is equal to T test squared

I need to show that F test is equal to T test squared, when the T test is for 2 independent groups and assuming variances are equal. I know that $F=\frac{MSB}{MSW}=\frac{SSB/k-1}{SSW/N-K}$ and I know ...
125
votes
21answers
85k views

What's the difference between probability and statistics?

What's the difference between probability and statistics, and why are they studied together?

1
2 3 4 5
14