Questions tagged [mathematical-statistics]
Mathematical theory of statistics, concerned with formal definitions and general results.
2,159
questions with no upvoted or accepted answers
17
votes
0answers
806 views
Is there a general expression for ancillary statistics in exponential families?
An i.i.d sample $X_1,\dots,X_n$ from a scale family with c.d.f. $F(\frac{x}{\sigma})$ has $S(X)$ as an ancillary statistic if $S(X)$ depends on the sample only through $\frac{X_1}{X_n},\cdots,\frac{X_{...
10
votes
0answers
1k views
Are there unbiased, non-linear estimators with lower variance than the OLS estimator?
Consider an ordinary least squares model,
$$y = \beta X + \epsilon \qquad \epsilon\sim N(0, \sigma)$$
The Gauss-Markov theorem tells us that the ordinary least-squares (OLS) estimator is the minimum-...
8
votes
0answers
1k views
What does it mean to take the expectation with respect to a probability distribution?
I see this expectation in a lot of machine learning literature:
$$\mathbb{E}_{p(\mathbf{x};\mathbf{\theta})}[f(\mathbf{x};\mathbf{\phi})] = \int p(\mathbf{x};\mathbf{\theta}) f(\mathbf{x};\mathbf{\phi}...
8
votes
0answers
2k views
Taylor Series and Multivariate Delta Method
I asked this question on https://math.stackexchange.com/ but did not get any answer. Sorry for cross posting.
I'm trying to understand delta method for matrices and vectors to find the variance-...
7
votes
1answer
10k views
What does "def" above an equals sign mean?
I am reading this:
https://www.cs.ubc.ca/~murphyk/Papers/bayesGauss.pdf
and on equation (17), there is a def on top of the equal sign. What does this mean?
7
votes
0answers
993 views
Proof of Kolmogorov-Smirnov test
Could someone provide me a reference, preferably a book, where I can find detailed proofs and explanations of the Kolmogorov-Smirnov test (including the two-sample variant) and the derivation of the $...
7
votes
1answer
227 views
Finding the distribution of sample range for a Beta population
Let $X_1,X_2,\ldots,X_n$ be i.i.d random variables having density
$$f(x)=2(1-x)\mathbf1_{0<x<1}$$
I am trying to derive the distribution of the sample range $R=X_{(n)}-X_{(1)}$.
The usual way I ...
6
votes
0answers
44 views
Simulate correlate random variables with given marginal distribution where one is always larger
Is it possible to simulate pairs of random variables with a given marginal distribution and population correlation where one random variable is larger than the other?
More formally, I need to simulate ...
6
votes
0answers
107 views
How do we call a more extreme case of fat tails than a power law?
According to Wikipedia the most extreme case of a fat tail follows a power law:
The most extreme case of a fat tail is given by a distribution whose tail decays like a power law.
That is, if the ...
6
votes
0answers
276 views
Strange connection between Bernouilli, Uniform and Geometric distributions
Final update on 11/29/2019: I have worked on this a bit more, and wrote an article summarizing all the main findings. You can read it here.
Let us consider $Z = X_1 + X_1 X_2 + X_1 X_2 X_3 +\cdots$ ...
6
votes
0answers
91 views
French website Providing Instruction/Tutorials on Statistical Theory
This is somewhat of an odd question for CV, but since it's a question about statistical education, I think it falls within the scope of CV. Several years ago I stumbled across a French website that ...
6
votes
1answer
118 views
Asymptotic joint distribution of the sample medians of a collection and a sub-collection of i.i.d. random variables
Let $X_1,\dots, X_n$ denote i.i.d. random variables, with a smooth density $f$ having unique median $m$.
It is known that the sample median $M_n$ of $X_1,\dots, X_n$, $M_n = \operatorname{med}(X_1, \...
6
votes
0answers
208 views
Time evolution of a Bayesian posterior
I have a question regarding the time evolution of a quantity related to a Bayesian posterior.
Suppose we have binary parameter space $\{ s_1, s_2 \}$ with prior $(p, 1-p)$,
The data generating ...
6
votes
0answers
673 views
Intuitive explanation for Marchenko-Pastur law
I am looking for an intuitive reasoning behind the Marchenko Pastur law, which is described as a law of large numbers analog for random matrices. I know the law gives the probability density function ...
6
votes
0answers
221 views
Cox's Theorem: ignorance, objective priors, and the Mind Projection Fallacy
I've been trying to understand Cox's Theorem and the problems surrounding it. There's so much information on this topic that I've become confused as to the exact state of the theorem. I've gathered ...
6
votes
1answer
366 views
minimizer weighted linear regression
In a regression problem, with $y=X\theta+\epsilon$ and $X$ is an $n$ by $p$ matrix
the ‘weighted least squares estimate is the minimizer $\theta^{*}$ of $f(\theta)=\sum_{i=1}^{n}\omega_{i}(y_i-x_i^{'}\...
6
votes
0answers
2k views
Deriving the maximum likelihood for a generative classification model for K classes
In Christopher Bishop's book "Pattern Recognition and Machine learning", there is the following question:
Consider a generative classification model for $K$ classes defined by the prior class ...
6
votes
0answers
264 views
Reference Request: Information Geometry for Ridge Regression
I am reading the book "regression estimators" by Gruber 2010 where he uses this technique to compare Ridge Regressors, however he concentrates on deriving the mathematical results without ...
6
votes
0answers
158 views
Hypothesis test on the Euclidean length of an unknown vector
Question
Suppose I observe a vector $\mathbf{x}=[X_1 \ldots X_n]$, where each $X_i=m_i+n_i$, with $n_i$ being an independent zero-mean Gaussian random variable with variance $\sigma^2$ (i.e. $n_i\sim\...
6
votes
0answers
134 views
Orthogonal intersection in a Riemannian manifold
Let $S$ be the set of all probability distributions on $\mathbb{R}$ and $S_n=\{p_\theta\}$ be an $n$ dimensional submanifold of parameterized family of probability distributions on $\mathbb{R}$ where $...
5
votes
0answers
70 views
Proof: Nearest Neighbor classifier achieves Bayes rate asymptotically on countable domains
I am trying to understand in which situations the 1-NN classifier asymptotically attains the Bayes error rate. My intuition is that if the domain is countable, then 1-NN will asymptotically do as well ...
5
votes
0answers
74 views
Symbolic Formulae for Linear Mixed Models
I would like to understand how to create a good formula for a linear mixed model, using the Machines data set from the package ...
5
votes
0answers
75 views
Does Fisher scoring always outperform Newton optimization?
My understanding is that Fisher scoring has several advantages over Newton raphson optimization such as
Computational efficiency: if certain conditions are met (example:During MLE estimation, if link ...
5
votes
0answers
87 views
Inequalities on Fisher Information / expected second derivative?
Under some regularity conditions we can compute fisher information as
$ - \mathbb{E}_{\theta_0} [\frac{\partial}{\partial \theta^2} \ln f(x;\theta_0)] $
I was wondering if there are some kind of ...
5
votes
2answers
94 views
How to win this dice probability game?
The game is a variation of Pig. Here is how the game works:
There are about 20 players. Each round, a single six sided die is rolled. All players add that rolled number to their "bank." However, if a ...
5
votes
0answers
386 views
Why we really need the concept of "Local" Rademacher complexity?
Recently, I have been studying High-Dimensional Statistics: A Non-Asymptotic Viewpoint written by Martin J. Wainwright. In this book, the author uses a special complexity measure which is called Local ...
5
votes
0answers
246 views
Is an inadmissible estimator necessarily dominated by some admissible estimator
Basic example: $X$ has a $p$-variate iid standard Normal distribution; the sample mean is not admissible if $p>2$ and is dominated by the Stein shrinkage estimator.
However, the Stein shrinkage ...
5
votes
1answer
2k views
How do I get the p value of AD test using the results of scipy.stats.anderson()
I am conducting an Anderson Darling test for normality using scipy.stats.anderson() command in python. I am getting test statistic, critical values at various significance levels as the output.
The ...
5
votes
0answers
446 views
Why can't the complete class theorem be easily generalized to all locally-compact spaces?
So I was reading Christian P. Robert's The Bayesian Choice, going through the constellation of results related to complete class theorems, and I don't see why all of them are necessary. In particular, ...
5
votes
1answer
242 views
Covariance of order statistics convergence?
Suppose I have a sample $(X_1 \dots X_n)$ and $(Y_1 \dots Y_n)$, all of which are $N(0,1)$ random variables. I am interested in the asymptotic behaviour of
$$\frac{1}{n} \sum_{i=0}^n X_{(i)}Y_{(i)} $$...
5
votes
0answers
2k views
Calculating a confidence interval for a weighted sample
In a nutshell, I'd like to compute a confidence interval for some weighted sample day where the final value I'm seeking is a sum of different weighted samples. Illustrated here with a toy example set ...
5
votes
0answers
154 views
Finding the Cauchy Principal Value Mean of the pdf for $Z=XY$ where $\mathbb{E}[Y]$ does not exist
Let's first define the Cauchy Principal Value Mean (PVM). For a continuous random variable $V$ with pdf $f_{V}(v)$, the PVM of $f_{V}$ is
\begin{equation}
\mathrm{PV}(\mathbb{E}[V])=\lim_{a\to\infty}\...
5
votes
0answers
293 views
Variance of quotient of Poisson random variable and sum of the Poisson sample
Let
$$Y_1\sim \operatorname{Poisson}(\lambda_1)\\Y_2\sim \operatorname{Poisson}(\lambda_2),$$ where $Y_1$ and $Y_2$ are independent, and $\lambda_1, \lambda_2>0$.
What is the variance of $$\frac{...
5
votes
0answers
777 views
Under what conditions will a Bayesian posterior fail to converge to a point mass?
Let's say you have a Bayesian model:
$$\theta' \sim g(\theta|\mu) $$
$$ y \sim p(y|\theta')$$
And we have some data ($n$ data points) $\mathbf{y}_n$, which we will use to perform inference on $\...
5
votes
0answers
63 views
Why is the $\chi^{2}$ approximation for deviance GLM $\sim \operatorname{Binomial}(n_{i},\pi_{i})$ not valid when $n_{i} = 1$?
I know from McCullagh & Nelder's text (p.118) that the $\chi^{2}$ approximation for deviance for the binomial family is based on a limiting operation in which $n$, the number of observations, is ...
5
votes
0answers
337 views
Calculating Formula with logs and Weighted Average
I am trying to figure out an equation from the following paper by Cadena and Kovak (2016): http://pubs.aeaweb.org/doi/pdfplus/10.1257/app.20140095 on pages 264 and 265. I don't think the context is ...
5
votes
0answers
259 views
I'm not asking for a conjugate prior. Is there a distribution $p(x|y)$ that satisfies $\int p(x|y)Beta(y|a,b) dy = Beta(x| a', b')$?
I know the result of integrating a Gaussian against another Gaussian is still Gaussian, $$\int N(x|\mu_y,\sigma_y)N(y|\mu,\sigma) dy = N(x|\mu',\sigma')\quad.$$
Can I get the same form for Beta ...
5
votes
0answers
221 views
Cox's Theorem: the necessity of (un)countably additivity
I've been trying to understand Cox's Theorem and the problems surrounding it. There's so much information on this topic that I've become confused as to the exact state of the theorem. I've gathered ...
5
votes
1answer
403 views
Why low rank expansions can exploit the redundancy that exist between different feature channels and filters?
I read Jaderberg et al., 2014 paper about Speeding up Convolutional Neural Network with Low Rank Expansions. In the introduction, it is written in bold font:
Our key insight is to exploit the ...
5
votes
0answers
1k views
Kurtosis, bias, unbiased and statistics
I apologize ahead of time if this is too vague or meta to be a valid question.
I've been looking at Algorithms (Sedgewick & Wayne). They define a class stdstats. In that they define min, max, ...
5
votes
0answers
3k views
Help with a proof of Bayes classifier optimality
I have a class assignment to provide a proof that Bayes classifier for the two label version is optimal in that it's error rate is always ${\le}$ any other classifier.
I've never worked through a ...
5
votes
0answers
983 views
Conditional expectation everywhere non-zero while unconditional one zero?
I have a real-valued random variable $X$ that takes on positive and negative values, and
$$E(X)=0 \tag{A}$$
There is also another real-valued random variable $Y$, not independent from $X$, neither $...
5
votes
0answers
203 views
Distribution of $X'X$ if $X\in\mathbb{R}^{T \times N}$ and $X_i'\sim N(\mu,\sigma^2I_N)$
Let $x_i\in\mathbb{R}^N$ be multivariate normal distributed with mean vector $\mu\in\mathbb{R}^N$ and no correlation: $x_i\sim N(\mu,\sigma^2 I_N)$.
Given $T$ iid samples, define the matrix
$$X:=\left(...
5
votes
0answers
128 views
Relation between asymptotic relative efficiency for tests and estimators
The asymptotic relative efficiency for unbiased estimators is the limit of the ratio of the variances as the $n\rightarrow \infty$. Is there a relation to asymptotic relative efficiency according to ...
5
votes
0answers
309 views
Restriction matrix for a VAR
In New Introduction to Multiple Time Series Analysis by Luetkepohl (2005), section 5.2.1, it says that one can specify linear restraints for a VAR, $Y = \beta X + U$, in the form
$$
\operatorname{vec}{...
5
votes
0answers
673 views
Sum of independent Wishart with same degrees of freedom but different scale matrices
Is there any result showing that a sum of independent Wishart with same degrees of freedom but different scale matrices is a Wishart?
For example, if I have two random variables:
$$
Y \sim W_p(n,\...
5
votes
0answers
349 views
Derivation of prediction intervals for a normally distributed population with unknown population standard deviation
I have via the ISO standard 16269 found the solution to a problem that I've been working on. Based on a couple of independent samples from a normally distributed population, I would like to determine ...
4
votes
0answers
89 views
What conditions are needed for $a_n = O_p(n^d) \implies E[a_n] = O(n^d)$?
Let $X_n$ be a uniformly integrable sequence of random variables. In a recent question I asked about the possibility of converting Big $O_p$ convergence in probability of the sequence $X_n$ to Big $O$ ...
4
votes
0answers
132 views
What is it called when a random variable is weakly greater than another for all elements of the sample space?
Suppose I have random variables $(X_1,X_2)$ defined on a probability space $(\Omega, \mathcal{F},P)$ such that for any element $\omega \in \Omega$, $X_1(\omega) \geq X_2(\omega)$.
I'm looking to work ...
