Questions tagged [convergence]

Convergence generally means that a sequence of a certain sample quantity approaches a constant as the sample size tends to infinity. Convergence is also a property of an iterative algorithm to stabilize on some aim value.

Filter by
Sorted by
Tagged with
52 votes
6 answers
5k views

Debunking wrong CLT statement

The central limit theorem (CLT) gives some nice properties about converging to a normal distribution. Prior to studying statistics formally, I was under the extremely wrong impression that the CLT ...
user avatar
  • 33.8k
27 votes
3 answers
14k views

Extreme Value Theory - Show: Normal to Gumbel

The Maximum of $X_1,\dots,X_n. \sim$ i.i.d. Standardnormals converges to the Standard Gumbel Distribution according to Extreme Value Theory. How can we show that? We have $$P(\max X_i \leq x) = P(...
user avatar
  • 1,151
13 votes
1 answer
3k views

Is there a theorem that says that $\sqrt{n}\frac{\bar{X} - \mu}{S}$ converges in distribution to a normal as $n$ goes to infinity?

Let $X$ be any distribution with defined mean, $\mu$, and standard deviation, $\sigma$. The central limit theorem says that $$ \sqrt{n}\frac{\bar{X} - \mu}{\sigma} $$ converges in distribution to a ...
user avatar
  • 131
42 votes
6 answers
14k views

Intuitive explanation of convergence in distribution and convergence in probability

What is the intuitive difference between a random variable converging in probability versus a random variable converging in distribution? I've read numerous definitions and mathematical equations, ...
user avatar
  • 1,173
19 votes
2 answers
5k views

Does log likelihood in GLM have guaranteed convergence to global maxima?

My questions are: Are generalized linear models (GLMs) guaranteed to converge to a global maximum? If so, why? Furthermore, what constraints are there on the link function to insure convexity? My ...
user avatar
24 votes
1 answer
4k views

Central limit theorem and the law of large numbers

I have a very beginner's question regarding the Central Limit Theorem (CLT): I am aware that the CLT states that a mean of i.i.d. random variables is approximately normal distributed (for $n \to \...
user avatar
  • 1,051
23 votes
2 answers
8k views

Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
user avatar
19 votes
3 answers
3k views

Asymptotic consistency with non-zero asymptotic variance - what does it represent?

The issue has come up before, but I want to ask a specific question that will attempt to elicit an answer that will clarify (and classify) it: In "Poor Man's Asymptotics", one keeps a clear ...
user avatar
15 votes
1 answer
9k views

root-n consistent estimator, but root-n doesn't converge?

I've heard the term "root-n" consistent estimator' used many times. From the resources I've been instructed by, I thought that a "root-n" consistent estimator meant that: the estimator converges on ...
user avatar
  • 2,019
24 votes
6 answers
12k views

Intuitive understanding of the difference between consistent and asymptotically unbiased [duplicate]

I am trying to to get an intuitive understanding and feel for the difference and practical difference between the term consistent and asymptotically unbiased. I know their mathematical/statistical ...
user avatar
  • 10.5k
2 votes
2 answers
3k views

Showing that estimator is consistent

Let $\hat{\theta}_n= -\frac{n}{\sum_{i=1}^n \log(X_i)}$, where $X_i$ are i.i.d. samples from distribution with pdf $\theta x^{\theta-1}$ for $x \in (0,1)$. How to prove that $\hat{\theta}_n$ is ...
user avatar
  • 463
18 votes
2 answers
66k views

r glmer warnings: model fails to converge & model is nearly unidentifiable

I have seen questions about this on this forum, and I have also asked it myself in a previous post but I still haven't been able to solve my problem. Therefore I am trying again, formulating the ...
user avatar
18 votes
1 answer
15k views

Gelman and Rubin convergence diagnostic, how to generalise to work with vectors?

The Gelman and Rubin diagnostic is used to check the convergence of multiple mcmc chains run in parallel. It compares the within-chain variance to the between-chain variance, the exposition is below: ...
user avatar
  • 285
4 votes
1 answer
3k views

Gibbs sampling convergence

In an astronomical context, the authors of a paper desire to use a Gibbs algorithm. Please note: I am inexperience in MCMC algorithms, and specifically in Gibbs sampling. What we want, in essence, is ...
user avatar
  • 229
8 votes
5 answers
4k views

Limit of $t$-distribution as $n$ goes to infinity

I found in my intro to stats textbook that $t$-distribution approaches the standard normal as $n$ goes to infinity. The textbook gives the density for $t$-distribution as follows, $$f(t)=\frac{\Gamma\...
user avatar
5 votes
1 answer
2k views

Convergence in distribution, probability, and 2nd mean

Let $\mathbb P(X=1) = \mathbb P(X=-1) = 1/2$. Define $$X_n = \begin {cases} X & \text{with probability } 1- \frac{1}{n}\\ e^n & \text{with probability } \frac{1}{n} \end {cases}$$ ...
user avatar
7 votes
2 answers
3k views

Limiting distribution of the first order statistic of a general distribution

Let $Z_i,Z_2,\ldots$ be IID Random Variables with density $f$. Suppose that $P(Z_i>0)=1$ and that $\lambda=\lim_{x \to 0+} f(x)>0$. How can I show that $X_n=n \times \min\{Z_i\}$ has a limiting ...
user avatar
  • 18.7k
7 votes
2 answers
715 views

Why convergence in probability is defined as convergence to R.V.?

Wikipedia defines convergence in probability as A sequence ${X_n}$ of random variables converges in probability towards the random variable $X$ if for all ε > 0 $$\lim_{n\rightarrow\infty} P(|X_n-X|&...
user avatar
  • 195
8 votes
2 answers
2k views

Convergence of moments of binomial to Poisson

As is well known, the $\mathsf{Binomial}(n,p)$ distribution converges to the $\mathsf{Poisson}(a)$ distribution as $n\rightarrow \infty$, $p\rightarrow 0$ with $np=a$. I'm pretty sure that the ...
user avatar
20 votes
6 answers
41k views

Why doesn't k-means give the global minimum?

I read that the k-means algorithm only converges to a local minimum and not to a global minimum. Why is this? I can logically think of how initialization could affect the final clustering and there is ...
user avatar
20 votes
1 answer
67k views

How to show that an estimator is consistent?

Is it enough to show that MSE = 0 as $n\rightarrow\infty$? I also read in my notes something about plim. How do I find plim and use it to show that the estimator is consistent?
user avatar
19 votes
3 answers
1k views

A dynamical systems view of the Central Limit Theorem?

(Originally posted on MSE.) I have seen many heuristic discussions of the classical central limit theorem speak of the normal distribution (or any of the stable distributions) as an "attractor" in ...
user avatar
7 votes
1 answer
9k views

Convergence in probability and $L_2$ for normal random variables

In an answer here: Convergence of identically distributed normal random variables, the following lemma is mentioned: Lemma: Let $X_1, X_2, \ldots$ be a sequence of zero-mean normal random ...
user avatar
7 votes
1 answer
219 views

If $X_n \sim \text{Beta}(n, n)$ Show that $[X_n - \text{E}(X_n)]/\sqrt{\text{Var}(X_n)} \stackrel{D}{\longrightarrow} N(0,1)$

Let $X_n \sim \mathbf{B}(n,n)$ (Beta distribution), with pdf $$ f_n(x) = \frac{1}{\text{B}(n,n)}x^{n-1}(1 - x)^{n-1},~~ x \in (0,1). $$ Knowing that $\text{E}(X_n) = 1/2$ and that $\text{Var}(X_n) = 1/...
user avatar
  • 529
12 votes
3 answers
517 views

Regarding convergence in probability

Let $\{X_n\}_{n\geq 1}$ be a sequence of random variables s.t $X_n \to a$ in probability, where $a>0$ is a fixed constant. I'm trying to show the following: $$\sqrt{X_n} \to \sqrt{a}$$ and $$\frac{...
user avatar
6 votes
2 answers
5k views

Distribution function of maximum of n iid standard uniform random variables where n is poisson distributed

I am studying probability theory on my own and am trying to work the following problem in the book - Let $X_1, X_2, . . .$ be independent, $U(0, 1)$-distributed random variables, and let $Nm \in Po(m)...
user avatar
  • 115
8 votes
2 answers
562 views

Econometrics text claims that convergence in distribution implies convergence in moments

The following lemma can be found in Hayashi's Econometrics: Lemma 2.1 (convergence in distribution and in moments): Let $\alpha_{sn}$ be the $s$-th moment of $z_{n}$, and $\lim_{n\to\infty}\alpha_{sn}...
user avatar
6 votes
1 answer
2k views

Markov chain convergence, total variation and KL divergence

I have a few related questions regarding the convergence of continuous-state Markov chains. The theorems that I found claim that Markov chains converge in total variation if they are $\phi$-...
user avatar
  • 5,812
6 votes
2 answers
5k views

Convergence to a Uniform Distribution

$\newcommand{\floor}[1]{\left\lfloor #1 \right\rfloor} $ Show that if $P(X_n = i/n)=1/n$ for every $i = 1,...,n$, then $X_n$ converges in distribution to a uniformly distributed random variable $X$. ...
user avatar
  • 345
3 votes
2 answers
962 views

Does a Binomial converge to Poisson or Normal?

I have read the answer here. Here the distinction is that If $n\to\infty$ and $p\to0$ while $np$ approaches some positive number $\lambda,$ then the binomial distribution approaches a Poisson ...
user avatar
6 votes
1 answer
305 views

Can we go from $X_n = \mu + O_p(n^{-1})$ to $E[X_n] = \mu + O(n^{-1})$?

Let $X_n$ be a uniformly integrable (UI) sequence of random variables. If we have $$ X_n = \mu + O_p(n^{-1}), $$ then for $0 \le \delta < 1$ this implies $$ X_n = \mu + o_p(n^{-\delta}) \quad \quad ...
user avatar
  • 860
6 votes
2 answers
871 views

Limit of a convolution and sum of distribution functions

I need to prove an induction step. $X_i$ are independently distributed with the distribution function $1-F_i=x^{-\alpha}L_{i}(x)$ where $\alpha \geq 0$ and $L_{i}(x)$ is regularly varying (If the ...
user avatar
  • 1,179
1 vote
3 answers
1k views

Prove that this doesn't converge almost sure to 0

Suppose we have $X_n$ a random variable, that can take two values: $X_n = \begin{cases} 0, & \text{with probability 1 - $\frac{1}{2n}$,} \\ n, & \text{with probability $\frac{1}{2n}$} \end{...
user avatar
  • 293
8 votes
1 answer
426 views

Convergence of the Matérn covariance function to the squared exponential

The Matérn covariance function converges to the squared exponential covariance function. Many sources, amongst them the GPML book and Wikipedia, state this result. None of them provide details. I ...
user avatar
4 votes
2 answers
249 views

Chebychev’s Weak Law of Large Numbers

This theorem is on Econometric Analysis (7th edition) by Greene (2012), Page 1071. It states that "If $x_i$, $i=1,2,...,n$ is a sample of observations such that $E(x_i)=\mu_i<\infty$ and $var(x_i)=\...
user avatar
  • 463
4 votes
1 answer
3k views

Convergence of identically distributed normal random variables

I had this example in my machine learning lecture. Let $X_2,\ldots,X_n$ be identically distributed (but not independent) copies of $X_1$ drawn from $\mathcal N(0,1)$. Then $X_n$ converges to $Y = -...
user avatar
3 votes
1 answer
179 views

Rate of convergence of $\hat Q_{xx}^{-1} = \left(\frac{\mathbf{X}^T \mathbf{X}}{n}\right)^{-1}$ to the probability limit?

Consider the simple linear regression model. $$ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \quad \quad \quad \quad i = 1,2,\dots,n. $$ Let $\mu_x$ and $\sigma_x^2$ represent the mean and variance of ...
user avatar
  • 860
2 votes
1 answer
629 views

Convergence issues with lme4 1.1-20 for models that converged when using earlier version of lme4

I am encountering convergence problems with some models after updating to lme4 1.1-20 that I did not encounter with earlier versions of lme4 (in particular, lme4 1.1-15). I am encountering these new ...
user avatar
  • 23
2 votes
1 answer
970 views

MCMC convergence, analytic derivations, Monte Carlo error

I'm trying to figure out some convergence statements on an MCMC example. The setup is: I'm generating data samples as observations from a (known) deterministic parameter, say $s$ (using a forward ...
user avatar
  • 65
0 votes
0 answers
146 views

Convergence in Distribution in Order Statistics

Let $X_1, X_2, \ldots$ be iid from Exp$(\theta)$ with density function $f(x) = \frac{1}{\theta}e^{-\frac{x}{\theta}}$. (a) Find the limiting distribution of $M_n = Y_1 - \theta\ln(n)$ and $T_n = nY_n$,...
user avatar
  • 333
9 votes
1 answer
3k views

Why does MAP converge to MLE?

In Kevin Murphy's "Machine learning: A probabilistic perspective", chapter 3.2, the author demonstrates Bayesian concept learning on an example called "number game": After observing $N$ samples from $...
user avatar
  • 10.4k
9 votes
1 answer
3k views

Deriving K-means algorithm as a limit of Expectation Maximization for Gaussian Mixtures

Christopher Bishop defines the expected value of the complete-data log likelihood function (i.e. assuming that we are given both the observable data X as well as the latent data Z) as follows: $$ \...
user avatar
  • 467
18 votes
2 answers
2k views

Optimizing OLS with Newton's Method

Can ordinary least squares regression be solved with Newton's method? If so, how many steps would be required to achieve convergence? I know that Newton's method works on twice differentiable ...
user avatar
  • 111
12 votes
2 answers
721 views

What happens to the likelihood ratio as more and more data is gathered?

Let $f$, $g$ and $h$ be densities and suppose you have $x_i \sim h$, $i \in \mathbb{N}$. What happens to the likelihood ratio $$ \prod_{i=1}^n \frac{f(x_i)}{g(x_i)} $$ as $n \rightarrow \infty$ ? (...
user avatar
  • 769
17 votes
1 answer
1k views

High-dimensional regression: why is $\log p/n$ special?

I am trying to read up on the research in the area of high-dimensional regression; when $p$ is larger than $n$, that is, $p >> n$. It seems like the term $\log p/n$ appears often in terms of ...
user avatar
  • 14.5k
13 votes
3 answers
4k views

Understanding the concept of "Bounded in probability"

My statistics book defines the concept of "bounded in probability" in the following way: Definition 5.2.2 (Bounded in Probability). We say that the sequence of random variables $\{X_n\}$ is ...
user avatar
  • 275
12 votes
2 answers
3k views

Is Slutsky's theorem still valid when two sequences both converge to a non-degenerate random variable?

I am confused about some details about Slutsky's theorem: Let $\{X_n\}$, $\{Y_n\}$ be two sequences of scalar/vector/matrix random elements. If $X_n$ converges in distribution to a random ...
user avatar
  • 299
8 votes
2 answers
4k views

What are some reasons iteratively reweighted least squares would not converge when used for logistic regression?

I've been using the glm.fit function in R to fit parameters to a logistic regression model. By default, glm.fit uses iteratively reweighted least squares to fit the parameters. What are some reasons ...
user avatar
  • 1,831
6 votes
1 answer
4k views

Big O and little o notation explained?

Throughout many of my statistics classes, I've had my professors attempt to explain big "O" (oh) and little "o" notation (especially as it involves convergence, the central limit theorem, and the ...
user avatar
  • 10.5k
2 votes
2 answers
4k views

How do I interpret my validation and training loss curve if there is a large difference between the two which closes in sharply

my CNN is meant to classify an image as one out of around 30 categories. I am training on 6400 samples using a batch size of 128. I am using Keras/ Tensorflow Architecture is Conv + Batch ...
user avatar