Questions tagged [convergence]

Convergence generally means that a sequence of a certain sample quantity approaches a constant as the sample size tends to infinity. Convergence is also a property of an iterative algorithm to stabilize on some aim value.

Filter by
Sorted by
Tagged with
26
votes
3answers
12k views

Extreme Value Theory - Show: Normal to Gumbel

The Maximum of $X_1,\dots,X_n. \sim$ i.i.d. Standardnormals converges to the Standard Gumbel Distribution according to Extreme Value Theory. How can we show that? We have $$P(\max X_i \leq x) = P(...
45
votes
6answers
4k views

Debunking wrong CLT statement

The central limit theorem (CLT) gives some nice properties about converging to a normal distribution. Prior to studying statistics formally, I was under the extremely wrong impression that the CLT ...
37
votes
6answers
12k views

Intuitive explanation of convergence in distribution and convergence in probability

What is the intuitive difference between a random variable converging in probability versus a random variable converging in distribution? I've read numerous definitions and mathematical equations, ...
13
votes
1answer
3k views

Is there a theorem that says that $\sqrt{n}\frac{\bar{X} - \mu}{S}$ converges in distribution to a normal as $n$ goes to infinity?

Let $X$ be any distribution with defined mean, $\mu$, and standard deviation, $\sigma$. The central limit theorem says that $$ \sqrt{n}\frac{\bar{X} - \mu}{\sigma} $$ converges in distribution to a ...
19
votes
2answers
4k views

Does log likelihood in GLM have guaranteed convergence to global maxima?

My questions are: Are generalized linear models (GLMs) guaranteed to converge to a global maximum? If so, why? Furthermore, what constraints are there on the link function to insure convexity? My ...
21
votes
1answer
4k views

Central limit theorem and the law of large numbers

I have a very beginner's question regarding the Central Limit Theorem (CLT): I am aware that the CLT states that a mean of i.i.d. random variables is approximately normal distributed (for $n \to \...
19
votes
3answers
3k views

Asymptotic consistency with non-zero asymptotic variance - what does it represent?

The issue has come up before, but I want to ask a specific question that will attempt to elicit an answer that will clarify (and classify) it: In "Poor Man's Asymptotics", one keeps a clear ...
13
votes
1answer
8k views

root-n consistent estimator, but root-n doesn't converge?

I've heard the term "root-n" consistent estimator' used many times. From the resources I've been instructed by, I thought that a "root-n" consistent estimator meant that: the estimator converges on ...
19
votes
2answers
5k views

Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
2
votes
2answers
2k views

Showing that estimator is consistent

Let $\hat{\theta}_n= -\frac{n}{\sum_{i=1}^n \log(X_i)}$, where $X_i$ are i.i.d. samples from distribution with pdf $\theta x^{\theta-1}$ for $x \in (0,1)$. How to prove that $\hat{\theta}_n$ is ...
21
votes
6answers
8k views

Intuitive understanding of the difference between consistent and asymptotically unbiased [duplicate]

I am trying to to get an intuitive understanding and feel for the difference and practical difference between the term consistent and asymptotically unbiased. I know their mathematical/statistical ...
16
votes
2answers
51k views

r glmer warnings: model fails to converge & model is nearly unidentifiable

I have seen questions about this on this forum, and I have also asked it myself in a previous post but I still haven't been able to solve my problem. Therefore I am trying again, formulating the ...
4
votes
1answer
3k views

Gibbs sampling convergence

In an astronomical context, the authors of a paper desire to use a Gibbs algorithm. Please note: I am inexperience in MCMC algorithms, and specifically in Gibbs sampling. What we want, in essence, is ...
5
votes
1answer
1k views

Convergence in distribution, probability, and 2nd mean

Let $\mathbb P(X=1) = \mathbb P(X=-1) = 1/2$. Define $$X_n = \begin {cases} X & \text{with probability } 1- \frac{1}{n}\\ e^n & \text{with probability } \frac{1}{n} \end {cases}$$ ...
7
votes
2answers
2k views

Limiting distribution of the first order statistic of a general distribution

Let $Z_i,Z_2,\ldots$ be IID Random Variables with density $f$. Suppose that $P(Z_i>0)=1$ and that $\lambda=\lim_{x \to 0+} f(x)>0$. How can I show that $X_n=n \times \min\{Z_i\}$ has a limiting ...
7
votes
5answers
2k views

Limit of $t$-distribution as $n$ goes to infinity

I found in my intro to stats textbook that $t$-distribution approaches the standard normal as $n$ goes to infinity. The textbook gives the density for $t$-distribution as follows, $$f(t)=\frac{\Gamma\...
5
votes
2answers
381 views

Why convergence in probability is defined as convergence to R.V.?

Wikipedia defines convergence in probability as A sequence ${X_n}$ of random variables converges in probability towards the random variable $X$ if for all ε > 0 $$\lim_{n\rightarrow\infty} P(|X_n-X|&...
7
votes
2answers
2k views

Convergence of moments of binomial to Poisson

As is well known, the $\mathsf{Binomial}(n,p)$ distribution converges to the $\mathsf{Poisson}(a)$ distribution as $n\rightarrow \infty$, $p\rightarrow 0$ with $np=a$. I'm pretty sure that the ...
19
votes
6answers
34k views

Why doesn't k-means give the global minimum?

I read that the k-means algorithm only converges to a local minimum and not to a global minimum. Why is this? I can logically think of how initialization could affect the final clustering and there is ...
19
votes
1answer
59k views

How to show that an estimator is consistent?

Is it enough to show that MSE = 0 as $n\rightarrow\infty$? I also read in my notes something about plim. How do I find plim and use it to show that the estimator is consistent?
17
votes
1answer
14k views

Gelman and Rubin convergence diagnostic, how to generalise to work with vectors?

The Gelman and Rubin diagnostic is used to check the convergence of multiple mcmc chains run in parallel. It compares the within-chain variance to the between-chain variance, the exposition is below: ...
7
votes
1answer
7k views

Convergence in probability and $L_2$ for normal random variables

In an answer here: Convergence of identically distributed normal random variables, the following lemma is mentioned: Lemma: Let $X_1, X_2, \ldots$ be a sequence of zero-mean normal random ...
12
votes
3answers
447 views

Regarding convergence in probability

Let $\{X_n\}_{n\geq 1}$ be a sequence of random variables s.t $X_n \to a$ in probability, where $a>0$ is a fixed constant. I'm trying to show the following: $$\sqrt{X_n} \to \sqrt{a}$$ and $$\frac{...
6
votes
2answers
5k views

Distribution function of maximum of n iid standard uniform random variables where n is poisson distributed

I am studying probability theory on my own and am trying to work the following problem in the book - Let $X_1, X_2, . . .$ be independent, $U(0, 1)$-distributed random variables, and let $Nm \in Po(m)...
6
votes
1answer
2k views

Markov chain convergence, total variation and KL divergence

I have a few related questions regarding the convergence of continuous-state Markov chains. The theorems that I found claim that Markov chains converge in total variation if they are $\phi$-...
5
votes
2answers
3k views

Convergence to a Uniform Distribution

$\newcommand{\floor}[1]{\left\lfloor #1 \right\rfloor} $ Show that if $P(X_n = i/n)=1/n$ for every $i = 1,...,n$, then $X_n$ converges in distribution to a uniformly distributed random variable $X$. ...
6
votes
1answer
148 views

Can we go from $X_n = \mu + O_p(n^{-1})$ to $E[X_n] = \mu + O(n^{-1})$?

Let $X_n$ be a uniformly integrable (UI) sequence of random variables. If we have $$ X_n = \mu + O_p(n^{-1}), $$ then for $0 \le \delta < 1$ this implies $$ X_n = \mu + o_p(n^{-\delta}) \quad \quad ...
6
votes
2answers
817 views

Limit of a convolution and sum of distribution functions

I need to prove an induction step. $X_i$ are independently distributed with the distribution function $1-F_i=x^{-\alpha}L_{i}(x)$ where $\alpha \geq 0$ and $L_{i}(x)$ is regularly varying (If the ...
1
vote
3answers
1k views

Prove that this doesn't converge almost sure to 0

Suppose we have $X_n$ a random variable, that can take two values: $X_n = \begin{cases} 0, & \text{with probability 1 - $\frac{1}{2n}$,} \\ n, & \text{with probability $\frac{1}{2n}$} \end{...
8
votes
1answer
333 views

Convergence of the Matérn covariance function to the squared exponential

The Matérn covariance function converges to the squared exponential covariance function. Many sources, amongst them the GPML book and Wikipedia, state this result. None of them provide details. I ...
4
votes
2answers
205 views

Chebychev’s Weak Law of Large Numbers

This theorem is on Econometric Analysis (7th edition) by Greene (2012), Page 1071. It states that "If $x_i$, $i=1,2,...,n$ is a sample of observations such that $E(x_i)=\mu_i<\infty$ and $var(x_i)=\...
4
votes
1answer
3k views

Convergence of identically distributed normal random variables

I had this example in my machine learning lecture. Let $X_2,\ldots,X_n$ be identically distributed (but not independent) copies of $X_1$ drawn from $\mathcal N(0,1)$. Then $X_n$ converges to $Y = -...
2
votes
1answer
102 views

Rate of convergence of $\hat Q_{xx}^{-1} = \left(\frac{\mathbf{X}^T \mathbf{X}}{n}\right)^{-1}$ to the probability limit?

Consider the simple linear regression model. $$ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i, \quad \quad \quad \quad i = 1,2,\dots,n. $$ Let $\mu_x$ and $\sigma_x^2$ represent the mean and variance of ...
2
votes
1answer
758 views

MCMC convergence, analytic derivations, Monte Carlo error

I'm trying to figure out some convergence statements on an MCMC example. The setup is: I'm generating data samples as observations from a (known) deterministic parameter, say $s$ (using a forward ...
2
votes
1answer
455 views

Convergence issues with lme4 1.1-20 for models that converged when using earlier version of lme4

I am encountering convergence problems with some models after updating to lme4 1.1-20 that I did not encounter with earlier versions of lme4 (in particular, lme4 1.1-15). I am encountering these new ...
0
votes
0answers
44 views

Convergence in Distribution in Order Statistics

Let $X_1, X_2, \ldots$ be iid from Exp$(\theta)$ with density function $f(x) = \frac{1}{\theta}e^{-\frac{x}{\theta}}$. (a) Find the limiting distribution of $M_n = Y_1 - \theta\ln(n)$ and $T_n = nY_n$,...
18
votes
3answers
1k views

A dynamical systems view of the Central Limit Theorem?

(Originally posted on MSE.) I have seen many heuristic discussions of the classical central limit theorem speak of the normal distribution (or any of the stable distributions) as an "attractor" in ...
9
votes
1answer
2k views

Why does MAP converge to MLE?

In Kevin Murphy's "Machine learning: A probabilistic perspective", chapter 3.2, the author demonstrates Bayesian concept learning on an example called "number game": After observing $N$ samples from $...
9
votes
1answer
2k views

Deriving K-means algorithm as a limit of Expectation Maximization for Gaussian Mixtures

Christopher Bishop defines the expected value of the complete-data log likelihood function (i.e. assuming that we are given both the observable data X as well as the latent data Z) as follows: $$ \...
12
votes
2answers
2k views

Is Slutsky's theorem still valid when two sequences both converge to a non-degenerate random variable?

I am confused about some details about Slutsky's theorem: Let $\{X_n\}$, $\{Y_n\}$ be two sequences of scalar/vector/matrix random elements. If $X_n$ converges in distribution to a random ...
11
votes
2answers
666 views

What happens to the likelihood ratio as more and more data is gathered?

Let $f$, $g$ and $h$ be densities and suppose you have $x_i \sim h$, $i \in \mathbb{N}$. What happens to the likelihood ratio $$ \prod_{i=1}^n \frac{f(x_i)}{g(x_i)} $$ as $n \rightarrow \infty$ ? (...
8
votes
2answers
4k views

What are some reasons iteratively reweighted least squares would not converge when used for logistic regression?

I've been using the glm.fit function in R to fit parameters to a logistic regression model. By default, glm.fit uses iteratively reweighted least squares to fit the parameters. What are some reasons ...
2
votes
2answers
3k views

How do I interpret my validation and training loss curve if there is a large difference between the two which closes in sharply

my CNN is meant to classify an image as one out of around 30 categories. I am training on 6400 samples using a batch size of 128. I am using Keras/ Tensorflow Architecture is Conv + Batch ...
17
votes
1answer
1k views

High-dimensional regression: why is $\log p/n$ special?

I am trying to read up on the research in the area of high-dimensional regression; when $p$ is larger than $n$, that is, $p >> n$. It seems like the term $\log p/n$ appears often in terms of ...
9
votes
3answers
212 views

When do posteriors converge to a point mass?

What are the necessary conditions for a model's posterior to converge to a point mass in the limit of infinite observations? What is an example that breaks this convergence result? Off the top of my ...
7
votes
2answers
437 views

Econometrics text claims that convergence in distribution implies convergence in moments

The following lemma can be found in Hayashi's Econometrics: Lemma 2.1 (convergence in distribution and in moments): Let $\alpha_{sn}$ be the $s$-th moment of $z_{n}$, and $\lim_{n\to\infty}\alpha_{sn}...
6
votes
1answer
545 views

Proving Bernoulli is the limit of Beta

It's clear to me by inspection that if we fix $\beta = \frac{1-\mu}{\mu} \alpha$ (thereby fixing the mean) and let $\alpha \rightarrow 0$, the Beta distribution approaches a Bernoulli($\mu$) ...
5
votes
1answer
626 views

Mean square convergence of linear processes

In Brockwell and Davis's book (Introduction to time series and forecasting), a linear process is defined as $ X_t = \sum_{j=-\infty}^{\infty} \psi_jZ_{t-j}$ where $Z_{t} \sim WN(0, \sigma^2)$, $\...
4
votes
1answer
3k views

Convergence of EM for Mixture of Gaussians

Is the Mixture of Gaussians model (an example of latent class analysis) gauranteed to converge on a viable solution even on Unimodal data using the Expectation Maximization algorithm to estimate the ...
6
votes
1answer
155 views

If $X_n \sim \text{Beta}(n, n)$ Show that $[X_n - \text{E}(X_n)]/\sqrt{\text{Var}(X_n)} \stackrel{D}{\longrightarrow} N(0,1)$

Let $X_n \sim \mathbf{B}(n,n)$ (Beta distribution), with pdf $$ f_n(x) = \frac{1}{\text{B}(n,n)}x^{n-1}(1 - x)^{n-1},~~ x \in (0,1). $$ Knowing that $\text{E}(X_n) = 1/2$ and that $\text{Var}(X_n) = 1/...