Questions tagged [convergence]

Convergence generally means that a sequence of a certain sample quantity approaches a constant as the sample size tends to infinity. Convergence is also a property of an iterative algorithm to stabilize on some aim value.

Filter by
Sorted by
Tagged with
22
votes
2answers
10k views

Extreme Value Theory - Show: Normal to Gumbel

The Maximum of $X_1,\dots,X_n. \sim$ i.i.d. Standardnormals converges to the Standard Gumbel Distribution according to Extreme Value Theory. How can we show that? We have $$P(\max X_i \leq x) = P(...
11
votes
1answer
2k views

Is there a theorem that says that $\sqrt{n}\frac{\bar{X} - \mu}{S}$ converges in distribution to a normal as $n$ goes to infinity?

Let $X$ be any distribution with defined mean, $\mu$, and standard deviation, $\sigma$. The central limit theorem says that $$ \sqrt{n}\frac{\bar{X} - \mu}{\sigma} $$ converges in distribution to a ...
19
votes
1answer
3k views

Central limit theorem and the law of large numbers

I have a very beginner's question regarding the Central Limit Theorem (CLT): I am aware that the CLT states that a mean of i.i.d. random variables is approximately normal distributed (for $n \to \...
17
votes
1answer
3k views

Does log likelihood in GLM have guaranteed convergence to global maxima?

My questions are: Are generalized linear models (GLMs) guaranteed to converge to a global maximum? If so, why? Furthermore, what constraints are there on the link function to insure convexity? My ...
18
votes
3answers
2k views

Asymptotic consistency with non-zero asymptotic variance - what does it represent?

The issue has come up before, but I want to ask a specific question that will attempt to elicit an answer that will clarify (and classify) it: In "Poor Man's Asymptotics", one keeps a clear ...
10
votes
1answer
6k views

root-n consistent estimator, but root-n doesn't converge?

I've heard the term "root-n" consistent estimator' used many times. From the resources I've been instructed by, I thought that a "root-n" consistent estimator meant that: the estimator converges on ...
32
votes
5answers
8k views

Intuitive explanation of convergence in distribution and convergence in probability

What is the intuitive difference between a random variable converging in probability versus a random variable converging in distribution? I've read numerous definitions and mathematical equations, ...
14
votes
1answer
2k views

Why second order SGD convergence methods are unpopular for deep learning?

It seems that, especially for deep learning, there are dominating very simple methods for optimizing SGD convergence like ADAM - nice overview: http://ruder.io/optimizing-gradient-descent/ They trace ...
1
vote
2answers
1k views

Showing that estimator is consistent

Let $\hat{\theta}_n= -\frac{n}{\sum_{i=1}^n \log(X_i)}$, where $X_i$ are i.i.d. samples from distribution with pdf $\theta x^{\theta-1}$ for $x \in (0,1)$. How to prove that $\hat{\theta}_n$ is ...
10
votes
2answers
35k views

r glmer warnings: model fails to converge & model is nearly unidentifiable

I have seen questions about this on this forum, and I have also asked it myself in a previous post but I still haven't been able to solve my problem. Therefore I am trying again, formulating the ...
36
votes
6answers
3k views

Debunking wrong CLT statement

The central limit theorem (CLT) gives some nice properties about converging to a normal distribution. Prior to studying statistics formally, I was under the extremely wrong impression that the CLT ...
3
votes
1answer
3k views

Gibbs sampling convergence

In an astronomical context, the authors of a paper desire to use a Gibbs algorithm. Please note: I am inexperience in MCMC algorithms, and specifically in Gibbs sampling. What we want, in essence, is ...
7
votes
2answers
2k views

Limiting distribution of the first order statistic of a general distribution

Let $Z_i,Z_2,\ldots$ be IID Random Variables with density $f$. Suppose that $P(Z_i>0)=1$ and that $\lambda=\lim_{x \to 0+} f(x)>0$. How can I show that $X_n=n \times \min\{Z_i\}$ has a limiting ...
5
votes
1answer
1k views

Convergence in distribution, probability, and 2nd mean

Let $\mathbb P(X=1) = \mathbb P(X=-1) = 1/2$. Define $$X_n = \begin {cases} X & \text{with probability } 1- \frac{1}{n}\\ e^n & \text{with probability } \frac{1}{n} \end {cases}$$ ...
1
vote
2answers
2k views

Consistency of unbiased estimator of error term variance in Multiple regression

Let $Y=X\beta+\epsilon$. We know that $\frac{e'e}{n-k}$ is an unbiased estimator of $Var(\epsilon)$, where $e$ is the vector of residuals, and $\epsilon$ is multivariate normal distributed in this ...
7
votes
2answers
1k views

Convergence of moments of binomial to Poisson

As is well known, the $\mathsf{Binomial}(n,p)$ distribution converges to the $\mathsf{Poisson}(a)$ distribution as $n\rightarrow \infty$, $p\rightarrow 0$ with $np=a$. I'm pretty sure that the ...
20
votes
6answers
27k views

Why doesn't k-means give the global minimum?

I read that the k-means algorithm only converges to a local minimum and not to a global minimum. Why is this? I can logically think of how initialization could affect the final clustering and there is ...
15
votes
1answer
13k views

Gelman and Rubin convergence diagnostic, how to generalise to work with vectors?

The Gelman and Rubin diagnostic is used to check the convergence of multiple mcmc chains run in parallel. It compares the within-chain variance to the between-chain variance, the exposition is below: ...
15
votes
4answers
4k views

Intuitive understanding of the difference between consistent and asymptotically unbiased

I am trying to to get an intuitive understanding and feel for the difference and practical difference between the term consistent and asymptotically unbiased. I know their mathematical/statistical ...
6
votes
1answer
6k views

Convergence in probability and $L_2$ for normal random variables

In an answer here: Convergence of identically distributed normal random variables, the following lemma is mentioned: Lemma: Let $X_1, X_2, \ldots$ be a sequence of zero-mean normal random ...
12
votes
3answers
386 views

Regarding convergence in probability

Let $\{X_n\}_{n\geq 1}$ be a sequence of random variables s.t $X_n \to a$ in probability, where $a>0$ is a fixed constant. I'm trying to show the following: $$\sqrt{X_n} \to \sqrt{a}$$ and $$\frac{...
6
votes
2answers
4k views

Distribution function of maximum of n iid standard uniform random variables where n is poisson distributed

I am studying probability theory on my own and am trying to work the following problem in the book - Let $X_1, X_2, . . .$ be independent, $U(0, 1)$-distributed random variables, and let $Nm \in Po(m)...
5
votes
1answer
2k views

Markov chain convergence, total variation and KL divergence

I have a few related questions regarding the convergence of continuous-state Markov chains. The theorems that I found claim that Markov chains converge in total variation if they are $\phi$-...
8
votes
1answer
232 views

Convergence of the Matérn covariance function to the squared exponential

The Matérn covariance function converges to the squared exponential covariance function. Many sources, amongst them the GPML book and Wikipedia, state this result. None of them provide details. I ...
6
votes
2answers
780 views

Limit of a convolution and sum of distribution functions

I need to prove an induction step. $X_i$ are independently distributed with the distribution function $1-F_i=x^{-\alpha}L_{i}(x)$ where $\alpha \geq 0$ and $L_{i}(x)$ is regularly varying (If the ...
4
votes
1answer
2k views

Convergence of identically distributed normal random variables

I had this example in my machine learning lecture. Let $X_2,\ldots,X_n$ be identically distributed (but not independent) copies of $X_1$ drawn from $\mathcal N(0,1)$. Then $X_n$ converges to $Y = -...
2
votes
1answer
315 views

Convergence issues with lme4 1.1-20 for models that converged when using earlier version of lme4

I am encountering convergence problems with some models after updating to lme4 1.1-20 that I did not encounter with earlier versions of lme4 (in particular, lme4 1.1-15). I am encountering these new ...
2
votes
1answer
539 views

MCMC convergence, analytic derivations, Monte Carlo error

I'm trying to figure out some convergence statements on an MCMC example. The setup is: I'm generating data samples as observations from a (known) deterministic parameter, say $s$ (using a forward ...
1
vote
0answers
50 views

Momentum updates average of g, Adagrad also of g^2 - any other interesting updated averages for SGD convergence?

Updating exponential moving average is a basic tool of SGD methods, starting with of gradient $g$ in momentum method to extract local linear trend from the statistics. Then e.g. Adagrad, ADAM family ...
16
votes
3answers
922 views

A dynamical systems view of the Central Limit Theorem?

(Originally posted on MSE.) I have seen many heuristic discussions of the classical central limit theorem speak of the normal distribution (or any of the stable distributions) as an "attractor" in ...
17
votes
1answer
51k views

How to show that an estimator is consistent?

Is it enough to show that MSE = 0 as $n\rightarrow\infty$? I also read in my notes something about plim. How do I find plim and use it to show that the estimator is consistent?
8
votes
1answer
2k views

Deriving K-means algorithm as a limit of Expectation Maximization for Gaussian Mixtures

Christopher Bishop defines the expected value of the complete-data log likelihood function (i.e. assuming that we are given both the observable data X as well as the latent data Z) as follows: $$ \...
12
votes
2answers
2k views

Is Slutsky's theorem still valid when two sequences both converge to a non-degenerate random variable?

I am confused about some details about Slutsky's theorem: Let $\{X_n\}$, $\{Y_n\}$ be two sequences of scalar/vector/matrix random elements. If $X_n$ converges in distribution to a random ...
11
votes
2answers
628 views

What happens to the likelihood ratio as more and more data is gathered?

Let $f$, $g$ and $h$ be densities and suppose you have $x_i \sim h$, $i \in \mathbb{N}$. What happens to the likelihood ratio $$ \prod_{i=1}^n \frac{f(x_i)}{g(x_i)} $$ as $n \rightarrow \infty$ ? (...
8
votes
2answers
4k views

What are some reasons iteratively reweighted least squares would not converge when used for logistic regression?

I've been using the glm.fit function in R to fit parameters to a logistic regression model. By default, glm.fit uses iteratively reweighted least squares to fit the parameters. What are some reasons ...
1
vote
2answers
2k views

How do I interpret my validation and training loss curve if there is a large difference between the two which closes in sharply

my CNN is meant to classify an image as one out of around 30 categories. I am training on 6400 samples using a batch size of 128. I am using Keras/ Tensorflow Architecture is Conv + Batch ...
7
votes
2answers
367 views

Econometrics text claims that convergence in distribution implies convergence in moments

The following lemma can be found in Hayashi's Econometrics: Lemma 2.1 (convergence in distribution and in moments): Let $\alpha_{sn}$ be the $s$-th moment of $z_{n}$, and $\lim_{n\to\infty}\alpha_{sn}...
2
votes
1answer
2k views

Convergence of EM for Mixture of Gaussians

Is the Mixture of Gaussians model (an example of latent class analysis) gauranteed to converge on a viable solution even on Unimodal data using the Expectation Maximization algorithm to estimate the ...
5
votes
1answer
311 views

Variance of $Z = X_1 + X_1 X_2 + X_1 X_2 X_3 +\cdots$

Here the $X_i$'s are i.i.d. and such that convergence in distribution for the infinite sum, is guaranteed. Probably the easiest case is when $X_i$ has a Bernouilli($p$) distribution, then $Z$ has a ...
11
votes
3answers
404 views

Yet another central limit theorem question

Let $\{X_n:n\ge1\}$ be a sequence of independent Bernoulli random variables with $$P\{X_k=1\}=1-P\{X_k=0\}=\frac{1}{k}.$$ Set $$S_n=\sum^{n}_{k=1}\left(X_k-\frac{1}{k}\right), \ B_n^2=\sum^{n}_{...
5
votes
1answer
1k views

Convergence of F(n,n) distribution to normal

Suppose that $X \sim F(n,n)$, an F distribution on $n$ and $n$ degrees of freedom. I'm trying to figure out why some literature state that $X$ converges in distribution to a normal distribution. My ...
4
votes
1answer
103 views

Proving that the product of 2 samples, when sorted in order, converges to the variance of the population

I have a question regarding the average value of the pairwise product of 2 ordered samples that are drawn from a random variable, $X$, which in this case is zero-mean Gaussian ($X \sim N(0,\sigma_x) $)...
4
votes
1answer
399 views

Mean square convergence of linear processes

In Brockwell and Davis's book (Introduction to time series and forecasting), a linear process is defined as $ X_t = \sum_{j=-\infty}^{\infty} \psi_jZ_{t-j}$ where $Z_{t} \sim WN(0, \sigma^2)$, $\...
3
votes
1answer
104 views

Can importance sampling be used as an actual sampling mechanism?

This question is a duplicate of How can we use importance sampling for drawing posterior distribution samples? , but that question seems to lack additional detail and goes unanswered (for more than 2 ...
3
votes
0answers
233 views

Convergence rate of the maximum of Weibull random variables to a Gumbel distribution

Given a sequence of iid samples $X_1, \dots, X_n,$ where each $X_i$ comes from a Weibull distribution with shape parameter $k$ and scale parameter $\lambda$. Then it is a well-known result that the ...
3
votes
1answer
347 views

Convergence of Metropolis Hastings with time varying proposal density

Suppose we have a Metropolis-Hastings sampler for a target distribution $f$, and we use a proposal density $Q_t$, that may depend on time $t$. By construction, $f$ is still an invariant density of ...
2
votes
0answers
114 views

If all components of a hierarchical model have not converged, can we say that any parameters have truly converged?

I'm working with a hierarchical regression model of the following form similar to that presented in Peter D. Hoff's book, A First Course in Bayesian Statistical Methods: $\boldsymbol{Y}_j \sim \text{...
2
votes
2answers
241 views

Convergence in probability and distribution

Let $P(K=k)=(1-\beta)^k\beta ; k=1,2,3,...$ Then it is required to show $\beta K$ converges in distribution to an exp (1) random variable as $\beta$ tends to zero. For this they have started with ...
1
vote
0answers
144 views

Saddle-free Newton method for SGD - while Newton attracts saddles, is it worth to actively replel them?

While 2nd order methods have many advantages, e.g. natural gradient (e.g. in L-BFGS) attracts to close zero gradient point, which is usually saddle. Other try to pretend that our very non-convex ...
1
vote
0answers
2k views

xtlogit: panel data transformation's recast to double makes model incomputable (STATA)

I am experiencing a very peculiar problem in STATA. I have a panel dataset with 8000 groups and 4 million observations. I want to run a fixed effects panel binary logistic regression (xtlogit). As ...