6
$\begingroup$

I was reading about normal distributions and the Central Limit Theorem (CLT) and I came up with a question.

Why do we bother ourselves to use machine learning techniques when the CLT gives us the permission to assume that the mean of our data has a normal distribution? I can understand the usage of deep learning when we have images as data or any other type of data that does not have clear numerical representation. But when it comes to numerical-in-nature data, how can one justify the use of machine learning techniques over the simple statistical tools theoretically?

$\endgroup$
4
  • 2
    $\begingroup$ The use of distributions is not a machine learning thing, statisticians often use distributions other than the normal. My personal definition of a statistician is someone that know what to assume is Gaussian, which has some truth to it, but even then it is not as simple as it sounds. For a start, we are not always interested only in (additive) means. $\endgroup$ May 29, 2023 at 11:39
  • $\begingroup$ Maybe I'm wrong, But I always considered a machine learning algorithm, a procedure that aims to find the probability distribution of some phenomenon which we have gathered some data regarding it $\endgroup$ May 29, 2023 at 11:46
  • 1
    $\begingroup$ I'm afraid that is not a good definition of machine learning. Machine learning is essentially a branch of computational statistics, and is just a different approach to the same sort of tasks as conventional statistics. Density estimation is only one of those tasks. There are however machine learning algorithms that are not really probabilistic in nature, e.g. decision trees and do not estimate probabilities in any really sound sense. $\endgroup$ May 29, 2023 at 11:55
  • 4
    $\begingroup$ Finding the distribution is far, far different from estimating a mean! Are you perhaps misinterpreting the CLT as stating that (asymptotically) all distributions are Normal? $\endgroup$
    – whuber
    May 29, 2023 at 16:20

3 Answers 3

6
$\begingroup$

The CLT does not give one permission to assume the mean is normally distributed in any and all circumstances. The mean of a sample, when viewed as a random variable obtained from many different samples from a distribution, has its own distribution. And the CLT gives criteria when that distribution is normal or when it approaches normality.

First, if the underlying distribution is normally distributed, then the distribution of the means is normal...regardless of sample size.

Second, if the underlying distribution is not normally distributed, then the distribution of the means approaches a normal distribution for large enough sample sizes.

Thus, there are instances when the distribution of the means is most definitely not normally distributed.

$\endgroup$
4
  • $\begingroup$ Thank you for your elaborate explanation. What I meant, was exactly what you mentioned. My question still persists, if the CLT allows us to assume that the distribution of the means approaches a normal distribution, we can use the normal distribution in our estimations, knowing that CLT guarantees that error will be minimal. So, Why do we try to find other distributions using machine learning classification algorithm? What am I getting wrong here? $\endgroup$ May 29, 2023 at 14:55
  • $\begingroup$ In many cases, the algorithms are not addressing the mean of the sample, but individual responses from that distribution...and those individual responses are not always normal. $\endgroup$
    – Gregg H
    May 29, 2023 at 15:06
  • $\begingroup$ Thanks. That was enlightening. $\endgroup$ May 29, 2023 at 15:25
  • $\begingroup$ Also, just because a distribution approaches a normal distribution as the sample size grows does not mean that the distribution is almost normal. Especially at small sample sizes, the distribution can be very non-normal. $\endgroup$
    – andypea
    May 29, 2023 at 21:29
9
$\begingroup$

Under certain regularity conditions the CLT does indeed guarantee that a properly normalized sum of random variables converges to a Gaussian limit. But even in classical problems those conditions aren't always met.


The "law of rare events" gives one example of where a sum of independent random variables converges to a non-Gaussian limit.

Suppose we have a random process where at each step $n$, we observe $n$ independent binary random variables $X_{1n}, X_{2n}, \dots, X_{nn}$ where $P(X_{nk} = 1) = p_{nk}$ and $P(X_{nk} = 0) = 1 - p_{nk}$ (so they are Bernoulli with success probability $p_{nk}$). It turns out that if $\sum_{k=1}^n p_{nk} \to \lambda \in (0,\infty)$ as $n\to\infty$ and $\max_{1\leq k \leq n} p_{nk} \to 0$ then $\sum_{k=1}^n X_{nk} \stackrel{\text d}\to \text{Pois}(\lambda)$. This means that if we have a collection of binary random variables where the probability that any one of them is 1 goes to zero, but the collection as a whole maintains a steady expected number of 1s, then the sum will have a Poisson limit, not a Gaussian limit. This is theorem 3.6.1 in Durrett's Probability: Theory and Examples, available here.


Beyond this, we aren't always interested in means. Suppose we have $X_1, \dots, X_n \stackrel{\text{iid}}\sim \text{Unif}(\theta, \theta+1)$ and we want to estimate $\theta$. It turns out that $X_{(1)} := \min_{1\leq k \leq n} X_k$ is about the best estimator we could consider if we're using squared loss (in a minimax sense). If we normalize $X_{(1)}$ by subtracting $\theta$ and dividing by $1/n$ we get a non-Gaussian limit: $$ \begin{aligned} P\left(\frac{X_{(1)} - \theta}{1/n} \leq t\right) &= 1 - P\left(X_{(1)} - \theta > t/n\right) \\ &= 1 - P\left(X_1 - \theta > t/n\right)^n \\ &= 1 - (1 - t/n)^n \\ &\to e^{-t} \end{aligned} $$ as $n\to\infty$ hence $n(X_{(1)} - \theta) \stackrel{\text d}\to \text{Exp}(1)$, i.e. we get an Exponential distribution as our limit rather than a Gaussian.


These are two very classical examples of tidy problems where we don't get a Gaussian limit. If we're doing "real world" modeling then all bets can be off. We may have deep dependence relationships that prevent the CLT from applying, like if our data have temporal or spatial correlations. Or maybe it's a non-stationary process so there isn't one "mean" for things to be Gaussian around. Or we might be predicting/forecasting or studying non-asymptotic problems where there is no sense of "converging" to a limit. The CLT and friends are great but there is a lot of behavior that they fail to describe.

$\endgroup$
2
  • $\begingroup$ That was extraordinarily helpful. I have one question if you don't mind. You seem to pretty knowledgeable about this. I come from a background of computer science and I am trying to make progress in Machine Learning. For that matter, I'm studying Larry Wasserman, "All of Statistics" to get the basis for statistics and probability theory. Do you think that resource is sufficient for that purpose? if no, do you know any alternative option? Thanks again. $\endgroup$ Jun 2, 2023 at 23:38
  • $\begingroup$ @GeorgeWilhelmHegel glad this helped! That's definitely a good place to start. The book by Durrett that I linked to above is really popular for probability classes in graduate statistics programs. Another popular book is Casella and Berger's Statistical Inference. Between those two you'll have plenty to keep you busy! $\endgroup$
    – jld
    Jun 5, 2023 at 1:23
1
$\begingroup$

First, the CLT doesn't guarantee that the mean of samples will be normally distributed.

But more importantly, what you seem to be getting at is that the CLT is sufficient for the expected value of your estimator to be equal to the population parameter of the mean. This is known as being "unbiased". Taking the population mean (assuming it's known) does result in an unbiased estimator, but in a trivial and not very useful way.

Just having an estimator that's unbiased is an exceedingly unimpressive accomplishment. The mean is the absolute floor of machine learning performance. It's a benchmark that every other method had been improve upon, otherwise the method is pointless. For instance, suppose you're trying to predict what a student's college grades will be based on their high school grades and SAT score. The simplest machine learning model would be to simply take the mean of all the students at the college, and give that as output. That would an unbiased estimate, but in a pointless way.

The goal of machine learning is to see how much better you can do than just taking the mean of all the X overall. It's about getting as educated of a guess about each individual in a sample, not predicting what the average over the whole sample will be. It's about identifying features of particular Xs that are informative as possible, and getting the mean of the Xs that have those particular features, rather than of all the Xs. For instance, if you have a student with a high school GPA of 3.4 and an SAT of 1400, you want to know what the mean college GPA of all students with high school GPA of 3.4 and an SAT of 1400 is, not what the mean college GPA over all students is.

$\endgroup$
1
  • $\begingroup$ Extremely helpful and enlightening. thanks $\endgroup$ Jun 2, 2023 at 23:33

Not the answer you're looking for? Browse other questions tagged or ask your own question.