I'm currently going through the textbook Introduction to Machine Learning 4e (Ethem Alpaydin) to brush up on my ML basics and had a question regarding the chapter on multivariate methods.

More specifically:

Say that we have a data matrix as follows: $$ \mathbf{X} = \begin{bmatrix} X_1^1 & X_2^1 \quad \cdots \quad X_d^1 \\ X_1^2 & X_2^2 \quad \cdots \quad X_d^2 \\ \vdots \\ X_1^N & X_2^N \quad \cdots \quad X_d^N \end{bmatrix} $$ where each column represents a feature (or attribute) and each row represents a data sample. Given such a multivariate sample, estimates for these parameters can be calculated as follows: The maximum likelihood estimator for the mean is the sample mean, $\mathbf{m}$. Its $i$th dimension is the average of the $i$th column of $\mathbf{X}$: $$ \begin{align} & \mathbf{m} = \frac{\sum_{t = 1}^N \mathbf{x}^t}{N} \\ \text{where}\quad & m_i = \frac{\sum_{t = 1}^N x_i^t}{N} \ (i = 1, \dots, d) \end{align} $$ The estimator of the covariance matrix $\mathbf{\Sigma}$ is $\mathbf{S}$, the sample covariance matrix, with entries: $$ \begin{align} & s_i^2 = \frac{\sum_{t = 1}^N (x_i^t - m_i)^2}{N} \\ & s_{i, j} = \frac{\sum_{t = 1}^N (x_i^t - m_i)(x_j^t - m_j)}{N} \end{align} $$ These are biased estimates, but if in an application the estimates vary significantly depending on whether we divide by $N$ or $N - 1$, we are in serious trouble anyway.

I put the part that I don't understand in bold font. I'm just curious why it would be a problem if these estimates varied greatly depending on whether we divide by $N$ or $N - 1$. My intuition tells me that typically the estimates wouldn't be that different, but I'm not well versed in statistics so I'm not too sure.

Any feedback is appreciated. Thanks.


The comment seems to be a way of saying that we like large sample sizes in machine learning.

The numerator is the numerator, whether you divide by $N$ or $N-1$, so all that matters to our discussion is the denominator.

The only way for the two fractions to differ immensely is if $N$ and $N-1$ are very different, say if $\frac{N}{N-1}$ is much greater than $1$ (whatever "much greater" means to us).

That only happens when $N$ is small. If we have $N=1000000$, $\frac{N}{N-1}$ is about 1.

$$\underset{N\rightarrow \infty}{\text{lim}} \dfrac{N}{N-1} = 1$$

So the comment seems to be a way of saying that we like large sample sizes in machine learning.

  • 1
    $\begingroup$ Upvote but I find (N - 1)/N more natural. The fraction becomes 1 - 1/N and the magnitude of 1/N gives how "much greater" the difference between the two is. $\endgroup$ – Rui Barradas Sep 13 '20 at 18:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.