# Why is it bad if the estimates vary greatly depending on whether we divide by N or (N - 1) in multivariate analysis?

I'm currently going through the textbook Introduction to Machine Learning 4e (Ethem Alpaydin) to brush up on my ML basics and had a question regarding the chapter on multivariate methods.

More specifically:

Say that we have a data matrix as follows: $$\mathbf{X} = \begin{bmatrix} X_1^1 & X_2^1 \quad \cdots \quad X_d^1 \\ X_1^2 & X_2^2 \quad \cdots \quad X_d^2 \\ \vdots \\ X_1^N & X_2^N \quad \cdots \quad X_d^N \end{bmatrix}$$ where each column represents a feature (or attribute) and each row represents a data sample. Given such a multivariate sample, estimates for these parameters can be calculated as follows: The maximum likelihood estimator for the mean is the sample mean, $$\mathbf{m}$$. Its $$i$$th dimension is the average of the $$i$$th column of $$\mathbf{X}$$: \begin{align} & \mathbf{m} = \frac{\sum_{t = 1}^N \mathbf{x}^t}{N} \\ \text{where}\quad & m_i = \frac{\sum_{t = 1}^N x_i^t}{N} \ (i = 1, \dots, d) \end{align} The estimator of the covariance matrix $$\mathbf{\Sigma}$$ is $$\mathbf{S}$$, the sample covariance matrix, with entries: \begin{align} & s_i^2 = \frac{\sum_{t = 1}^N (x_i^t - m_i)^2}{N} \\ & s_{i, j} = \frac{\sum_{t = 1}^N (x_i^t - m_i)(x_j^t - m_j)}{N} \end{align} These are biased estimates, but if in an application the estimates vary significantly depending on whether we divide by $$N$$ or $$N - 1$$, we are in serious trouble anyway.

I put the part that I don't understand in bold font. I'm just curious why it would be a problem if these estimates varied greatly depending on whether we divide by $$N$$ or $$N - 1$$. My intuition tells me that typically the estimates wouldn't be that different, but I'm not well versed in statistics so I'm not too sure.

Any feedback is appreciated. Thanks.

## 1 Answer

The comment seems to be a way of saying that we like large sample sizes in machine learning.

The numerator is the numerator, whether you divide by $$N$$ or $$N-1$$, so all that matters to our discussion is the denominator.

The only way for the two fractions to differ immensely is if $$N$$ and $$N-1$$ are very different, say if $$\frac{N}{N-1}$$ is much greater than $$1$$ (whatever "much greater" means to us).

That only happens when $$N$$ is small. If we have $$N=1000000$$, $$\frac{N}{N-1}$$ is about 1.

$$\underset{N\rightarrow \infty}{\text{lim}} \dfrac{N}{N-1} = 1$$

So the comment seems to be a way of saying that we like large sample sizes in machine learning.

• Upvote but I find (N - 1)/N more natural. The fraction becomes 1 - 1/N and the magnitude of 1/N gives how "much greater" the difference between the two is. – Rui Barradas Sep 13 '20 at 18:28