I'm currently going through the textbook Introduction to Machine Learning 4e (Ethem Alpaydin) to brush up on my ML basics and had a question regarding the chapter on multivariate methods.
More specifically:
Say that we have a data matrix as follows: $$ \mathbf{X} = \begin{bmatrix} X_1^1 & X_2^1 \quad \cdots \quad X_d^1 \\ X_1^2 & X_2^2 \quad \cdots \quad X_d^2 \\ \vdots \\ X_1^N & X_2^N \quad \cdots \quad X_d^N \end{bmatrix} $$ where each column represents a feature (or attribute) and each row represents a data sample. Given such a multivariate sample, estimates for these parameters can be calculated as follows: The maximum likelihood estimator for the mean is the sample mean, $\mathbf{m}$. Its $i$th dimension is the average of the $i$th column of $\mathbf{X}$: $$ \begin{align} & \mathbf{m} = \frac{\sum_{t = 1}^N \mathbf{x}^t}{N} \\ \text{where}\quad & m_i = \frac{\sum_{t = 1}^N x_i^t}{N} \ (i = 1, \dots, d) \end{align} $$ The estimator of the covariance matrix $\mathbf{\Sigma}$ is $\mathbf{S}$, the sample covariance matrix, with entries: $$ \begin{align} & s_i^2 = \frac{\sum_{t = 1}^N (x_i^t - m_i)^2}{N} \\ & s_{i, j} = \frac{\sum_{t = 1}^N (x_i^t - m_i)(x_j^t - m_j)}{N} \end{align} $$ These are biased estimates, but if in an application the estimates vary significantly depending on whether we divide by $N$ or $N - 1$, we are in serious trouble anyway.
I put the part that I don't understand in bold font. I'm just curious why it would be a problem if these estimates varied greatly depending on whether we divide by $N$ or $N - 1$. My intuition tells me that typically the estimates wouldn't be that different, but I'm not well versed in statistics so I'm not too sure.
Any feedback is appreciated. Thanks.