One possibility is that the sample covariance matrix is becoming near singular somewhere along the way, and your iterations involve inverting this matrix
The eigenvalues of your sample covariance matrix will follow a Marcenko-Pastur distribution (https://en.wikipedia.org/wiki/Marchenko%E2%80%93Pastur_distribution), and from this distribution you will see that it is not inconceivable that you get high condition numbers (the condition number is the ratio of the largest to the smallest eigenvalue of your covariance matrix)