Linear Discriminant Analysis have Small Sample Size problem (SSS) is it n<<d

It is said that LDA has a Small Sample Size problem (SSS)

This problem arises whenever the number of samples is smaller than the dimensionality of the samples.

(Source: Chen, L.F., Liao, H.Y.M., Ko, M.T., Lin, J.C. and Yu, G.J., 2000. A new LDA-based face recognition system which can solve the small sample size problem. Pattern recognition, 33(10), pp.1713-1726.)

Is this correct that the number of samples and dimensional is refer to by the diagram below?
Is that LDA have problem if sample size (n) << dimensions (d)

Source: https://sebastianraschka.com

• Could you describe more context for your question. You placed the quote "This problem arises whenever the number of samples is smaller than the dimensionality of the samples.", but it is unclear what 'this problem' refers to. Maybe it is somewhere on the source site (it is not on the landing page, so people that wish to answer your question need to search for it) but you could better explain it directly in the question. – Sextus Empiricus May 26 at 8:39
• @SextusEmpiricus Thanks for your comment. I had edited it. The references from Chen, L.F., Liao, H.Y.M., Ko, M.T., Lin, J.C. and Yu, G.J., 2000. A new LDA-based face recognition system which can solve the small sample size problem. Pattern recognition, 33(10), pp.1713-1726. – aan May 26 at 13:37

Is this correct that the number of samples and dimensional is refer to by the diagram below?

In the diagram below it seems you have 150 samples (from the leftmost ID columns) and each sample has dimensionality 4. So you will have no problems computing LDA here as 150 >> 4.

Is that LDA have problem if sample size (n) << dimensions (d)

Yes, correct, this would be an issue for LDA (and many others for that matter).

• Thanks. you are very helpful. Do you mind answer the calculation of covariance matrix in LDA? stats.stackexchange.com/questions/466952/… – aan May 26 at 13:38
• @Renthal, you are answering the questions with 'yes', but could you explain why? – Sextus Empiricus May 26 at 20:22
• @SextusEmpiricus I am not sure my answer is correct or not, But i think is because it is called small sample size problem which a lot of information in published paper. – aan May 28 at 21:02

Modelling based on many variables is effectively multiple parallel hypothesis testing, which leads to overfitting as the false positive rate is going to be high, especially for classification problems where the number of states is very limited. Overfitting occurs where a model is fitted that includes features unrelated to the variation of interest (i.e. 'noise' in a generic sense of information unrelated to the target analyte). This problem is linked to the assumed independence of noise between variables, with each variable independently given a chance to discriminate between the small number of classes.

Modelling based on many more variables than observations increase the risk of a type II error in which a 'statistically significant' result is in fact based on random chance. For example, when working at a significance level of 5%, a dataset with 1000 randomly distributed variables and an infinite number of samples, would exhibit about 50 'statistically significant’ single-variable correlations due to chance alone, which would then appear as coefficients in the discriminant function. In the real world such 'long-run' experiments have to be compromised by available resources and the situation is much worse. In a study powered to fairly common statistical practices such as powering using an alpha of 5% and beta 80%, the false positive rate will be much higher, around 25% (Ioannidis), or in other words 250 variables will appear significant (the noise in each variable is assumed independent). PCA and other data reduction methods prior to LDA help by first combining redundant information across the variables through the process of data reduction you can average out independent stochastic noise while simultaneously reducing the number of statistical tests carried out and therefore the risk of a type II error is also reduced.

Note I have only discussed independent noise, I have not touched on bias nor correlated noise as the outcome of such structured issues is highly dependent on the nature of the issue.