maybe I should rather recompute PCA at each iteration because it's considered to be part of the model
YES! You definitively need to do this (at least unless you have a vast amount of experience with this type of data, and have repeatedly shown in the past that it doesn't matter - which would typically occur only in n >> p situations where the PCA isn't really needed)
PCA is likely to create a very strong dependency between cases/rows on moderate to small sample sizes (compared to dimensionality). And that is likely the case: if it weren't you could usually do the LDA without PCA first.
I usually have wide (n < p) data, and small sample size situations (though something close to repeated measurements), and I've seen the error rate underestimated by about an order of magnitude when the cross validation was not including the PCA.
There are various heuristics to optimize model complexity here. The key is IMHO to get some guesstimate of the uncertainty of the measured performance and account for this in the decision.
one fairly widely used one is the one-standard-deviation rule as discussed e.g. in the Elements of Statistical Learning.
In your case, I'd say the absolute number of cases is likely the bottleneck, which I guess to be roughly 45 from the steps in the data.
A back of the envelope calculation for the 5 PC model gives a 95 % confidence interval of very roughly 85 - 100 % based on 43 correct out of 45 tested cases.
Which immediately tells you that you need not consider anything above 5 PCs here.
(You could set up more precise tests, e.g. McNemar's test to check whether the improvement 4 -> 5 PCs is significant; key would be to keep the number of tests low and penalize multiple testing)
Another component to the uncertainty is (in)stability, which you may want to consider more or less independently of the sample size consideration. This is possible from the predictions in repeated k-fold cross valiation, or from model coefficients of the "whole" PCA-LDA model after aligning the models in a way that keeps them equivalent (prediction of PCA-LDA models is invariant to flipping and rotation in LD-score space).
I work with spectroscopic data, where I can often see whether the coefficient vectors (for the full PCA-LDA projection) get noisy, but this is a specialty of the particular kinds of spectroscopic data I work with.
Two papers where we used the 1st and 2nd approach:
- Beleites, C. & Salzer, R.: Assessing and improving the stability of chemometric models in small sample size situations Anal Bioanal Chem, 2008, 390, 1261-1271.
- C. Beleites, K. Geiger, M. Kirsch, S. B. Sobottka, G. Schackert and R. Salzer: Raman spectroscopic grading of astrocytoma tissues: using soft reference information, Anal. Bioanal. Chem., 400 (2011), 2801 - 2816.
manuscript without paywall
(I do have a sneaking suspicion that due to the very close link between PCA and LDA, PCA-LDA may not be that much more stable than LDA in general. But I haven't properly thought about this yet.)