I'm currently working with radiomics in MRI for cancer diagnosis, and I need to solve a dilemma about the best approach to analyze my data, here's the problem:

I have 80 patients with surgical resection of a tumor AND pre-surgical MRI images of the same tumor. Using python we are extracting a large number of image features: texture features, morphology features, intensity features, etc. In this particular case, we'll use 58 features, and we have two interest outputs: Malignancy (1/0) and percentage of fibrosis (measured on a continuous scale).

A priori, I would solve this problem with ad-hoc classical statistic tools for the sample size: Classic regression and variable selection methods (I was planning to use dredge function of the R package MuMIn), setting the maximum number of possible variables in a model to 8-10 (for avoiding overfitting) and using corrected Akaike's Information Criterion for ranking models and selecting the best.

Usually this would have been pretty straightforward, but for some reason I wasn't part of the project's initial planning and a couple of engineers that work with our team wrote the statistical analysis proposal: They proposed principal component analysis and Fisher's discriminant analysis for dimensionality reduction, and after that they wanted to use the best components within a Support vector machine algorithm for developing a predictive model.

As I'm not a beginner with biostatistics nor machine learning, I'm having big concerns about the proposed approach (PCA, LDA, SVM) in this small dataset due to important overfitting issues with these techniques in small sample sizes. My problems: I haven't so much linear algebra background as an engineer for giving them really heavy arguments for avoiding the proposed approach (I'm an MD with an MSc in Health sciences research with focus on advanced statistics), and there is a lot of literature circulating in medical journals with approaches like this in small datasets without concerning on "spectacular" overtfitted results that have not applicability in real life.

After some studying and research work, I would like to have more expert opinions about applying PCA and LDA on small datasets (80 individuals and 58 variables in this particular case), for taking a more informed decision (I've found a big heterogeneity of opinions on similar issues).

What do you think guys? Thanks for your answers!!


1 Answer 1


You need adequate dimension reduction to avoid overfitting. Overfitting comes from maintaining too many fully weighted features in a model, regardless of the particular modeling approach used. I'll focus here on regression techniques. LDA and SVM might or might not work better than regression in your case, but that's not because of differences with respect to overfitting.

PCA sits well within the classic set of tools for dimension reduction in ordinary and logistic regression. To avoid overfitting with PCA-based regression techniques you just don't retain all the principal components. In your case you might retain on the order of 3 to 5 components.* It has the advantage that you don't throw away all of the information from any your original 58 features, you just re-weight them according to their contributions to the retained principal components.

When predictors are highly correlated, as seems likely in your study, PCA-based approaches protect you from getting results that are highly dependent on the sample at hand. The correlated predictors tend to be represented in the same principal components. So if one of a pair of correlated predictors happens to be most important in your data sample, there's a good chance that the corresponding principal component will still do well in a sample where the other of the pair is dominant. Selection of a subset of the original features, as you propose, loses that advantage.

Standard principal-components regression does make an all-or-none choice of retained components. The method of ridge regression can be thought of as retaining all the principal components but weighting them differentially. That relative weighting penalizes the magnitudes of the regression coefficients of the original features to avoid overfitting. For prediction in cases like yours, with a moderate ratio of cases to features, that can be a very useful choice, whether for logistic or ordinary regression. Cross-validation is typically used to choose the level of penalization in a way that minimized overfitting.

So PCA is perfectly acceptable as a way to get the dimension reduction you need, however you happen to apply it. An Introduction to Statistical Learning is a reasonably accessible reference for further study on these and many other topics.

*To avoid overfitting in typical biomedical studies you should retain about 1 fully weighted predictor per 15 cases of the minority class in logistic regression, and 1 per 15 total cases in ordinary regression.

  • $\begingroup$ Thanks a lot, great explanation. One doubt I have left is about stability of PCA in small datasets, I've been reading about a minimum needed ratio of observations to variables of at least 5 : 1, which my dataset is far from accomplishing (I've another project with 84 observations and 198360 variables, so a better understanding of this subject is pretty important to me) $\endgroup$
    – crlagos0
    Commented Jun 12, 2020 at 0:07
  • $\begingroup$ @crlagos0 with principal component regression the “number of variables” is effectively the number of components that are kept, not the original number of features submitted to PCA. So even if you start with 58 features. If you only retain the first 5 principal components from PCA that’s like choosing 5 features—but with the advantages noted in the answer. For many more features than observations, you should look into Lasso or elastic net; start with the text I linked in the answer. $\endgroup$
    – EdM
    Commented Jun 12, 2020 at 1:46
  • $\begingroup$ I get your point perfectly, but I'm talking about another thing really with "stability of PCA" (it's not about the use of the principal components as variables for regression, sorry I'm not expressing well myself). There are some concerns in different scientific areas (like ecology) about performing PCA with a ratio < 5:1 of observations to variables. Here you can read an example of a discussion about "stability of PCA results " regarding to sample size. tandfonline.com/doi/abs/10.1577/T08-091.1 $\endgroup$
    – crlagos0
    Commented Jun 12, 2020 at 3:53
  • $\begingroup$ @crlagos0 the use of PCA in that paper is different from how it's used in regression. Principal components (PCs) in morphometrics are interpreted functionally in terms of size (PC1) and shape (PC2, PC3). Stability of loadings of features onto PCs is needed for such interpretation. In regression, you use cross-validation to select a number of PCs that fits but doesn't overfit the relationships of features to outcome. No interpretations are assigned to particular PCs. Ridge regression smooths out the PCs rather than forcing a cutoff, providing further protection from overfitting. $\endgroup$
    – EdM
    Commented Jun 12, 2020 at 11:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.