Radiomics: Machine learning vs Classic Multivariate Statistics for small datasets and a lot of variables... which is better?

Question

I'm currently working with radiomics in MRI for cancer diagnosis, and I need to solve a dilemma about the best approach to analyze my data, here's the problem:

I have 80 patients with surgical resection of a tumor AND pre-surgical MRI images of the same tumor. Using python we are extracting a large number of image features: texture features, morphology features, intensity features, etc. In this particular case, we'll use 58 features, and we have two interest outputs: Malignancy (1/0) and percentage of fibrosis (measured on a continuous scale).

A priori, I would solve this problem with ad-hoc classical statistic tools for the sample size: Classic regression and variable selection methods (I was planning to use dredge function of the R package MuMIn), setting the maximum number of possible variables in a model to 8-10 (for avoiding overfitting) and using corrected Akaike's Information Criterion for ranking models and selecting the best.

Usually this would have been pretty straightforward, but for some reason I wasn't part of the project's initial planning and a couple of engineers that work with our team wrote the statistical analysis proposal: They proposed principal component analysis and Fisher's discriminant analysis for dimensionality reduction, and after that they wanted to use the best components within a Support vector machine algorithm for developing a predictive model.

As I'm not a beginner with biostatistics nor machine learning, I'm having big concerns about the proposed approach (PCA, LDA, SVM) in this small dataset due to important overfitting issues with these techniques in small sample sizes. My problems: I haven't so much linear algebra background as an engineer for giving them really heavy arguments for avoiding the proposed approach (I'm an MD with an MSc in Health sciences research with focus on advanced statistics), and there is a lot of literature circulating in medical journals with approaches like this in small datasets without concerning on "spectacular" overtfitted results that have not applicability in real life.

After some studying and research work, I would like to have more expert opinions about applying PCA and LDA on small datasets (80 individuals and 58 variables in this particular case), for taking a more informed decision (I've found a big heterogeneity of opinions on similar issues).

What do you think guys? Thanks for your answers!!

Answer 1

You need adequate dimension reduction to avoid overfitting. Overfitting comes from maintaining too many fully weighted features in a model, regardless of the particular modeling approach used. I'll focus here on regression techniques. LDA and SVM might or might not work better than regression in your case, but that's not because of differences with respect to overfitting.

PCA sits well within the classic set of tools for dimension reduction in ordinary and logistic regression. To avoid overfitting with PCA-based regression techniques you just don't retain all the principal components. In your case you might retain on the order of 3 to 5 components.* It has the advantage that you don't throw away all of the information from any your original 58 features, you just re-weight them according to their contributions to the retained principal components.

When predictors are highly correlated, as seems likely in your study, PCA-based approaches protect you from getting results that are highly dependent on the sample at hand. The correlated predictors tend to be represented in the same principal components. So if one of a pair of correlated predictors happens to be most important in your data sample, there's a good chance that the corresponding principal component will still do well in a sample where the other of the pair is dominant. Selection of a subset of the original features, as you propose, loses that advantage.

Standard principal-components regression does make an all-or-none choice of retained components. The method of ridge regression can be thought of as retaining all the principal components but weighting them differentially. That relative weighting penalizes the magnitudes of the regression coefficients of the original features to avoid overfitting. For prediction in cases like yours, with a moderate ratio of cases to features, that can be a very useful choice, whether for logistic or ordinary regression. Cross-validation is typically used to choose the level of penalization in a way that minimized overfitting.

So PCA is perfectly acceptable as a way to get the dimension reduction you need, however you happen to apply it. An Introduction to Statistical Learning is a reasonably accessible reference for further study on these and many other topics.

---

*To avoid overfitting in typical biomedical studies you should retain about 1 fully weighted predictor per 15 cases of the minority class in logistic regression, and 1 per 15 total cases in ordinary regression.