Checking Multicollinearity and building a classification model when dependent is a factor and other independent variables are numerical in r

Question

Problem statement

Y - Dependent variable is a factor (with levels A, B, and C) Independent variables are all numerical variables. Important: I have only 70 data points.

End Goal: Building a classification model to classify a future entry as A, B or C.

1. First I need to check multicollinearity among the independent variables and get rid of any variables to tackle this. I know we can have a look at VIF values or covariance among variables to decide this. But the fact that dependent is a factor with levels keeps me throwing me off.
2. After dealing with multicollinearity what are my options to build a good classification model, that would give me a model at the end. So nothing like knn or random forest.
3. I did a principal component analysis and when I plot PC1 vs PC2 I could see two clusters A separate and B&C together. Then when I plot PC1 vs PC5 I could see a vague separation between B and C as well. How can I utilize this information?

What I have done so far 1. I did a principal component analysis. I could see a separation between the levels but I do not know how to utilize this information as I mentioned above. 2. I divided the data set into 80/20 training and validation and did k nearest neighbor classification method.

To be honest I feel like everything is all over the place and would like some guidance on how to properly approach this question using R.

Answer 1

1. Actually, if you are interested in classification, you do not need to get rid of collinear variables. These will influence your coefficients, but not your predictions. You can just leave them in.

2. A random forest would be one of the most logical candidates for a "good", as in, a model that predicts well. Random forests are robust and require little tuning. They also work fine with collinearity, and will accept categorical features without any modifications. No need to think about interactions much either. All in all, probably the most obvious choice without additional information. Now clearly, you rule these out, but it is unclear why. If you are looking to fit some kind of more classical model with coefficients, a first choice would be to go for a logistic regression. Then, you would have to consider adding interactions or nonlinearities to your features if they play a large role.

3. This would mean that A is recognizable from a linear combination of the first and the second feature, and C and B are separable with a linear combination of the first and the fifth feature. Which exact linear combination that is depends on where the clusters are. You could create these linear combinations and base your model only on these two combinations. This would simplify the model, and if the separation is indeed very strong on these axes, might not hurt the performance of the resulting models very much. It is very unlikely to improve the performance of your models, but it's possibly in theory. Again, if you are using a random forest or a logistic regression or any other classification algorithm, those algorithms will learn this information without any help.