Maximum accuracy of a set of features, can a small number of relevant features may impact negatively on model performance?

Question

I'm working on a project to predict students at risk of dropping out, my dataset currently consists only of the students academic records (their grades and what courses they took). I'm hoping to get more data, like their demographics, social interactions, etc. But until then, I'm trying to work with what I have. The nature of the dataset is highly imbalanced (it's like 4 to 100).

I've tried to do some preprocessing, cleaning up data, etc, and did some feature extraction to create features such as the students GPA and how many courses they failed (mandatory and elective), have they ever skipped any term, etc.

I've also tried to use several models such as naive bayes, decision tree, MLP, and random forest. But my models could only achieve a maximum of around 0.6-0.7 of F2-measure and around 0.7-0.8 recall. I'd really like to improve it. I've tried to tinker with quite a lot of things but nothing really worked. I have done some hyperparameter tuning with GridSearch and also tried to do undersampling/oversampling like with SMOTE.

So here's my question:

1. Given a dataset with a set of features, is it possible that no matter what you do, a model won't be able to predict with high performance? My hypothesis is that in my case, because students don't necessarily drop out only because of bad grades (for example, it may be because they have financial troubles or perhaps not adjusting well to the social life at campus), my model will not be able to predict said students as the dataset does not contain these information.

2. Do you have any recommendation on what I should try or might be missing? I'm still new to the field so any feedback would be highly appreciated.

Thanks! If there's more context that you need, feel free to ask, I'd be glad to tell more.

Answer 1

There is no substitute for good variables that can help classify who will succeed or likely drop out in their first year.

My observations is that student who are better prepared for their first year college courses do better. This means taking recommended courses even if this results in a drop in their overall GPA (which is likely a good variable reflecting a work ethic).

Also, higher entrance exam scores may reflect abilities to be a faster learner, which is much more important in college.

Financial, family, and friends providing support is likely material as well. The success (or lack thereof) of related family members is a possible guide in this regard.