I'm working on a project to predict students at risk of dropping out, my dataset currently consists only of the students academic records (their grades and what courses they took). I'm hoping to get more data, like their demographics, social interactions, etc. But until then, I'm trying to work with what I have. The nature of the dataset is highly imbalanced (it's like 4 to 100).
I've tried to do some preprocessing, cleaning up data, etc, and did some feature extraction to create features such as the students GPA and how many courses they failed (mandatory and elective), have they ever skipped any term, etc.
I've also tried to use several models such as naive bayes, decision tree, MLP, and random forest. But my models could only achieve a maximum of around 0.6-0.7 of F2-measure and around 0.7-0.8 recall. I'd really like to improve it. I've tried to tinker with quite a lot of things but nothing really worked. I have done some hyperparameter tuning with GridSearch and also tried to do undersampling/oversampling like with SMOTE.
So here's my question:
Given a dataset with a set of features, is it possible that no matter what you do, a model won't be able to predict with high performance? My hypothesis is that in my case, because students don't necessarily drop out only because of bad grades (for example, it may be because they have financial troubles or perhaps not adjusting well to the social life at campus), my model will not be able to predict said students as the dataset does not contain these information.
Do you have any recommendation on what I should try or might be missing? I'm still new to the field so any feedback would be highly appreciated.
Thanks! If there's more context that you need, feel free to ask, I'd be glad to tell more.