This is an interesting and precarious situation you are in, which should be treated with caution.
First, let's establish some terms: let's not use "overfit" and "non-overfit" to describe your approaches, as both (or neither) could potentially be overfitted. The difference between you two approaches is whether or not you tuned the hyperparameters through CV, right? So let's call them "not tuned/optimized" and "tuned/optimized".
The reason I say your situation is interesting, is because it appears that the "incorrect" or "improper" approach gave a better result than the "proper" approach. The situation is precarious because in order to come to the conclusion I just mentioned, you're using the test-set results as a point of comparison. This is problematic because the test set is supposed to be used as a realistic error estimation of your final model, which you should have selected during cross validation. The final test set is not for multiple comparisons of models for purposes of model selection. Let me repeat that the test set is not for model selection. This is because you could keep retrying 1,000 different times until you finally got a great result on the final test set and think "wow this is a great model I found" when in reality, you did many comparisons. I hope this intuitively makes sense. Any model/hyperparameter/variable selection should be done through cross validation. The test set is just for getting a realistic error estimation of the model selected through CV.
So, our discussion is a bit flawed to begin with, as we are using the test set as a basis for comparison of two models/approaches (which we really shouldn't be doing). But nevertheless, your question stands! Why is it that my un-optimized model gave better results than the one that was optimized through a technique that's supposed to deliver superior results?
There are several ways of looking at this, and I will offer all the perspective I can.
1. False Premise : You say that the optimized model gives worse predictions, but is this really the case? I know it appears that way because having a model (like the optimized one) which gives almost exclusively 1 as the prediction seems less useful (and it's F1-score for the 0 class is worse), but consider this: the accuracy of the unoptimized model on the test set was 77.5%, whereas the optimized gave accuracy of 77.9% . You will say "Sure, but accuracy isn't a good performance metric for this because my dataset is imbalanced, we should look at F1 score." Yes! You're correct, but your RF model isn't trying to optimize the F1 score - your model is optimizing accuracy . Even though the dataset is imbalanced, getting a 1 correct is just as valuable to the model as getting a 0 correct (because they are weighted equally). And because there are way less 0's than 1's, why waste resources trying to get the 0's correct (sacrificing precision on 1's) when you can focus heavily on getting all the 1's right because that's how you maximize overall accuracy (which is what the model is trying to do, and what it's done). The optimized model does not perform worse than the unoptimized one, it's just that for your situation, you don't like how it arrives at that accuracy level (by basically ignoring the 0 class). We'll go into how to fix this a bit lower down, but you must understand that the model is doing the task you gave it - to get the biggest number of predictions correct, regardless of whether they are 1's or 0's.
2. Luck : Cross validation hyperparameter tuning works because it essentially selects those hyperparameters which on average produce the greatest out of sample accuracy. It's on average because you're averaging out of sample fold performance across all k folds. This, of course, means that sometimes the performance may be higher or lower (you can check the performance on individual folds to see how much of a spread/variance there is). What this also means is that if you randomly/arbitrarily select hyperparameters and run the model on the test set, you might have just gotten lucky - the arbitrary model did pretty well on the test set, but on average, it doesn't do so well. Same thing with the optimized model - it is optimized because on average, this is the best combination of parameters - maybe it just got a bit unlucky on your test set, and happened to perform below average. My point here is that your optimized model is still your best bet because it was chosen based on average performance, whereas you are looking at your unoptimized model as a candidate based solely on one out of sample prediction set (the test set).
I hope you can see why comparing the two approaches (random/arbitrary hyperparameter selection vs. CV) based on test set performance isn't correct. Think of it this way - if your arbitrary choice of hyperparameters was the best combination on average.. that would have been the combination chosen during CV. But since it wasn't, it's not the combination that does best on average, and so it's likely that the model just got "lucky" on your test set.
It's also possible, though, that hyperparameter tuning just doesn't make a huge difference in accuracy. However, you correctly observed a large difference in the distribution of predictions between the optimized and unoptimized model. You don't like the distribution of predictions of the optimized model because it basically just predicts a 1 every single time. Yet at the same time, that appears to be the optimal approach! Right? That's how the model gets the largest number of data points correct.
To get around this conundrum, you need to employ techniques to deal with your imbalanced dataset.
Essentially, right now, your model is saying "Why bother getting good with predicting 0's, they are only 25% of the dataset and are worth the same as a 1 to me, I can do better by just catching all the 1's " .
You have basically two options, which I have copy-pasted from my answer here.
Your options to fix the problem
Oversampling/undersampling - creating a new dataset which doesn't have the imbalance problem, therefore forcing the model to contend with the rare category of data.
Weighting - not changing the data, but changing the scoring so that the model is penalized differently fir getting different categories wrong. (you can do this easily in RandomForest by adjusting a hyperparameter that's called something like "class weight", "sample weight", "weight" or something similar. Read into your RF package to find this hyperparameter).
I will explain both, but I would recommend trying weighting first, as it's usually a much easier fix than changing your sampling.
Category 1: Sampling (oversampling/undersampling)
Right off the bat, I (like many) would advise against the undersampling approach, where you would keep all of the rare category examples, but randomly sample some fraction of the "common" category samples to include. You already have a limited amount data and you want to make use of all of it. Undersampling gives your model just gives your model less information to work with. So if you're doing some sampling approach, I would go with oversampling. Undersampling extreme example: You have 100 "common" cases and 10 "rare" cases, so you sample randomly 10 of the common cases and now you have a new, balanced dataset with 10 common cases and 10 rare cases.
Simply speaking, in over-sampling, you want to create a synthetic dataset where you have, in some way, repeated the rare data so that the model encounters more often and therefore has to address it in order to find the optimal solution (i.e. optimal solution can't be found by ignoring it anymore). Oversampling extreme example: You still have 100 "common" cases and 10 "rare" cases, so you randomly sample (with replacement) from the 10 "rare" cases until you have 100 of them (of course, 90 of the 100 are randomly-selected repeats of the original 10). But now you have 100 "common" and 100 "rare" cases.
There's a technique called SMOTE, which you can read about here: https://machinelearningmastery.com/smote-oversampling-for-imbalanced-classification/ . I personally have not used it, but it's very well-known and considered to be quite effective (it's a specific kind of oversampling technique). If simple oversampling doesn't solve the issue, consider SMOTE.
Category 2: Weighting
This is pretty straightforward too. Besides making it more common, you can try to force the model to pay attention to the rare cases by assigning a larger weight to them - that is, the model is penalized more than usual when it gets them wrong. Many algorithms allow some kind of incorporation of weights into it (as parameters).
I hope this helps!
