Do you know the concrete costs of misclassification in both directions? I assume that missing the rare class must be much more costly than falsely adding a record to the rare class. (If not, then your problem is per definition trivial, ignore the rare class!) Only you can know the answer to that question based on your application scenario. Put those misclassification costs, and perhaps also profits due to correct classification, in a loss function and compare performance on this function. You also want to use the label confidences that a random forest classifier provides. You can adjust the classification cutoff according to your misclassification costs. This allows you to maximize expected profits/ minimize expected costs. In order to avoid manual overfitting by reverse engineering your classifier to the test-set and to do it in a principled way, you should set that cutoff based on classification costs alone, thus before training the classifier.