I need to find a situation in which logistic regression does not work well. Furthermore, I would like to know when a random forest might perform better than a logistic regression model.
Consider these data (copied from @Sycorax's answer here: Can Random Forest be used for Feature Selection in Multiple Linear Regression?):
There are two aspects to the data in this figure. First, the relationship is non-linear. That isn't actually a problem for logistic regression properly specified. In some cases, a logistic regression might fair better than a standard decision tree (cf., my answer here: How to use boxplots to find the point where values are more likely to come from different conditions?, although vis-a-vie a random forest is more ambiguous). The bigger problem is that there is complete separation at the decision boundary. There are ways of trying to deal with that (see @Scortchi's answer here: How to deal with perfect separation in logistic regression?), but it adds complexity and requires considerable sophistication to address well. I think a random forest would handle this as a matter of course.
The answers so far emphasize the predictive failure of logistic regression. However there's also issues of poor feature importance/inference. For example, when your classes are highly correlate or highly nonlinear, the coefficients of your logistic regression will not correctly predict the gain/loss from each individual feature. In gung's example, if you were to train a logistic regression on the picture of points shown, it will likely create a linear split somewhere in the middle of the red region (for example a vertical line), implying that an increase of say, $x_1$, will lead to a higher probability of being a red class, which is true for $x_1$ starting to the left of the prediction boundary, and false for $x_1$ starting to the right.
@gung had a good answer. Logistic regression is a linear model, so it may not work well on non-linear cases. But as I mentioned in the comment, it might be some ways to transform data into another space, where logistic regression will be good again, but finding the basis expansion / feature transformation may be not trivial.
Essentially, certain model will work well when the data satisfy the assumption of the model, e.g., if the decision boundary is linear, then logistic regression will work well.
On the other hand, I would strongly recommend you to review bias variance trade off.
In terms of model complexity, logistic regression has high bias and low variance. And random forest is opposite. Which means, in general, logistic regression will preform with "less accurate" but "more stable", and random forest is opposite.
A simple example for a case when logistic regression can’t work properly: https://towardsdatascience.com/when-logistic-regression-simply-doesnt-work-8cd8f2f9d997
Situations where features combined effect on response are not linear.
Imagine you try classifying documents as pet related or not per related.
You have two features -
* Number of words in the document(X1).
* Number of times the word Dog appears in the document(X2).
Intuitively, X2/X1 is a good way determining a document's class.
This is not a linear relation, so where tree like models can use a:
Not Pet related
A logistic regression will have no such option, and will result in model described by
if aX1 + bX2 > Z:
Not Pet related
For some Z.