When to admit that your target cannot be predicted? [duplicate]

Question

Assume you have a dataset, say churn. You sit down, do the data cleaning, the data engineering etc. etc. Since you want to predict if a customer churns, you decide on a logistic regression as a benchmark and get around 0.5 accuracy (it's a balanced dataset). You try different other classifiers since there wasn't something in your data cleaning which suggested one particular model as ideal, and all of them is around 0.5 accuracy. You gather some more data, create some more features, do backward/forward feature elimination, PCA everything you can imagine, but your models just simply won't improve significantly.

Have you ever concluded "there is simply no pattern; the target is pure random from person to person" or will you argue that there will always be some pattern somewhere you just haven't found yet?

Answer 1

Relevant question. The answer is that it depends on your prior probabilities $-$ when you have to conclude that the features don't provide any discriminative information. For a two-class problem with equal priors, $P(C)=P(\neg C)$, the minimal accuracy $\rho$ is: $(\frac{1}{2})^2+(\frac{1}{2})^2 = \frac{1}{2}$.

The general formula for the minimal error rate is:

$ \begin{split} \min(\rho)=\sum_{i=1}^{n_C} P(C_i)^2 \end{split} $

with $n_C$ indicating the number of classes to discriminate and $P(C_i)$ the prior probability of class $i$.