Say I have a simple machine learning problem like a classification. With some benchmarks in vision or audio recognition, I, as a human, are a very good classifier. I therefore have an intuition on how good a classifier can get.
But with lots of data one point is that I do not know how good the classifier I train is possible to get. This is data where I personally am not a very good classifier (say, classify the mood of a person from EEG data). It is not really possible to get an intuition on how hard my problem is.
Now, if I am presented with a machine learning problem, I would like to find out how good I can get. Are there any principled approaches to this? How would you do this?
Visualize data? Start with simple models? Start with very complex models and see if I can overfit? What are you looking for if you want to answer this question? When do you stop trying?