Is there a way of estimating in advance the number of observations one needs to achieve a maximum prediction error given a machine learning problem?
There are a huge number of questions on CV and elsewhere relating to choosing sample size for estimation or hypothesis testing. I haven't seen, though, much discussion on how sample size relates to the prediction accuracy of a given model (SVM, logistic regression, etc).
Note that I'm not referring to splitting a data set into folds, but rather determining the size of the total data set itself.
My hunch is that this is a very contextual problem (for starters, if it were simple the answer would be much easier to find!). I imagine one would need very specific assumptions about the distributions of every feature and the ground truth, and then for a specific model with fixed hyperparameters you could derive the expected error. But, since one has so many models and hyperparameters to consider, is this a practical exercise? The alternative of collecting as much data as you can afford is much simpler, but it would be nice to know in advance if you can even reach the minimum for your application.
This relates to a previous question that is unanswered as of now.