Classification learning curve: function of number of features

Question

I have a binary classification problem where I am using linear SVM. I am interested in diagnosing underfitting/overfitting by visualizing learning curves. My models have different feature sizes; for each feature size, I have optimized SVM C value and found the misclassification error using CV.

My questions are:

1. Generally, why are most classification learning curves plotted against number of samples? I would think that plotting a learning curve could be plotted against any parameter that is being optimized to see how training and test errors are behaving.

2. Are there any theoretical predictions on how such a learning curve would look? If this was a regression problem and I was looking at unadjusted R-squared values, one would expect training error to keep coming down with increasing number of variables added and then perhaps become more or less constant. What about classification case? Should I expect the training error to reduce with increasing number of features till a point and then increase as noisy features begin to get added or would it show a similar trend like unadjusted R-squared value? Most classification learning curves that I have seen have shown training/test error as a function of sample size...could that mean that this is rather problem specific and cannot be theoretically predicted?

3. In general, when plotting learning curves, is it necessary that all points come from models with every other hyperparameter kept constant, except the one being studied? My hunch is that that should be so, otherwise the learning curve would be a n-dimensional plot depending on the number of parameters being manipulated but would love to have some confirmation.

Answer 1

why are most classification learning curves plotted against number of samples?

not only most: all. The reason is simply that the definition of learning curve is that it is the predictive performance of a model as function of training sample size.

would think that plotting a learning curve could be plotted against any parameter that is being optimized to see how training and test errors are behaving

Of course you can plot performance over other parameters - it's just not called learning curve, it's performance-over-$name of parameter.

As for 2., yes, one can often formulate how these functions look. It certainly depends on the type of model, and possibly also on the application task. But for sufficiently similar tasks and a given model I'd say one can formulate a general idea how these "performance landscapes" look.
The maybe most important such formulation is actually what you already describe (and which holds also for classification): if you look at performance as function of model complexity at a given sample size, one expects systematic error (underfitting) to drop and variance error (instability) to increase, leading to a U- or trough-shaped curve for total error.

As for 3. yes, they can have multiple dimensions, depending on how many parameters you consider.
There is a general rule in Design of Experiments that is important to keep in mind here: unless you are certain that the parameters behave independently, you should consider the n-dimensional landscape because by looking at n one-dimensional curves you may totally miss the optimum you're after.