I have a binary classification problem where I am using linear SVM. I am interested in diagnosing underfitting/overfitting by visualizing learning curves. My models have different feature sizes; for each feature size, I have optimized SVM C value and found the misclassification error using CV.
My questions are:
Generally, why are most classification learning curves plotted against number of samples? I would think that plotting a learning curve could be plotted against any parameter that is being optimized to see how training and test errors are behaving.
Are there any theoretical predictions on how such a learning curve would look? If this was a regression problem and I was looking at unadjusted R-squared values, one would expect training error to keep coming down with increasing number of variables added and then perhaps become more or less constant. What about classification case? Should I expect the training error to reduce with increasing number of features till a point and then increase as noisy features begin to get added or would it show a similar trend like unadjusted R-squared value? Most classification learning curves that I have seen have shown training/test error as a function of sample size...could that mean that this is rather problem specific and cannot be theoretically predicted?
In general, when plotting learning curves, is it necessary that all points come from models with every other hyperparameter kept constant, except the one being studied? My hunch is that that should be so, otherwise the learning curve would be a n-dimensional plot depending on the number of parameters being manipulated but would love to have some confirmation.