Let's say I have a supervised machine learning problem, where I am trying to map $D$ dimensional feature vectors $(x_1,x_2,...,x_D)$ to $1$ of $C$ possible class labels. I have $N$ training examples to "learn" from, and $K$ testing examples to classify. My goal is to maximize classification accuracy on the $K$ testing examples. My model, $M$, has $P$ free parameters.
My questions are:
- Are there any rules of thumb for how I can choose $P$ so that I neither underfit nor overfit to the data?
- Will the answer depend on $M$ (i.e. whether I am using Neural Nets, Random Forests, etc)?
- Also, if I adjust the hyperparameter settings of $M$, $Q$ times, does this affect the answer?