Let's say I have a supervised machine learning problem, where I am trying to map $D$ dimensional feature vectors $(x_1,x_2,...,x_D)$ to $1$ of $C$ possible class labels. I have $N$ training examples to "learn" from, and $K$ testing examples to classify. My goal is to maximize classification accuracy on the $K$ testing examples. My model, $M$, has $P$ free parameters.

My questions are:

  1. Are there any rules of thumb for how I can choose $P$ so that I neither underfit nor overfit to the data?
  2. Will the answer depend on $M$ (i.e. whether I am using Neural Nets, Random Forests, etc)?
  3. Also, if I adjust the hyperparameter settings of $M$, $Q$ times, does this affect the answer?
  • $\begingroup$ Have you looked for the answer to this in other questions. I'm pretty sure I have seen several since I started here. I know the folks in charge like to have clear reasons why you think this is not a repeat of one of them. $\endgroup$ – EngrStudent Mar 22 '15 at 17:29
  • $\begingroup$ @EngrStudent A unique aspect of the question above may be that I am asking if the "best" $P$ is dependent on how many times you change hyperparameters. I wasn't able to find a question that tackled that specific question, although if there was one I would love to read the answer! Edit: Also, are all parameters equal? I.e. does $P$ depend on the model, I haven't seen a question tackling that in specific (though I could be wrong). I have seen questions asking very generally about overfitting, but I think there are specific aspects to my question that I haven't seen here before. $\endgroup$ – kyphos Mar 22 '15 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.