I have a very basic machine learning question.
My likelihood function includes a parameter $\alpha$ which I set to a fixed value and do not learn from the model, which makes it a "hyperparameter". And I intend to select the optimal value of this hyperparameter using a cross-validation test. Apparently, the value of my likelihood function depends on the value of $\alpha$. So, should I necessarily use the fixed value of $\alpha$ I used in training while computing the test fold log-likelihood?
First I thought that would be the case, but then I remembered that in penalized regression models, the penalty parameter is treated as the hyperparameter, and while computing the test error, we don't include the penalty term, but just the standard mean squared error. This makes sense but also caused me a confusion because such a computation of log-likelihood without including the value of the hyperparameter is not possible in my case.
Any suggestions are welcome!