I have been wondering about the differences between model paramters and model hyperparameters, as well as what their categorization means for a learning problem.
Is the distinction between model parameters and hyperparameters only about reducing the complexity of the problem or are there implications for the 'model quality' to be considered?
Let's consider an example from (1, Eq. 2.43), a linear expansion of basis functions
$$f_\theta(x) = \sum_{m=1}^{M} \theta_m h_m(x), $$
where $h_m\:\forall m=1,...,M$ is some function of $x$, $\theta=[\theta_1,...,\theta_M]^\intercal$ is a vector of the parameters of the model and $M$ is the number of basis functions.
In my understanding, we would consider $\theta$ to contain the parameters of the model, while $M$ is a hyperparameter.
To quote the Wikipedia article of hyperparameter that I linked above:
In machine learning, a hyperparameter is a parameter whose value is set before the learning process begins. By contrast, the values of other parameters are derived via training.
It makes sense treating the values in $\theta$ as parameters that should be learned from the data, similar to the coefficients in a linear model - it is surely far more reliable and specifying them manually would be quite annoying, although it's not impossible.
But what about $M$? It surely is easier to pick some integer value for it. But would it really be terrible to determine $M$ based on the data, too? Probably not, as that is what we would do if we search a grid of possible values for the best performing one.
Let's consider the above basis expansion model and choose Gaussian kernel functions to serve as basis functions. As the Kernel itself has parameters too this makes the problem a bit more complicated.
$$f_\theta(x) = \sum_{m=1}^{M}\theta_m K_{\lambda_m}(\mu_m,x), $$ with $\mu_1,...,\mu_m$ being the location of the kernels and $\lambda_1,...,\lambda_m$ their scales/bandwidths.
Friedman et al. write in (1, p. 36)
"In general we would like the data to dictate them as well. Including these as parameters changes the regression problem from a straightforward linear problem to a combinatorially hard nonlinear problem. In practice, shortcuts such as greedy algorithms or two stage processes are used."
I see that if we were to treat $M$, $\theta_1 ... \theta_M$, $\mu_1 ... \mu_M$ and $\lambda_1 ... \lambda_M$, as model parameters, this problem would be really complex, especially with the choice of $M$ determining the number of the other parameters.
But what if we simplify the model definition - let us for example use the same scale/bandwidth for each kernel, that gives us only one parameter $\lambda$. Furthermore, let's say we specify $\mu_1 ... \mu_M$ based on some heuristic, which would mean we treat it as a hyperparameter.
This gives us
$$f_{\theta,\lambda}(x) = \sum_{m=1}^{M}\theta_m K_\lambda(\mu_m,x), $$
Besides increased complexity of the involved optimization problem, does treating $\lambda$ as a parameter rather than a hyperparameter have other negative or undesired effects?
(1) Friedman, Jerome, Trevor Hastie, and Robert Tibshirani. The elements of statistical learning. Vol. 1. No. 10. New York: Springer series in statistics, 2001.