Why do we fix parameters and metaparameters differently? To optimize parameters of a model we minimize a loss function.
To optimize metaparameters we look at a loss/metric on a validation set ( or, if we are worried to overfit on the validation set, we do cross validation ).
(Than we check performance on the test set.)
Why this difference ? Couldn't we consider metaparameters and parameters on the same ground and optimize both of them on the training set ? Is there some fundamental mistake with this approach? What would be the consequence of such a choice ?
My only explanation is that we can get an estimate of the generalization error and it is best to use this to select an optimal model with smaller overfitting. One could do the same with parameters but would simply be too expensive (e.g. it is too expensive to consider in a neural net half of the weights as parameters and the other half as hyperparameters).
Opinions?
 A: There are few different reasons for the distinction between parameters and hyperparameters (the more common term for what you refer to as meta parameters).

*

*Some hyperparameters can not be considered "just another parameter" because doing so would lead us to overfit. As an obvious example, discussed here, is how it would be hazardous to allow the polynomial degree of a polynomial regression model to be a parameter because then the model could output a function with degree n+1 (where n is the number of data points in the training set) and achieve no training loss but generalize poorly, due to overfitting.


*Further, some hyperparameters could not be efficiently trained as parameters using traditional optimization techniques. As described here, some hyperparameters could not be optimized efficiently using traditional optimization techniques such as gradient descent. For more details on optimization and why certain functions/relationships are harder to optimize see this article for a lay person's view or here for a more academic view if you have institutional access.


*The hyperparameter may be something we choose to influence the time needed to train, such as the learning rate in Neural Networks.
These are just three of the multiple reasons we differentiate hyperparameters and parameters. A full detailed explanation is outside the bounds of acceptable length for Stack Exchange.
Edit: I will add some more information here in response to the following clarifying question and suggestion for more detail:

For point 1, also parameter optimization leads to overfitting, but we do not bother too much... Any comment on this ?


Thank you. I will accept your answer. Feel free to add a more detailed explanation or links, if you feel like :)

The difference in treatment between parameters and hyperparameters with regard to overfitting is a good question. To start, I do not think it is particularly accurate to say that "we do not bother too much" with regard to overfitting with parameters. In fact, this fear of overfitting and focus on generalizability may very well be the most important distinction between traditional statistical modeling and data fitting compared with machine learning. Machine learning models employ various techniques to counter overfitting in their selection of parameters. To give just a few examples, we can use regularization, where we punish the "size" of the model as in how far the parameter values stray from the origin. Or, we can use "early stopping", where we stop the parameters from reaching the point where they would have the least possible loss on training data in order to avoid overfitting. For a discussion of "early stopping" see this Wikipedia article.
Putting that aside, the different treatment of overfitting in parameters, as opposed to hyperparameters, has to do with the expressiveness of the hypothesis space being examined. To speak not very rigorously, the expressiveness of a class of functions (which may make up the hypothesis space) has to do with how "complicated" they can be, which impacts how "complicated" a set of data points (and thus the relationship between input values) that they can accurately model. To see more rigorous discussions on expressiveness see Vapnik–Chervonenkis dimension,  Shattered Sets, and perhaps most importantly Rademacher complexity.
So, when it comes to parameters, we often attempt to prevent overfitting by instituting some sort of punishment over what we deem to be more expressive functions – but that does not necessarily make the class of functions in the hypothesis space less expressive, as we can still theoretically get expressive outputs that overfit the relationship between the inputs and outputs if we do not have a high enough regularization term.
On the contrary, when we deal with overfitting with hyperparameters, we often take steps to explicitly limit the expressiveness of the functions in the hypothesis space, which may for example decrease the number of points they can shatter, which would directly lower the expressiveness of a binary classifier according to the Vapnik–Chervonenkis dimension.
To give an example, without loss of generality, imagine one model where we treat the degree of a polynomial as a parameter and attempt to prevent overfitting through a regularization term which in some way punishes the degree of the polynomial, perhaps by adding $-d*a$ to the loss, where $d$ is the degree of the polynomial and $a$, is some real number $0<a<1$. In this example, it is still "possible" to have polynomials with very high degree, and further, it is difficult to know what to set as the value for $a$, because we most likely do not know the "true degree" of "true relationship" of data inputs and outputs. Therefore, this approach makes it difficult to appropriately prevent overfitting.
Now compare this to a model where the degree of the polynomial is a hyperparameter, by using some hyperparameter tuning technique such as cross-validation we can attempt to estimate what degree polynomial is needed to represent the underlying relationship between inputs and outputs, and thus restrict the hypothesis space to only those polynomials with this degree (or likely this degree or less), and as a result, directly decrease the expressiveness of the function and also eliminate the "guesswork" of how much regularization is needed for the degree of the polynomial based on the perceived relationship between the variables.
A: The previous post already addressed the question, so I just want to add some further points where they differ:
The parameter is dependent on the hyper-parameter. For example, in the case of support vector machines, you set hyper-parameter on the degree of polynomial kernel, only after that, you can obtain the optimal parameter, the loss function is convex and there is a unique solution for the parameter.
But is there an optimal hyper-parameter? No, here the problem is no longer convex and thus different methods (e.g. grid search, simplex) are available to search for the better hyper-parameters.
For one problem (finding parameter), there is often a unique solution, while for the other problem (finding hyper-parameter), there isn't. So they shouldn't be considered on the same ground or being optimized together.
Regarding "parameter optimization leads to overfitting, but we do not bother too much". Many methods incorporate some forms of regularization in the model, e.g. shrinking the coefficients through penalty term. But the use of validation set is also detecting an overfitting issue where the coefficients might be unnecessarily large.
