So in a normal distribution, we have two parameters: mean $\mu$ and variance $\sigma^2$. In the book Pattern Recognition and Machine Learning, there suddenly appears a hyperparameter $\lambda$ in the regularization terms of the error function.
What are hyperparameters? Why are they named as such? And how are they intuitively different from parameters in general?
The term hyperparameter is pretty vague. I will use it to refer to a parameter that is in a higher level of the hierarchy than the other parameters. For an example, consider a regression model with a known variance (1 in this case)
$$ y \sim N(X\beta,I) $$
and then a prior on the parameters, e.g.
$$ \beta \sim N(0,\lambda I) $$
Here $\lambda$ determines the distribution of $\beta$ and $\beta$ determines the distribution for $y$. When I want to just refer to $\beta$ I may call it the parameter and when I want to just refer to $\lambda$, I may call it the hyperparameter.
The naming gets more complicated when parameters show up on multiple levels or when there are more hierarchical levels (and you don't want to use the term hyperhyperparameters). It is best if the author's specify exactly what is meant when they use the term hyperparameter or parameter for that matter.
A hyperparameter is simply a parameter that impacts, completely or partly, other parameters. They do not directly solve the optimization problem you face, but rather optimize parameters that can solve the problem (hence the hyper, because they are not part of the optimization problem, but rather are "addons"). For what I've seen, but I have no reference, this relationship is unidirectional (a hyperparameter cannot be influenced by the parameters it has influence on, hence also the hyper). They are usually introduced in regularization or meta-optimization schemes.
For example, your $\lambda$ parameter can freely impact $\mu$ and $\sigma$ to adjust for the regularization cost (but $\mu$ and $\sigma$ have no influence on $\lambda$). Thus, $\lambda$ is a hyperparameter for $\mu$ and $\sigma$. If you had an additional $\tau$ parameter influencing $\lambda$, it would be a hyperparameter for $\lambda$, and a hyperhyperparameter for $\mu$ and $\sigma$ (but I've never seen this nomenclatura, but I wouldn't feel it would be wrong if I saw it).
I found the hyperparameter concept very useful for cross-validation, because it reminds you of the hierarchy of parameters, while also reminding you that if you are still modifying (hyper-)parameters, you are still cross-validating and not generalizing so you must remain careful about your conclusions (to avoid circular thinking).
The other explanations are a bit vague; here's a more concrete explanation that should clarify it.
Hyperparameters are parameters of the model only, not of the physical process that is being modeled. You introduce them "artificially" to make your model "work" in the presence of finite data and/or finite computation time. If you had infinite power to measure or compute anything, hyperparameters would no longer exist in your model, since they wouldn't be describing any physical aspect of the actual system.
Regular parameters, on the other hand, are those that describe the physical system, and aren't merely modeling artifacts.
It's not a preciseley defined term, so I'll go ahead and give you yet another definition that seems to be consistent with common usage.
A hyperparameter is a quantity estimated in a machine learning algorithm that does not participate in the functional form of the final predictive function.
Let me unwind that with an example, ridge regression. In ridge regression we solve the following optimization problem:
$$ \beta^*(\lambda) = \text{argmin}_{\beta} \left( (y - X\beta)^t (y - X\beta) + \lambda \beta^t \beta \right)$$ $$ \beta^* = \text{argmin}_{\lambda} (y' - X'\beta(\lambda))^t (y' - X'\beta(\lambda)) $$
In the first problem $X, y$ is the training data, and in the second $X', y'$ is a hold out data set. The final functional form of the model, which I called above the predictive function is
$$ f(X) = X \beta^* $$
in which $\lambda$ does not appear. This makes $\beta$ a parameter vector, and $\lambda$ a hyper parameter.
As precisely pointed out by @jaradniemi, one use of the term hyperparameter comes from hierarchical or multilevel modeling, where you have a cascade of statistical models, one built over/under the others, using usually conditional probability statements.
But the same terminology arises in other contexts with different meanings as well. For instance, I have seen the term hyperparameter been used to refer to the parameters of the simulation (running length, number of independent replications, number of interacting particles in each replication etc.) of a stochastic model, which did not result from a multilevel modeling.
