A hyperparameter is simply a parameter that impacts, completely or partly, other parameters. They do not directly solve the optimization problem you face, but rather optimize parameters that can solve the problem (hence the hyper, because they are not part of the optimization problem, but rather are "addons"). For what I've seen, but I have no reference, this relationship is unidirectional (a hyperparameter cannot be influenced by the parameters it has influence on, hence also the hyper). They are usually introduced in regularization or meta-optimization schemes.
For example, your $\lambda$ parameter can freely impact $\mu$ and $\sigma$ to adjust for the regularization cost (but $\mu$ and $\sigma$ have no influence on $\lambda$). Thus, $\lambda$ is a hyperparameter for $\mu$ and $\sigma$. If you had an additional $\tau$ parameter influencing $\lambda$, it would be a hyperparameter for $\lambda$, and a hyperhyperparameter for $\mu$ and $\sigma$ (but I've never seen this nomenclatura, but I wouldn't feel it would be wrong if I saw it).
I found the hyperparameter concept very useful for cross-validation, because it reminds you of the hierarchy of parameters, while also reminding you that if you are still modifying (hyper-)parameters, you are still cross-validating and not generalizing so you must remain careful about your conclusions (to avoid circular thinking).