The importance of lambda in a regularization function, with respect to the hypothesis. I'm working through some parts of Russel & Norvig's Artificial Intelligence, and this is the cost function they give:
Cost(h) = EmpericalLoss(h) + λComplexity(h)
How does choosing a value for lambda affect how a hypothesis is selected, if lambda is not present in the hypothesis? Or am I thinking about this in the wrong way?
 A: Framed in this way, $\lambda$ isn't a parameter that appears in the hypothesis itself, but a hyperparameter that affects what hypothesis is chosen (i.e. how the other parameters are selected). You wrote a cost function that takes a hypothesis as input and returns the associated cost, which is a scalar value that reflects the 'badness' of the hypothesis. The hypothesis is chosen that minimizes the cost. One term in the cost function is the empirical loss. According to this criterion, hypotheses that fit the data better have lower cost. The cost function also includes a penalty term for model complexity. According to this criterion, models that have lower complexity have lower cost. In many circumstances, these two criteria are opposed, because higher complexity models are capable of fitting the data better. When a cost function is written this way, the goal is to make a tradeoff between how well the data is fit and how complex the model is. The hyperparameter $\lambda$ controls this tradeoff by adjusting the weight of the penalty term. If $\lambda$ is increased, model complexity will have a greater contribution to the cost. Because the minimum cost hypothesis is selected, this means that higher $\lambda$ will bias the selection toward models with lower complexity.
A: I'm not well versed in that book, but a factor added to a cost function which is proportional to complexity is generally used to prevent model overfitting.
Fairly obviously, if you could increase the complexity effectively to infinity, then you could make the loss as small as you like (on your training data), but the model you select would be almost useless when you test it on other data. So you want the model you select to trade off accuracy (on your training data) and complexity.
Someone will no doubt come in with more detail, but that's generally the idea.
