Methods of modifying objective functions to control the solutions of optimization problems.
Penalization is a way of modifying a criterion function, like a likelihood or a sum of squares, so as to take into account other criteria than pure fit criteria in estimation. Examples can be penalized least squares or smoothing splines. Regularization methods like lasso or ridge regression also uses penalization.
Wikipedia has an article https://en.wikipedia.org/wiki/Smoothing_spline with further references. Some books dedicated to the topic are https://www.amazon.com/Penalized-Likelihood-Estimation-Springer-Statistics/dp/0387952683/ref=sr_1_1?ie=UTF8&qid=1509819398&sr=8-1&keywords=penalized+likelihood and https://www.amazon.com/Maximum-Penalized-Likelihood-Estimation-Regression-ebook/dp/B00FB28GLM/ref=sr_1_2?ie=UTF8&qid=1509819398&sr=8-2&keywords=penalized+likelihood