I do not know for which model the L2 penalization is presented. I will suppose it is linear regression, but the same discussion applies to other models, also. For other more complex models the discussion is more involved, but not impossible.
I don't understand why the weights need to be penalized?
The weights are the parameters of your model. For linear model you have $y = w^T x + \epsilon$. Since $\epsilon$ is a model for random noise, you remain only with $w$ to change in order to impose some learning bias into the model.
In linear models if the features are not independent, they geometrically look like a very closed vectors. Doing regression is the same as to find a direction in which the errors are smaller, or in other words a normal hyper plane for that direction. However this direction is not stable, because small changes in errors of features will lead to very different vectors for errors (remember they are close one to another because correlation). As a consequence it often lead to high values for weights.
Another explanation for linear models is that with correlated features we will have a matrix which is hard to be inverted, as such it leads to high values for correlation.
In general, however, having features which share a lot of information have the potential to lead to not stable models. Instability means high variance. At the same time instability means high values for weights. All those situations are covered under the statement: "small changes in input data leads to very different values in output data".
Without the regularization term, wouldn't we be able to get to the center (optimal), which is the exact purpose of optimization of the loss function?
What you said is not completely true. We have training data. Training data is just a sample of reality. Finding optimal error on sample data does not mean you can generalize. What we want to find is the optimal setup for any new data. Because of our limited sample, possible model inadequacy, possible wrong assumptions, the optimal on training data is different than optimal for any new data.
In order to search for better generalization, we need have a good compromise between bias and variance. But those to comes in opposite directions. When you decrease bias, aka go as closer as possible to optimal in training data, you start to depend on data a lot, and of course on any error or fine structure. By doing that you model become unstable. At the other side, if you decrease the variance of the model, you increase the stability and go farther from training optimal. The most stable model is the one which is independent of all features, the one which have weight 0. That model is really stable, it simply does not change no matter the input, but is useless.
You have to find a compromise, and reducing weights by regularization is one way to go.
Why large weights means high variance?
Large weights means if your input has a small error, when multiplied with large weight it become large error in output / prediction, so the model is unstable, so it has high variance.
"Why large weights means the model is overfitted, or not well generalized?"
Over fitting means you learn too much from data. That means your model learns also the irreducible error. Because it learns the noise, it have a high variance, the model is unstable. As a consequence the model, on new data, will predict using other noise (from new data) and of course will be poor as a predictor, small generalization. Since we have a high variance it means small changes in inputs leads to large changes in outputs. When this idea is applied to a function where weights multiplies or amplifies inputs in some way, one mathematical possibility is that the weights are large, most of the time.
