"Penalizing" a Machine Leaning algorithm essentially means that you do not want your algorithm to be overfitted to your data. Have a look at this picture
The first plot shows a ML model that is under fitted to the data and thus is not able to capture the pattern of the data.
The second plot shows that what your ML will predict (dashed line) follows the trend of your data in some way.
The third picture on the right is very fitted to the data you train your algorithm on. This is bad for many reasons, but the main reason is that your training data does not contain all the data in the world.
The model in the second plot is better than the third because is more robust to predictions on new data (usually named test data).
Now, There exists a large amount of algorithms that can fit the distribution of your data and you need to pick among these many.
A good way to do that is by "penalizing" the complexity of your model (e.g. assigning a negative cost (linear or quadratic cost are the most common) to the size of your weight parameter. This will result in a more robust model, i.e. similar to the one in the center. I hope this will help to clarify a bit.