What does Penalize a learning algorithm mean in Machine Learning? I am new to Machine Learning and have taken Andrew Ng's course on Machine Learning.
In one of the Logistic regression videos  for binary classification for the error case where predicted value through logistic regression = 1 and actual value = 0 so error = 1. I didn't quite understand the statement " We'll have to penalize the learning algorithm by a very large cost". What does penalize a learning algorithm mean ?
 A: Penalizing a Machine Leaning algorithm essentially means that you do not want your algorithm to be overfitted to your data. Have a look at this picture

The first plot shows an ML model that is under-fitted to the data and thus is not able to capture the pattern of the data.
The second plot shows that what your ML model will predict (dashed line) follows the trend of your data in some way.
The third picture on the right is very fitted to the data you train your algorithm on. This is bad for many reasons, but the main reason is that your training data does not contain all the data in the world.
The model in the second plot is better than the third because is more robust to predictions on new data (usually named test data).
Now, There exists a large number of algorithms that can fit the distribution of your data and you need to pick among these many.
A good way to do that is by "penalizing" the complexity of your model (e.g. assigning a negative cost (linear or quadratic cost are the most common) to the size of your weight parameter. This will result in a more robust model, i.e. similar to the one in the center.
A: When we penalize a machine learning algorithm, we penalize the algorithm for fitting a model that fits the training data tightly. Usually this is done by estimating the training error as the sum of squared errors plus some measurement of the strength of the fit. For LASSO and Ridge Regression, we can choose to penalize larger coefficients of our model as the training data may be suspected of being too influential on the model parameters.
For example:
Training error = SSE + $\lambda*\Sigma^p_{i=1}\beta_i^2$
Here the $\lambda$ is a tuning parameter for how much we want to account for strong coefficients, but hopefully you can see how larger coefficients ($\beta$ values) would create a larger training error. Our machine learning algorithm will lead us to pick the model that minimizes our training error. In the case you described, $\lambda$ would be assigned a larger value to penalize the model for overfitting to the training data.
The idea is to create a model that doesn’t fit training data close so that the model may better predict the test data.
A: In ordinary regression, the returned fit is the best fit on the training data. This can lead to over-fitting. Penalizing means that we add a penalty for over-confidence in the parameter values. Thus, we accept a slightly worse fit in order to have a simpler model.
