1
$\begingroup$

I am new to Machine Learning and have taken Andrew Ng's course on Machine Learning.

In one of the Logistic regression videos for binary classification for the error case where predicted value through logistic regression = 1 and actual value = 0 so error = 1. I didn't quite understand the statement " We'll have to penalize the learning algorithm by a very large cost". What does penalize a learning algorithm mean ?

Improve this question
$\endgroup$
4
  • 5
    $\begingroup$ We could make good guesses on what was meant, but nonetheless please add more context to the question. $\endgroup$
    – Tim ♦
    Jul 3, 2020 at 20:06
  • $\begingroup$ Here is the video link link what does penalizing algorithm at 6:33 mean ? $\endgroup$
    – vishrut pundir
    Jul 3, 2020 at 20:15
  • $\begingroup$ I think you are talking about regularization, overfitting and bias-variance trade-off. Give me some time and I will provide a question. $\endgroup$
    – Apprentice
    Jul 4, 2020 at 13:19
  • 1
    $\begingroup$ The answers thusfar discuss regularisation. Another form of penalisation you might encounter is in the case of predictive models. For example, say you have a model that detects fraud. False positives are fine; they'll be investigated and resolved, with small cost. False negatives will go unseen and can cost the company millions. You may want to penalise false negatives when fitting the model to account for this disparity in cost. $\endgroup$
    – alan ocallaghan
    May 14, 2021 at 23:52

3 Answers 3

Reset to default
5
$\begingroup$

Penalizing a Machine Leaning algorithm essentially means that you do not want your algorithm to be overfitted to your data. Have a look at this picture

enter image description here

The first plot shows an ML model that is under-fitted to the data and thus is not able to capture the pattern of the data.

The second plot shows that what your ML model will predict (dashed line) follows the trend of your data in some way.

The third picture on the right is very fitted to the data you train your algorithm on. This is bad for many reasons, but the main reason is that your training data does not contain all the data in the world.

The model in the second plot is better than the third because is more robust to predictions on new data (usually named test data).

Now, There exists a large number of algorithms that can fit the distribution of your data and you need to pick among these many.

A good way to do that is by "penalizing" the complexity of your model (e.g. assigning a negative cost (linear or quadratic cost are the most common) to the size of your weight parameter. This will result in a more robust model, i.e. similar to the one in the center.

Improve this answer
$\endgroup$
0
$\begingroup$

When we penalize a machine learning algorithm, we penalize the algorithm for fitting a model that fits the training data tightly. Usually this is done by estimating the training error as the sum of squared errors plus some measurement of the strength of the fit. For LASSO and Ridge Regression, we can choose to penalize larger coefficients of our model as the training data may be suspected of being too influential on the model parameters.

For example: Training error = SSE + $\lambda*\Sigma^p_{i=1}\beta_i^2$

Here the $\lambda$ is a tuning parameter for how much we want to account for strong coefficients, but hopefully you can see how larger coefficients ($\beta$ values) would create a larger training error. Our machine learning algorithm will lead us to pick the model that minimizes our training error. In the case you described, $\lambda$ would be assigned a larger value to penalize the model for overfitting to the training data.

The idea is to create a model that doesn’t fit training data close so that the model may better predict the test data.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ You don't penalize a model for fitting the training data, that's exactly what you're training it to do. How well the model fits the training data is captured in the SSE, which should be minimized. I don't think I've seen SSE + penalty described as "training error", as error is a difference between observed and predicted values. The penalty term isn't an error term, so describing error + penalty as simply error doesn't seem correct. A simple model that fits data perfectly doesn't get penalized because it "fits the data tightly". $\endgroup$
    – Nuclear Hoagie
    May 14, 2021 at 19:24
-1
$\begingroup$

In ordinary regression, the returned fit is the best fit on the training data. This can lead to over-fitting. Penalizing means that we add a penalty for over-confidence in the parameter values. Thus, we accept a slightly worse fit in order to have a simpler model.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ The regularized model is also the "best fit" to the training data. The difference is that the regularized model is the best fit as measured using a penalized loss function, a function which is different from the ordinary regression case. "Over-confidence" in parameter values is not a statistical concept; it seems that your last two sentences are allusions to the concept of bias-variance-tradeoff. $\endgroup$
    – Sycorax ♦
    May 14, 2021 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.