I'm doing the Machine Learning specialization from the University of Washington on Coursera, and I have to answer some questions in a quiz from the Regression course.

They ask which model would have the lowest training error if the features of Model 1 are a strict subset of those in Model 2.

How do I check this?

In my point of view this is not enough information to say which model have the lowest training error, because I think the training error doesn't depend on the number of features, but on the number of observations.

Am I wrong?


Recall that in both cases, you are trying to solve an optimisation problem. That is, you have a loss function (e.g. $L^2$ or $L^1$ norm) and you want to find the value of the parameter vector that minimises loss. So your problem has two components: a loss function (or objective function) and a parameter space (this is the key part for you).

Your two models (Model 1 and Model 2) are related, of course: they share the same loss function, and the parameter space for Model 1 is a subset of the parameter space for Model 2.

Can you take it from here?

  • $\begingroup$ Note that you are correct in saying that sample size impacts the training error. But in this case, the sample size is the same for each model. $\endgroup$ – M Turgeon Jun 29 '16 at 16:22
  • $\begingroup$ One of the lectures says that the more complex a model is, the lower the training error will be. Thus, I can say that Model 1 will have a higher training error because Model 2 has a more complex model. Is that right? $\endgroup$ – Juliano Nunes Silva Oliveira Jun 29 '16 at 16:46
  • $\begingroup$ @JulianoNunesSilvaOliveira Yes, that's essentially it. $\endgroup$ – M Turgeon Jun 29 '16 at 16:50
  • $\begingroup$ What if the feature that is left out of the model is perfectly dependent on a feature in the model, would the error be the same ? $\endgroup$ – Xavier Bourret Sicotte Jun 5 '18 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.