# How do I know if a model with a subset of the features of another model has lowest training/test error?

I'm doing the Machine Learning specialization from the University of Washington on Coursera, and I have to answer some questions in a quiz from the Regression course.

They ask which model would have the lowest training error if the features of Model 1 are a strict subset of those in Model 2.

How do I check this?

In my point of view this is not enough information to say which model have the lowest training error, because I think the training error doesn't depend on the number of features, but on the number of observations.

Am I wrong?

Recall that in both cases, you are trying to solve an optimisation problem. That is, you have a loss function (e.g. $L^2$ or $L^1$ norm) and you want to find the value of the parameter vector that minimises loss. So your problem has two components: a loss function (or objective function) and a parameter space (this is the key part for you).

Your two models (Model 1 and Model 2) are related, of course: they share the same loss function, and the parameter space for Model 1 is a subset of the parameter space for Model 2.

Can you take it from here?

• Note that you are correct in saying that sample size impacts the training error. But in this case, the sample size is the same for each model. – M Turgeon Jun 29 '16 at 16:22
• One of the lectures says that the more complex a model is, the lower the training error will be. Thus, I can say that Model 1 will have a higher training error because Model 2 has a more complex model. Is that right? – Juliano Nunes Silva Oliveira Jun 29 '16 at 16:46
• @JulianoNunesSilvaOliveira Yes, that's essentially it. – M Turgeon Jun 29 '16 at 16:50
• What if the feature that is left out of the model is perfectly dependent on a feature in the model, would the error be the same ? – Xavier Bourret Sicotte Jun 5 '18 at 12:47