I have taken up a machine learning course at my college. In one of the quizes, this question was asked.

Model 1 : $$ y = \theta x + \epsilon $$ Model 2 : $$ y = \theta x + \theta^2 x + \epsilon $$

Which of the above models would fit data better? (assume data can be modelled using linear regression)

The correct answer (according to the professor) is that both models would perform equally well. However I believe that the first model would be a better fit.

This is the reason behind my answer. The second model, which can be rewritten as $ \alpha x + \epsilon $, $\alpha = \theta + \theta^2$ would not be the same as the first model. $\alpha$ is in fact a parabola, and hence has a minimum value ($ -0.25 $ in this case). Now because of this, the range of $ \theta $ in the first model is greater than the range of $ \alpha $ in the second model. Hence if the data was such that the best fit had a slope less than $-0.25$, the second model would perform very poorly as compared to the first one. However in case the slope of the best fit was greater than $-0.25$, both models would perform equally well.

So is the first one better, or are both the exact same?

  • 3
    $\begingroup$ I think you are correct. Requiring that a parameter $\alpha$ be expressible as $\theta + \theta^2$ (for some $\theta$) does indeed enforce a constraint on what $\alpha$'s are possible. This means that the second model can express less relationships than the first, as it is essentially now a constrained optimization problem. Your reasoning seems solid to me. $\endgroup$ – Matthew Drury Sep 17 '16 at 20:06
  • $\begingroup$ @MatthewDrury I just figured out where I went wrong, have a look at the answer below (and the comment) $\endgroup$ – kush Sep 17 '16 at 20:11
  • 3
    $\begingroup$ I see your comment, but that is some pretty serious gymnastics to assume that $\theta$ would take complex values. I would definately attend some office hours to talk through this with your professor. You'll get a good discussion out of it either way. $\endgroup$ – Matthew Drury Sep 17 '16 at 20:16
  • 1
    $\begingroup$ It is not clear to me where the -0.25 comes from. Can you clarify? $\endgroup$ – Mad Jack Sep 18 '16 at 2:55
  • 1
    $\begingroup$ I would be interested in how your professor would fit each model to the two-point dataset $\{(1,-1),(2,-2)\}$. With Model 1 and $\theta=-1$ the fit is perfect, but how would s/he estimate $\theta$ in Model 2 to obtain a perfect fit? $\endgroup$ – whuber Sep 19 '16 at 19:57

Model 2 can be written as: $$y=(\theta + \theta^{2}) x+\epsilon=\beta x+\epsilon.$$ This seems similar to model 1, just with different notation for the hyperparameters ($\theta, \beta $). However, for model 1 we can write

But since in model 2 we have that $$\beta=\theta + \theta^{2},$$ then as you mentioned indeed the range of $\hat{\beta}$ should belong to $[-0.25,+\infty]$ for $\theta \in R$. Which will lead to difference in these 2 models.

Thus in model 2 you are constraining your coefficient estimate unlike model 1. To make this more clear, it should be noted that in model 1, $\hat{\theta}$ is obtained through minimizing the square loss function $$\hat{\theta}=\arg\min_{\theta\in{R}} \ \ (y-X\theta)^{'}(y-X\theta)=(X^{'}X)^{-1}X^{'}y.$$ However in the model 2 the estimate is obtained through $$\hat{\beta}=\arg\min_{\beta\geq-0.25} \ \ (y-X\beta)^{'}(y-X\beta)$$ which might lead to a different result.

  • 1
    $\begingroup$ That makes sense, it just struck me that there is no constraint on $ \theta $ in the second model! In case $ \theta + \theta^2 $ is negative, $ \theta $ might have complex values. However that doesn't really affect the model, right? I don't have rep to upvote, but thanks a lot! $\endgroup$ – kush Sep 17 '16 at 20:10
  • $\begingroup$ @kush Please check my edited response that also adresses your concern $\endgroup$ – Wis Sep 17 '16 at 20:32

Not sure I understand your reasoning. If you take:

$$y = \alpha x+\epsilon$$ and $$y = \theta x + \epsilon$$

and estimate $\alpha$ and $\theta$ using a simple linear regression, you will get $\alpha$=$\theta$. Moreover, since the methodology is exactly the same there is no difference in the $R^2$ value you would get in either equation. The underlying value of $\theta$ in the first equation will of course be different, since $\alpha = \theta + \theta^2$, but this has nothing to do with fit.

  • 5
    $\begingroup$ $ \theta$ in the first model can take any value in range $ (- \infty, \infty) $ However $ \alpha $ in the second model can take values only in range $ (- 0.25, \infty) $. So when we treat both of them as a simple linear regression model, are we not putting a restriction on the coefficient of $ x $ (in the second model)? Wouldn't this raise an issue in case the best fit for the data has a negative slope? $\endgroup$ – kush Sep 17 '16 at 19:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.