I have created a small Pytorch template to try to overfit a small dataset (of only ten points) by using linear regression with polynomial features. The method works by using gradient descent. In principle, the problem is convex and the algorithm should overfit the training data. However that does not happen. I'm not sure if i'm missing something in concept or if there is a problem with the PyTorch code.

  • $\begingroup$ Your code to create the polynomials looks wrong. Are you sure that you create them correctly? $\endgroup$ – lyinch Dec 9 '20 at 13:25
  • $\begingroup$ I believe that part is correct. What I do is to generate polynomials features and fit a linear model. But if you found a mistake, please let me know. $\endgroup$ – Ambesh Dec 9 '20 at 13:30

Maybe try with smaller learning rate? (e.g. 1e-4). Also, try to plot the loss (and maybe the models parameters) during the training process to get some more insight.

  • $\begingroup$ Hi Jacub. I have tried that with no success. I have reduced the learning rate and increased the number of iterations, but it does not seem to converge. I find this strange and perhaps I have some mistake in my code. $\endgroup$ – Ambesh Dec 9 '20 at 11:51
  • 1
    $\begingroup$ yep, so try to plot the loss during training? (e.g. if you see that it is constant over whole training -> probably bug in the code, if it decrease but too slowly -> try higher learning rate, if it oscilates too much -> try lower learning rate). $\endgroup$ – Jakub Koubele Dec 9 '20 at 12:29
  • $\begingroup$ Yes tried that too. Different settings make things slightly better. But in principle I'm fitting with a polynomial of (order 9) 10 different data points. So I guess it is possible to completely overfit the data, such that the resulting model exactly passes over the data points. However that does not happen. $\endgroup$ – Ambesh Dec 9 '20 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.