I have created a small Pytorch template to try to overfit a small dataset (of only ten points) by using linear regression with polynomial features. The method works by using gradient descent. In principle, the problem is convex and the algorithm should overfit the training data. However that does not happen. I'm not sure if i'm missing something in concept or if there is a problem with the PyTorch code.