Actually, your question is about model selection and not about statistical significance of regression parameters. Although the latter can be, and often is used for reducing a model, it is somewhat dubious for picking the degree of a polynomial, because the variables $x$ and $x^2$ are far from independent and, depending on their range, can be highly correlated:
> x <- -5:5
> cor(x, x^2)
[1] 0
> x <- 0:10
> cor(x, x^2)
[1] 0.9631427
Generally, using higher degree polynomials for fitting will always increase the fitting accuracy (measured by $R^2$ in linear regression) on the data used for fitting, but at some point, it will result in overfitting. To find this turnaround point, different indices can be used, e.g.:
- the leave-one-out mean squared error or, equivalently, the leave-one-out $R^2$
- the Akaike Information Criterion (AIC)
For linear models, both criteria are asymptotically equivalent. It is thus easiest to use AIC (which is a builtin R function) to find an appropriate degree: lower AIC is better. Code outline:
for (degree in 1:3) {
cat(sprintf("degree=%i: AIC=%f\n", degree, AIC(lm(y ~ poly(x,degree), data))))
}