Understanding bias and variance for different models over same dataset Consider we have 1-D data generated by a polynomial of degree 5. Which will of thhe following give higher / lower bias and higher / lower variance?

*

*Regression with linear basis functions

*Regression with polynomial basis functions of degree at most 5

*Regression with polynomial basis functions of degree at most 15

My understanding is as follows:

*

*Linear basis function will give least variance but highest bias

*Degree 15 polynomial basis function will end up behaving similar to degree 5 polynomial basis function and hence both will give same bias and variance.

Q1. I am correct with these?
I will like to generalise point 2 above to a question Q2. Will higher degree polynomial basis function alway give higher variance than lower degree polynomial basis function?
 A: I'd be a little more careful about your answer. Specifically, you are correct in the first part, in that a linear model has high bias.
Check your understanding for 2, however. Even though you know your data was generated by the degree-5 polynomial, the degree-15 polynomial will still have higher variance. Intuitively, this has to do with the lesser "flexibility" of the degree-5 polynomial versus that of the degree-15 polynomial.
The variance of a model $f$ is the variance of the predicted label, $\hat{y} = f(x)$, where $x$ is the point we evaluate our model at, drawn from the underlying distribution. See that the "randomness" in the variance is from training $f$ on different sets of data. If our label changes wildly depending on the sets of data we choose, we know our model is "high variance". This solidifies the intuition one has about a high variance vs low variance model.
Notice, we can't usually calculate the variance of our model directly; doing so would require us to know how the data is generated, which defeats the purpose of our prediction task!
This is all a long-way of saying that a degree-15 polynomial is more prone to giving us predictions that "vary more" than a degree-5 polynomial's, and, thus, has higher variance.
A: In a model there is a trade-off between variance and bias. Complexity of a model is the number of regressors used, e.g. multiple polynomials as in your case. With increasing complexity of a model variance increases and bias decreases. Hence the well-known terms overfitting (low bias, high variance) and underfitting (high bias, low variance).
I assume that by "linear basis function" you mean a polynomial of 1st degree (i.e. a line). If the DGP (data-generating process, i.e. true model) is a polynomial of 5th degree, fitting a lower-order polynomial (such as a line) means that the whole complexity was not captured, leading to higher bias (the function is not able to wiggle enough) and less variance (as a result), i.e. we have an underfit (here you are correct). The inverse holds for fitting a higher-order polynomial, such as 15th degree: too much complexity was assumed and we are now overfitting, i.e. going after the random noise in the data leading to little bias (the function adapts every time) but high variance (very wiggly function). Fitting a 5-degree polynomial which is the true function leads to no bias and least variance. It leads to less variance than the higher order polynomial, since no redundant variance is incurred by adding unnecessary regressors (here you are not correct).
Regarding your Q2: it depends on the degree of complexity of the DGP. Assuming you have enough data points, if you fit something more complex than needed it's an overfit, otherwise underfit - always relative to the DGP.
A: Assume you have housing data to predict prices. A polynomial equation of order 5 fits your training data well.Instead, you tried simple model, built a linear model(d=1), your estimates are too off from real values. Your model has a high bias, less variance.
You tried, d= 2, 3, 4, 5  => Your bias decreases and variance increases. At d = 5, with the equation fitting your data, you have minimum bias and highest variance(out of all the cases)
If you try higher order d = 6, 7, 8, 9, 10, 11, 12, 15 to fit your data, bias will decrease and variance will increase but at very slower rate.
 (Machine Learning:Andrew Ng-Coursera)
