Why do these learning curves look the same in cases of overfitting and proper fit? For the following randomly generated data with 1 feature, I plot the learning curves.
Data generation
m = 500
X = 6 * np.random.rand(m, 1) -3
# y = f(x) + gaussian noise
y = 0.5 * (X**2) + X + 2 + np.random.randn(m, 1)

function to plot learning curves
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

def plot_learning_curves(model, X, y):
    X_train, X_val, y_train, y_val = \
        train_test_split(X, y, test_size=0.2, random_state=42)
    train_errors, val_errors = [], []
    for m in range(1, len(X_train)):
        model.fit(X_train[:m], y_train[:m])
        y_train_predict = model.predict(X_train[:m])
        y_val_predict = model.predict(X_val)
        train_errors.append(mean_squared_error(y_train_predict, y_train[:m]))
        val_errors.append(mean_squared_error(y_val_predict, y_val))
    plt.plot(np.sqrt(train_errors), "r-.", linewidth=2, label="train")
    plt.plot(np.sqrt(val_errors), "b-", linewidth=3, label="val")
    plt.legend(loc='upper right')
    plt.ylim([0, 3])

Polynomial Regression pipeline
from sklearn.pipeline import Pipeline

def polynomial_regression(n):
    poly_reg = Pipeline([
        ('poly_features', PolynomialFeatures(
                                degree=n,
                                include_bias=False)),
        ('lin_reg', LinearRegression())
    ])
    return poly_reg

Learning Curve (degree=1) - Underfit
from sklearn.linear_model import LinearRegression
plot_learning_curves(LinearRegression(), X, y)


Learning Curve degree 2 - Correct fit

Learning Curve degree 10 - Overfit

I am having trouble interpreting these curves.

*

*Shouldn't the curves be closer in the underfitting case?

*Why does the correct fit(degree 2) look almost exactly like the overfit case (degree 10)?

*Shouldn't there be a larger gap between the training and validation curves in the overfitting case?

 A: 
Shouldn't the curves be closer in the underfitting case?

I think normally they should be close but this may be an issue not of how the $y$ are generated but how you sample the $x$ and what metric you choose. In this case you take MSE as an error metric (which seems weird because when repeating the experiments in R I obtain MSEs and RMSEs much higher than 1.5...). In any case this should be something like
$$\frac{1}{\text{#samples}} \sum_{i} (y_i - \hat{y}_i)^2$$
which is sensitive to outliers. I.e. since you have chosen the $x_i$ to be normally distributed as well (and not uniformly distributed), most of the $x_i$ cuddle up close to zero. Hence, when adding a single $x_i$ which is far away from zero in the validation set which is not contained in the test set then you will end up having an unaturally high error in test but not in train.
Maybe you should try to choose the x uniformly from some range $[-3, 3]$ then this difference should disappear.

Shouldn't the curves be closer in the underfitting case?
Shouldn't there be a larger gap between the training and validation curves in the overfitting case?

Well actually, they do not look alike at all :-)
There is something you need to understand about the concept of overfitting: overfitting some data set does not state that we are trying to fit a model that is somewhat 'generally' bad, it means that the model is 'overexpressive' for the (possibly underrepresentative) data set given. Let me illustrate that with a simple example from classification:
We use a dataset that has some distribution of the $x \in [0,1]$ and $y$ is like
$$y = \text{true iff. $x > 0.5$}$$
i.e. the 'true' function looks like this:

We try to fit this function with training sets and function candidates of the form
$$f(x) = \mathbf{1}_{x > \theta} + (-2)*\mathbf{1}_{x \in [a,b]}$$
where $\theta < a < b$, i.e. these candidates look like so:

Clearly these candidates are too complicated to fit an easy step function so let us see what happens if we start with a trainign set where one $y$ value (the last one) is disturbed: it should be positive but due to random effects, it is negative.
$x = (0.15, 0.61, 0.78) ~~~ y = (-1, +1, -1)$$
Now when trying to fit this dataset with the overexpressive functions we get out something bad:

but here comes the catch: what happens when we add more and more 'correct' data points to the training set? The bad gap on the right will become smaller and smaller! I.e. the function fits the true function better and better and the gap between training performance and test performance disappears:

In short: We do not try to fit a bad function, we try to fit a function that might overinterpret little errors ion the data.

As these little errors in the data reduce with an increasing amount of data you feed the model, the complcicated function bows more and more into the direction of the correct function:

I.e. overfitting is only a problem when giving an insufficient amount of data to the model. When feeding a suffient amount, the complex model will be as good as the simple model. That point is reached in your case when feeding it more than 150 data points or so. Before that, the two curves do look very different!
In real world data sets we are in this critical area of $<150$ samples, so we need to take good care of regularizing the models that we train.
EDIT:
That is weird, when I repeat the experiment using R I get the expected tigh curves for question no. 1... So that seems to be an issue either of the sampling process of x or there is something weird in the code (although I cannot spot anything that is obviously wrong with it)...

red=RMSE, black=MAE and dotted is on train, straight line is on test.
