Yes, it still matters because you are biasing your predictions based on the training data. In your graph, you can see that the model forecasts a higher value for the values in the middle and lower for the values on the end. This means, that if there is actually a linear relationship between the diameter and the price, you will see higher errors due to this bias.
If on the other hand, the relationship isn't actually linear, it may be ok to have a graph like this. If you do have a relationship between diameter and price where it increases quickly for the low values and then levels off as the diameter gets larger, you would want to capture that in the model. A logarithmic model might be a better way of doing this, but if for one reason or another that isn't a reasonable model for you to build, you might be able to approximate the behavior in the value range of interest using a quadratic model.
The important thing is to try to capture the relationship between your predictors and your outcome with the acknowledgement that generally simpler models can do that better in small data. If you know your data don't behave the way your model does, you are unlikely to get good results.