0
$\begingroup$

I am using OLS to predict price for a data set of cars. I have the following categorical features:

 ['ModelYear', 'Province', 'BodyType', 'Year', 'Month','TransmissionType', 'CarFuelType']

and only one non-categorical feature:

['Milage']

When I use OLS for the entire data set, I get the following results:

Test r^2 score:  0.884958272024
(normalized) Mean Sqr Error:  14787783.2542
AVG price:  172792220.37448004
STD price:  43601151.72460971

However, when I subset on the ModelYear (let say on 2016) I get the following results:

Test r^2 score:  0.0697490850982
(normalized) Mean Sqr Error:  16922600.0875
AVG Test price:  209564708.85093167
STD Test price:  17548958.668762047

As you see on the subset we have lower test variance, as is expected. However, MSE is increased and R^2 is decreased. This is the case for all other ModelYears as well. How is this possible? My intuition is that on a smaller data set I should have better prediction.

I am standard scaling all the features including dummy variables for categorical features.

$\endgroup$
8
  • 1
    $\begingroup$ You probably know that as sample size increases, the variance of the OLS estimator decreases. That is not surprising: given more data we can estimate the model parameters with higher precision. So what is the puzzle? $\endgroup$ May 14, 2017 at 8:27
  • $\begingroup$ Here it is not with higher precision, as the the data is increased. It is the other way around. The MSE is increased as the data has become smaller (second case). $\endgroup$
    – Hamed
    May 14, 2017 at 8:39
  • 2
    $\begingroup$ Exactly. Less data --> higher MSE --> lower precision. More data --> lower MSE --> higher precision. Probably you misunderstood what MSE indicates: high MSE means low precision. $\endgroup$ May 14, 2017 at 8:48
  • $\begingroup$ what is your dependent variable ? $\endgroup$
    – user10619
    May 14, 2017 at 9:16
  • $\begingroup$ I think something is misunderstood. Note that the data is subset on one feature, namely ModelYear. So I am considering all cars built in 2016, and since this is a categorical feature, cars built on other years should NOT influence the line fitted for each of them (since different coefficient is learnt for each year, using dummy variables). Also note that, as I said I have this increase in MSE when I subset on all years. Moreover the data is big enough to compensate for possible reduction. @Richard $\endgroup$
    – Hamed
    May 14, 2017 at 9:16

1 Answer 1

0
$\begingroup$

What is the difference between 'Model Year' and 'Year'?

Your intuition that a smaller data set has better prediction is wrong because predictive power of a model increases with more information in terms of both breadth (more variables) and breadth (number of observations).

R squared indicates the total amount of variation that is explained by each of your independent variables. Each independent variable can contribute differently towards explaining the variation in the dependent variable.

It is possible that 'Model Year' is a better predictor of price than the other variables, which led to a high R squared and low MSE. And that predictive power was lost when you removed the 'Model Year' variable, leading to a higher MSE in the subset model.

It is also possible that there are other variables that were not considered in your model that are important to predict price correctly.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.