1
$\begingroup$

I want to predict sales with lm and machine learning so I did this:

library("caret")
set.seed(1)
in_train <- createDataPartition(open_store$Sales, p = 0.75, list = FALSE)
training <- open_store[in_train,]
testing <- open_store[-in_train,]  
total_fit <- lm(Sales ~ DayOfWeek + Promo + SchoolHoliday + StateHoliday + Month + Year + StoreType + Assortment + log(CompetitionDistance), data = training, na.action = na.omit)
summary(total_fit)
plot(total_fit$fitted.values, total_fit$residuals)
qqnorm(total_fit$residuals,  ylab = "Residual Quantiles")
total_predict <- predict(total_fit, testing)
plot(total_predict, testing$Sales)
abline(lm(total_predict~testing$Sales), col="red")
predict_eval <- lm(testing$Sales~total_predict)
summary(predict_eval)
rmse_train <- sqrt(mean(total_fit$residuals ^ 2))
rmse_test <- sqrt(mean(predict_eval$residuals ^ 2))
rmse_test/rmse_train

My problem is that after taking out the zeros from the initial dataset my R-sq gets really low (around 0.25) which means that my model is not good enough although my RMSE ratio is pretty good (approx. 0.99). If I leave them in the R-sq is around 0.85 but the plot looks awful. I went through some discussions and some people suggest to do my model with rlm:

library(MASS)    
total_fit <- rlm(Sales ~ DayOfWeek + Promo + SchoolHoliday + StateHoliday + Month + Year + StoreType + Assortment + log(CompetitionDistance), 
    data = training, 
    na.action = na.omit)

But it neither provides R-sq nor p values. Does anyone have a suggestion how to improve this model and how to estimate if the model performs better with lm or rlm?

Thank you in advance!

$\endgroup$
1
  • $\begingroup$ Maybe edit your question to skip the irrelevant stuff - all you need is the dataset and the rlm call. $\endgroup$
    – dash2
    Commented Mar 21, 2017 at 9:33

1 Answer 1

-1
$\begingroup$

Looks to me that rlm won't provide you with this info. So, you'll have to calculate it yourself.

Hmm. I Am Not A (Proper) Statistician, but something like:

1 - sum(residuals(total_fit)^2)/sum((testing$Sales-mean(testing$Sales))^2)

might do it.

$\endgroup$
5
  • $\begingroup$ Tnx but I got 2.322097 which can't be the R-sq as it's supposed to vary from 0 to 1. May also ask if it's okay to have almost the same RMSE = 2000+ for both the model and the test if the observations are approx 1M? And if the RMSE ratio is 0.99, is it a good indicator or it just says the both the model and the test are not good? Tnx $\endgroup$
    – Desislava Spirova
    Commented Mar 21, 2017 at 10:00
  • $\begingroup$ Sounds like a q for cross validated $\endgroup$
    – dash2
    Commented Mar 21, 2017 at 13:02
  • $\begingroup$ I find it extremely unlikely that 1 minus the ratio of two sums of squares, both of which must of course be positive, came to greater than 1. Perhaps check your code. (Or mine.) $\endgroup$
    – dash2
    Commented Mar 21, 2017 at 13:04
  • $\begingroup$ oh yeah.. it's actually -1.32 but still not really helpful. Do you think if I take out the outliers from the initial dataset would be a good option? $\endgroup$
    – Desislava Spirova
    Commented Mar 21, 2017 at 17:29
  • $\begingroup$ Not sure. Negative R2 is possible if your prediction line is worse than a flat one through the mean. To me, this sounds as if rlm is not doing much for you. $\endgroup$
    – dash2
    Commented Mar 21, 2017 at 17:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.