2
$\begingroup$

After development of recommendation engine with the R, before removal of outliers from data-set value of residual standard error was 1351 and after removal of outlier its 656. Still there is no accurate prediction which gives 10% correct(near) prediction. For more fitting i also have tried polynomial model with two ,three and four degree but still no improvement. Is there any most important thing to consider without R-squared or adjusted R-squared.

Where i am using dataset with linear regression model for prediction of product purchase revenue on the base of total numbers of time product added to cart, removed from cart, total numbers of page views of product page. For checking model prediction accuracy i am considering only minimum residual standard error.

Here is Model summary

> summary(model_out)

Call:
lm(formula = yitemrevenue_out ~ xcartaddtotalrs_out + xcartremove_out + 
    xproductviews_out + xuniqprodview_out + xprodviewinrs_out, 
    data = as)

Residuals:
    Min      1Q  Median      3Q     Max 
-2671.1  -173.6   -83.4   -42.9 14288.6 

Coefficients:
                      Estimate Std. Error t value Pr(>|t|)    
(Intercept)          3.992e+01  1.254e+01   3.183  0.00147 ** 
xcartaddtotalrs_out -7.888e-03  2.570e-03  -3.070  0.00216 ** 
xcartremove_out     -3.410e+01  2.431e+01  -1.403  0.16076    
xproductviews_out    1.248e+01  1.222e+00  10.215  < 2e-16 ***
xuniqprodview_out   -1.350e+01  1.487e+00  -9.076  < 2e-16 ***
xprodviewinrs_out    3.705e-04  5.151e-05   7.193 7.62e-13 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 656.4 on 3721 degrees of freedom
Multiple R-squared: 0.1398, Adjusted R-squared: 0.1386 
F-statistic: 120.9 on 5 and 3721 DF,  p-value: < 2.2e-16 

Thanks

$\endgroup$
6
  • 1
    $\begingroup$ How many observations have you? How many outliers have been deleted? Does the 10% correct prediction mean that you obtained R-squared close to 0.1? Does the prediction concern some external sample or it uses the same observations as for training? The answers on these will clarify your question. $\endgroup$
    – O_Devinyak
    Commented Sep 13, 2012 at 10:17
  • $\begingroup$ 457 observations have been removed from 4184, now its 3727. $\endgroup$ Commented Sep 13, 2012 at 10:46
  • 2
    $\begingroup$ Outlier rejection is a very dangerous approach to data analysis especially when so many are removed. What justification do you have external to the idea that statistics show that they are relatively extreme. You could be throwing out some important and informative data. You might want to consider looking for other factors that could improve the prediction rather than an apparent arbitrary rejection of observations to the extent that the sample could now be biased. If the data is biased the model can look good on the remaining data and still be a poor predictor! $\endgroup$ Commented Sep 13, 2012 at 11:08
  • $\begingroup$ @ViGnEsH is 'yitemrevenue_out' always > 0 ? I assume that one row in your dataset corresponds to one product ? Why don't you add the price of the product then, too ? Prices do influence the conversion rate which is implicitly modeled here. $\endgroup$
    – steffen
    Commented Sep 13, 2012 at 11:09
  • $\begingroup$ Thanks steffen and Micheal chernick. @Steffen, answer -1> no yitemrevenue_out is not alwasy > 0. answer-2->originally its one product per line but here i have ignored product ids. $\endgroup$ Commented Sep 13, 2012 at 13:55

2 Answers 2

3
$\begingroup$

It may simply be that the variables you have measured do not capture behaviour in a predictive way.

I would recommend using 10-fold validation in order to build and assess your models. This is a machine learning technique that allows to to evaluate how data dependent your models are. If theere is a lot of drift in your beta coefficients across folds it means you are modelling noise (overfitting). This approach builds a model on 90% of your data and predicts in the remaining 10%. It does this iteratively until you have predictions for all data points without considering those points in the training process.

You may want to look at mixed-effect modelling - this can account for outliers to some degree. Also, other non-linear models - take a look at fractional polynomials (capture asymptotic data better).

This gives you a good idea of whether you can achieve good prediction accuracy. I suggest reading about prognosis, forecasting and validation.

$\endgroup$
3
$\begingroup$

If by "correct" you mean classification accuracy, that is the cause of a good deal of your problem. Choose a proper accuracy score that is continuous, not arbitrary, and has high precision, for example root mean squared error, mean absolute error, Somers' $D_{xy}$ rank correlation between predicted and observed (a translation of ROC area or concordance probability), $R^2$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.