Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite

I have created a multiple linear regression model with R using lm and glm. I am using lm on a training set and predict on a testing set to validate the model. In one test my results are within 80% of what they should be for 80% of the cases. It correlates with 40% for one response variable and with 63% for another response variable (but the response variable with 63% correlation isn't near the actual values of the prediction). I have 53 predicates. What is the probability of that occurring randomly? I've tried to build an multi-class svm off of the features using the predicates but so far the svm has been unable to properly predict the results.

share|improve this question
You could validate by simulation when the predictors truly have no effect or by permuting the responses to give an idea what prediction accuracy to expect purely by chance. – Macro May 20 '12 at 14:43
@Macro thanks macro i think your pointing towardss cross validation. I ran cv.lm with 10 fold cross validation on the datset with 63% corrolation and i found the Sum of squares = 2.91e+08 is Mean square = 1063519 and the data set with 40% corolation has Sum of squares = 23.1 Mean square = 0.08. A low mean square means that it has predictive ability so my main questions is how to i find the the probability of it concurring randomly? – caseyr547 May 20 '12 at 16:36
1  
If you simulate data where the predictors truly have no effect, this will give you a reference. If you permute the response variables, you will also have a situation where the predictors have no effect, so that would be another way of getting a reference. – Macro May 20 '12 at 16:38

1 Answer

up vote 0 down vote accepted

I found the binomial distribution assuming the probability of each 10% point is equal falling within 10%+/- of the actual answer is 3/10. From there you take the total number, the amount that fit the criteria and the probability and plug them into this page and you find the number is very small. http://homepages.wmich.edu/~bwagner/StatReview/Binomial/binomia%20in%20excel.htm I'll post this as an answer in a couple of hours.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.