3
$\begingroup$

I am using R and the caret package.

My code is very straight forward.

set.seed(2112)
trainingDataset <- read.csv("Input.txt")
train_control <- trainControl(method="repeatedcv", number = 10, repeats = 3)
fit <- train(eval ~ ., trControl = train_control, method = "treebag", data = trainingDataset)

My first run has 928 data points

When I key in fit, I get:

Bagged CART 

928 samples 

158 predictors 

  2 classes: 'SELL', 'BUY' 

No pre-processing

Resampling: Cross-Validated (10 fold, repeated 3 times) 

Summary of sample sizes: 836, 835, 835, 835, 835, 835, ... 

Resampling results

  Accuracy  Kappa  Accuracy SD  Kappa SD

  0.694     0.368  0.0544       0.113   

My second run has 1436 data points

When I key in fit, I get:

Bagged CART 

1436 samples

 158 predictors

   2 classes: 'SELL', 'BUY' 

No pre-processing

Resampling: Cross-Validated (10 fold, repeated 3 times) 

Summary of sample sizes: 1292, 1294, 1292, 1293, 1292, 1292, ... 

Resampling results

  Accuracy  Kappa  Accuracy SD  Kappa SD

  0.678     0.34   0.0457       0.0962  

Any thoughts on why a 50% increase in the amount of data would result in a drop in the accuracy of the model?

Is my model not very good? Am I overfitting?

$\endgroup$
  • 2
    $\begingroup$ I'm guessing that you're overfitting. I always assume my accuracy will decrease when I increase my n. $\endgroup$ – Andrew Brēza Jul 13 '17 at 13:37
  • 5
    $\begingroup$ .694 == .678, with a margin of error $\endgroup$ – Hong Ooi Jul 13 '17 at 13:47
  • 5
    $\begingroup$ Take a look at the standard devitions of your accuracy then you see that is was already plausible that your original accuracy was actually lower than what you estimated with the 2nd model where the error margin becomes narrower: 0.694 +/- 0.0544 and 0.678 +/- 0.0457 mean there is basically no (significant) change, as Hong Ooi pointed out. $\endgroup$ – Alex2006 Jul 13 '17 at 13:57
  • 1
    $\begingroup$ @AndrewBrēza Do not assume that. Over fitting is one end; the other end is under feeding your model with not sufficient data. There is an optimum. $\endgroup$ – M-- Jul 13 '17 at 14:15
  • $\begingroup$ Is this two different datasets? or is the smaller dataset a subset of the larger? If so, how was the subset choosen? $\endgroup$ – kjetil b halvorsen Jul 15 '17 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy