# Why Does My Model Accuracy Go Down With More Data?

I am using R and the caret package.

My code is very straight forward.

set.seed(2112)
train_control <- trainControl(method="repeatedcv", number = 10, repeats = 3)
fit <- train(eval ~ ., trControl = train_control, method = "treebag", data = trainingDataset)


My first run has 928 data points

When I key in fit, I get:

Bagged CART

928 samples

158 predictors

No pre-processing

Resampling: Cross-Validated (10 fold, repeated 3 times)

Summary of sample sizes: 836, 835, 835, 835, 835, 835, ...

Resampling results

Accuracy  Kappa  Accuracy SD  Kappa SD

0.694     0.368  0.0544       0.113


My second run has 1436 data points

When I key in fit, I get:

Bagged CART

1436 samples

158 predictors

No pre-processing

Resampling: Cross-Validated (10 fold, repeated 3 times)

Summary of sample sizes: 1292, 1294, 1292, 1293, 1292, 1292, ...

Resampling results

Accuracy  Kappa  Accuracy SD  Kappa SD

0.678     0.34   0.0457       0.0962


Any thoughts on why a 50% increase in the amount of data would result in a drop in the accuracy of the model?

Is my model not very good? Am I overfitting?

• I'm guessing that you're overfitting. I always assume my accuracy will decrease when I increase my n. – Andrew Brēza Jul 13 '17 at 13:37
• .694 == .678, with a margin of error – Hong Ooi Jul 13 '17 at 13:47
• Take a look at the standard devitions of your accuracy then you see that is was already plausible that your original accuracy was actually lower than what you estimated with the 2nd model where the error margin becomes narrower: 0.694 +/- 0.0544 and 0.678 +/- 0.0457 mean there is basically no (significant) change, as Hong Ooi pointed out. – Alex2006 Jul 13 '17 at 13:57
• @AndrewBrēza Do not assume that. Over fitting is one end; the other end is under feeding your model with not sufficient data. There is an optimum. – M-- Jul 13 '17 at 14:15
• Is this two different datasets? or is the smaller dataset a subset of the larger? If so, how was the subset choosen? – kjetil b halvorsen Jul 15 '17 at 13:53