I have a large dataset (over 6 million records) and I have aggregated the dataset (
dplyer package) and tested
rpart() on the two datasets. However the results of the two trees are not the same. I wanted to use aggregated data to speed up the calculations – Is this approach incorrect?
Tree: Data no aggregated - no weights
rpart.fraud <- rpart(PERCENTAGE_FRAUD ~ VERTICALNAME + ROUNDED_AMOUNT_TRANSACTIONS_EUR, data = fraud_data)
The tree has 1 node unless I used
control = rpart.control(cp=0.001) and then I have some nodes: 14 nodes
fraud_data_sum <- select(fraud_data,VERTICALNAME,ROUNDED_AMOUNT_TRANSACTIONS_EUR,NUMBER_TRXS_PAID, NUMBER_TRXS_PAID_CBFRAUD) %>% group_by(VERTICALNAME,ROUNDED_AMOUNT_TRANSACTIONS_EUR) %>% summarise(NUMBER_TRXS_PAID=sum(NUMBER_TRXS_PAID), NUMBER_TRXS_PAID_CBFRAUD = sum(NUMBER_TRXS_PAID_CBFRAUD), PERCENTAGE_FRAUD = sum(NUMBER_TRXS_PAID_CBFRAUD)/sum(NUMBER_TRXS_PAID))
Tree: Data aggregated - using
rpart.fraud <- rpart(PERCENTAGE_FRAUD ~ VERTICALNAME + ROUNDED_AMOUNT_TRANSACTIONS_EUR, data = fraud_data_sum, weights = NUMBER_TRXS_PAID)
the tree (using aggregate data and weights) has 30 nodes (using the default control cp).