1
$\begingroup$

I have a large dataset (over 6 million records) and I have aggregated the dataset (dplyer package) and tested rpart() on the two datasets. However the results of the two trees are not the same. I wanted to use aggregated data to speed up the calculations – Is this approach incorrect?

Tree: Data no aggregated - no weights

rpart.fraud <- rpart(PERCENTAGE_FRAUD ~ VERTICALNAME + ROUNDED_AMOUNT_TRANSACTIONS_EUR,
                     data = fraud_data)

The tree has 1 node unless I used control = rpart.control(cp=0.001) and then I have some nodes: 14 nodes

fraud_data_sum <- select(fraud_data,VERTICALNAME,ROUNDED_AMOUNT_TRANSACTIONS_EUR,NUMBER_TRXS_PAID, NUMBER_TRXS_PAID_CBFRAUD) %>%
            group_by(VERTICALNAME,ROUNDED_AMOUNT_TRANSACTIONS_EUR) %>%
            summarise(NUMBER_TRXS_PAID=sum(NUMBER_TRXS_PAID),
                      NUMBER_TRXS_PAID_CBFRAUD = sum(NUMBER_TRXS_PAID_CBFRAUD),
                      PERCENTAGE_FRAUD = sum(NUMBER_TRXS_PAID_CBFRAUD)/sum(NUMBER_TRXS_PAID))

Tree: Data aggregated - using weights argument

rpart.fraud <- rpart(PERCENTAGE_FRAUD ~ VERTICALNAME + ROUNDED_AMOUNT_TRANSACTIONS_EUR,
                     data = fraud_data_sum, weights = NUMBER_TRXS_PAID)

the tree (using aggregate data and weights) has 30 nodes (using the default control cp).

$\endgroup$
2
$\begingroup$

I did try out both above mentioned methods once in the past. Both methods are correct. However, you will get different trees because the minsplit and minbucket criteria doesn't take weights of the records into consideration. minsplit and minbucket is checked on the number of records.

Hope that answers your question.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.