Avoiding Data Leakage using Cross Validation during aggregation

Question

This question is very conceptual in nature so if it is too general please close it

I have a fictitious dataset. The data set consists of 10k students across 5 schools It consists of an id, attrib_1, attrib_2, grp_attrib_a, school ID and dep (the dependent variable im interested in predicting) The first two attributes are student scores in something or other. The grp_attrib_a is a score aggregated to school level based on student scores in some test (3 attributes). Its a mean aggregate and is calculated by first adding up three individual scores of each student Then taking the mean of that sum. The mean is then subsequently taken per school

grp_attrib_a = mean(mean(indiv_attrib_a + indiv_attrib_b + indiv_attrib_c), per_school)

My question concerns data leakage in modeling specifically with respect to prediction using cross validation. I have been reading quite bit on it and the general consensus is that any data calculations should be carried out within the folds of the cross validation. I would just like to confirm my understanding if what this means.

If I created the grp_attrib_a prior to splitting my data into train and test sets, I would - I think - be giving my model more information than would be honest, resulting in optimistic model fits. Essentially I would be leaking information from the data I am using to train the model into the test data; peaking if you will.

My thinking to avoid this when dealing with grp_attrib_a would be:

• Split my data into train and test sets
• Take the train data-set and further split that up into 10 folds for cross validation comprising of ten pairs of training and test sets.
• Calculate the grp_attrib_a for each fold. So the formula above is applied to the train segment of my fold and then separately to my test set based in the data in each section
• This would give me two values per fold per school. One grp_attrib_a for each school for each of my ten train data-sets and one grp_attrib_a for each of my test sets. The train and test sets would have slightly different values in the same train-test fold assuming my data gathering is representative
• If I am using some sort of grid search to optimize the model. I would then apply the optimal model to the entire training set. I would need to calculate the grp_attrib_a based on the total training set at this point
• Once I have created a satisfactory model, I would apply the transformation for the grp_attrib_a on the test set again using the same formula and only the attributes in the test set

Are the above steps the correct way to approach this?

Thank you all for your time

Answer 1

This whole question hinges on several considerations:

• is the data possibly clustered by school (answer: yes), and further
• do you want to predict for individual students or for schools, and
• should the predictions work for unknown schools?

If the data is possibly clustered by school, and you want to predict for unknown schools (whether individual students or whole school), you need to do the cross validation splits by school to avoid leakage.

Keep in mind that cross validation simulates what happens during production use of the model:

• If the prediction is for individual students (of unknown school), you cannot use grp_attrib_a for prediction (other than general population [of schools] behavior from the training data - school is then a random factor) for prediction as the test set school behaviour of an unknown school would be unknown (and you could have only one student to predict of that school).
• If you'll always have unknown but complete schools for application use for which you predict sets of individual students, you can calculate their grp_attrib_a and use it (school becomes a fixed factor)
• If you predict school characteristics, again, you can generate the school-specific feature for the test school.