This question is very conceptual in nature so if it is too general please close it
I have a fictitious dataset. The data set consists of 10k students across 5 schools It consists of an id, attrib_1, attrib_2, grp_attrib_a, school ID and dep (the dependent variable im interested in predicting) The first two attributes are student scores in something or other. The grp_attrib_a is a score aggregated to school level based on student scores in some test (3 attributes). Its a mean aggregate and is calculated by first adding up three individual scores of each student Then taking the mean of that sum. The mean is then subsequently taken per school
grp_attrib_a = mean(mean(indiv_attrib_a + indiv_attrib_b + indiv_attrib_c), per_school)
My question concerns data leakage in modeling specifically with respect to prediction using cross validation. I have been reading quite bit on it and the general consensus is that any data calculations should be carried out within the folds of the cross validation. I would just like to confirm my understanding if what this means.
If I created the
grp_attrib_a prior to splitting my data into train and test sets, I would - I think - be giving my model more information than would be honest, resulting in optimistic model fits. Essentially I would be leaking information from the data I am using to train the model into the test data; peaking if you will.
My thinking to avoid this when dealing with
grp_attrib_a would be:
- Split my data into train and test sets
- Take the train data-set and further split that up into 10 folds for cross validation comprising of ten pairs of training and test sets.
- Calculate the
grp_attrib_afor each fold. So the formula above is applied to the train segment of my fold and then separately to my test set based in the data in each section
- This would give me two values per fold per school. One
grp_attrib_afor each school for each of my ten train data-sets and one
grp_attrib_afor each of my test sets. The train and test sets would have slightly different values in the same train-test fold assuming my data gathering is representative
- If I am using some sort of grid search to optimize the model. I would then apply the optimal model to the entire training set. I would need to calculate the
grp_attrib_abased on the total training set at this point
- Once I have created a satisfactory model, I would apply the transformation for the
grp_attrib_aon the test set again using the same formula and only the attributes in the test set
Are the above steps the correct way to approach this?
Thank you all for your time