paired t test to check if features in test and holdout datasets are from same distribution The machine learning model we are developing is having different resuls on test and holdout and we want to check if the test and holdout are from the same distributions or if there is a feature drift between the two datasets.
For doing that i am doing the following :

*

*I calculate the summary statistics and calculate mean for all the features in the 2 datasets

*By taking the mean values for every feature, i now have two columns having the feature means from 2 dataset

*I run a paired sample t test on those two columns
I calculate the t score on google sheets with the T.test function (2 tailed test and a paired test) and see the score to 0.42

Assuming that this statistical methodology is correct, how should the tscore of 0.42 interpreted to check if the distributions are same or different?
 A: If I've understood this correctly, you've got a test set $T$ and hold out set $H$, and each has features $\{x_1, ..., x_n\}$. It should be ok to compare $x_i$ from $T$ to $x_i$ from $H$, but it does not make sense to collate the means of all $x_i$s and then run a t-test on all those means. At your request, I'll briefly explain why. Let's imagine the following datasets:
Holdout:

Test:

I hope it's quite plain to see that if you take the means of x1, x2 we may get something like Holdout: [25, 2500], and Test: [2500, 25]. That is, the groups are incredibly different. However, if you run a t-test to compare [25, 2500] and [2500, 25] you will find absolutely no difference, it makes no sense to compare these two vectors of means of means.
Instead, you should run a seperate t-test for each of the $n$ features, and if $n$ is quite large you will need to seriously consider issues with multiple comparison testing.
I believe that the Google Sheets T.test function outputs the "p-value" of the t-test; This tells you the probability of getting the difference in means between feature $i$ of $T$ and $H$ under the assumption that there was actually no difference. We conventionally say that if the p-value is $\lt 0.05$, the difference is significant and conclude that there is sufficient evidence to say that the samples came from different populations. A p-value of 0.42 implies that we would see this sample difference that we found (or greater) 42% of the time when there was actually no population difference. This is quite likely if there was no difference, so we don't have evidence to suggest that there is a difference. Of course with my above suggestions, you should not consider that 0.42, but instead rerun with $n$ independent t-tests, (there will be some dependence but it can likely be ignored for your purposes).
Lastly, addressing your last comment, the number of rows is not relevant here, and you have no reason not to use all of them. The problems that arise with multiple comparison testing come from how many columns you have in the dataset.
