# Replacing Hypothesis Testing with Cross Validation: Testing the Accuracy of Custom Estimators

Suppose I have a dataset (a "sample") that contains a person's height and whether or not they play basketball. I illustrate this below using the R programming language (I mixed three datasets together to add some variability to the data):

    library(dplyr)
library(plotly)

set.seed(123)

height <- rnorm(1000,210,5)
basketball_status <- as.character(sample(status, 1000,
replace=TRUE, prob=c(0.80, 0.20)))
data_1 <- data.frame(height, basketball_status)

height <- rnorm(1000,190,1)
basketball_status <- as.character(sample(status, 1000,
replace=TRUE, prob=c(0.8, 0.2)))
data_2 <- data.frame(height, basketball_status)

height <- rnorm(1000,170,5)
basketball_status <- as.character(sample(status, 1000,
replace=TRUE, prob=c(0.20, 0.80)))
data_3 <- data.frame(height, basketball_status)

my_data <- rbind(data_1, data_2, data_3)

I showed some descriptive plots below:

plot_ly(type = "scatter", mode = "markers", data = my_data,
x = ~height, y = " ", color = ~basketball_status )


    library(ggplot2)

ggplot(my_data, aes(x=my_data$$height, group=my_data$$basketball_status, fill =
my_data$basketball_status)) + geom_density(adjust=2, alpha = 0.5) + xlab("Height") + ylab("Density") + ggtitle("Distribution of Heights for Basketball Players vs. Non Basketball Players")  Suppose I have theory that says : "If you were to rank the entire population of people (e.g. USA) by height, ** and take the top 20% of these people : 65% of these (top 20%) people play basketball**". Normally, I have seen these kinds of questions being answered using traditional statistical methods (e.g. hypothesis testing) that often rely on the distribution of the data. However, I had the following idea: Question: Given this data that I have (assuming that it is "well representative" ), could I use a "cross-validation style approach" to test this idea that : in the top 20% of the population (by height - i.e. over the 80th percentile), at least (minimum) 60% of these people play basketball? This would involve taking random samples (e.g. test, train) from the data I have and calculating the 80th percentile for the training data - and then seeing how accurately this 80th percentile predicts whether a person plays basketball from the test data. If I repeatedly take many such samples (i.e. many iterations) and take the average accuracy of how well the 80th percentile from the training data characterizes the test data for all iterations - is this a statistically valid approach? I have illustrated this approach below:  results <- list() for (i in 1:100) { my_data$$row = 1:nrow(my_data) train_i <- sample_frac(my_data, 0.7) sid <- train_i$$row quantiles = data.frame( train_i %>% summarise (quant_1 = quantile(height, 0.80))) test_i$$basketball_pred = as.character(ifelse(test_i$$height > quantiles$quant_1 , "basketball",   "not_basketball" ))

test_i$$accuracy = ifelse(test_i$$basketball_pred ==
test_i$basketball_status, 1, 0) results_tmp = data.frame(test_i %>% dplyr::summarize(Mean = mean(accuracy, na.rm=TRUE))) results_tmp$iteration = i

results_tmp$$total_mean = mean(test_i$$accuracy)
results[[i]] <- results_tmp
}

results

results_df <- do.call(rbind.data.frame, results)


A table containing the average results of each iteration can be seen below:

        head(results_df)

Mean iteration total_mean
1 0.6433333         1  0.6433333
2 0.6433333         2  0.6433333
3 0.6388889         3  0.6388889
4 0.6388889         4  0.6388889
5 0.6411111         5  0.6411111
6 0.6511111         6  0.6511111


And the final result (i.e. the average of all iterations) can be seen here (64%):

    mean(results_df$total_mean) [1] 0.6449111  Extra: When we look the top 20% of the full data, Roughly 78% of people in the top 20% of the data play basketball (2361 / 2361 + 635) :  sort_data <- my_data[order(-my_data$height), ]
top_20 <- sort_data[ 1:nrow(my_data) *0.2,]

table(top_20$basketball_status) basketball not_basketball 2361 635  And the actual 80th Percentile of the full data is 209 cm (last row):  tail(sort_data) height basketball_status 603 209.0169 basketball 603.1 209.0169 basketball 603.2 209.0169 basketball 603.3 209.0169 basketball 603.4 209.0169 basketball 880 209.0094 basketball  Question: Is the statistical methodology presented in this question reasonable? Based on these simulations (and assuming that the data we have is well-representative of the true population) - can we conclude that at least 64% of people that are taller than 209 cm (the 80th percentile of the full data) play basketball? In my opinion, the above cross validation procedure is somewhat able to recreate a variety of possible conditions in which you observe people of different heights and test your theory even when you observe data that is unfavorable to your theory (e.g. test samples that contain higher proportions of tall people that don't play basketball). The following graphs can be used to build confidence intervals of your estimate :  par(mfrow=c(1,2)) hist(results_df$total_mean, breaks = 90)
plot(density(results_df\$total_mean))


And here are the same charts when the cross validation procedure is repeated 10,000 times:

To wrap everything up - technically, a "mean" can be considered as a statistical model: a single parameter statistical model. On the other hand, a neural network is a multi-parameter model. For the same data, the "mean" and a trained neural network estimate different things about this same data. In theory, the "80th percentile" (e.g. of a variable) can also be considered as a "model" - and regarding the issue of performance metrics and generalization, I do not see why this "model" (i.e. the 80th percentile) should be treated any differently than how we validate common statistical models like decision trees and regression models. Thus, I attempted to show (or disprove) that in general, the generalizability and the information provided to us from the "80th percentile" of a particular dataset - and to what extent do we have reason to believe that this "80th percentile" should describe "unseen data", as consistently as it described the "seen data".

Can someone please tell me if what I am proposing is statistically correct? I had initially thought of using hypothesis testing, but I was not sure how well standard hypothesis testing performs on non-normally distributed data. I am aware of non-parametric hypothesis testing methods, but these in general provide less information (and apparently wider bounds) compared to o parametric hypothesis tests.