0
$\begingroup$

Suppose I have a dataset (a "sample") that contains a person's height and whether or not they play basketball. I illustrate this below using the R programming language (I mixed three datasets together to add some variability to the data):

    library(dplyr)
    library(plotly)
    
    set.seed(123)
    
    height <- rnorm(1000,210,5)
    status <- c("basketball", "not_basketball")
    basketball_status <- as.character(sample(status, 1000, 
                        replace=TRUE, prob=c(0.80, 0.20)))
    data_1 <- data.frame(height, basketball_status)
    
    height <- rnorm(1000,190,1)
    status <- c("basketball", "not_basketball")
    basketball_status <- as.character(sample(status, 1000, 
                 replace=TRUE, prob=c(0.8, 0.2)))
    data_2 <- data.frame(height, basketball_status)
        
    height <- rnorm(1000,170,5)
    status <- c("basketball", "not_basketball")
    basketball_status <- as.character(sample(status, 1000, 
               replace=TRUE, prob=c(0.20, 0.80)))
    data_3 <- data.frame(height, basketball_status)    
    
    my_data <- rbind(data_1, data_2, data_3)

    head(my_data)
        height basketball_status
    1 206.7631        basketball
    2 211.4847        basketball
    3 219.4436    not_basketball
    4 200.9625        basketball
    5 213.9713    not_basketball
    6 201.7033        basketball
    
I showed some descriptive plots below:

    plot_ly(type = "scatter", mode = "markers", data = my_data, 
           x = ~height, y = " ", color = ~basketball_status )

enter image description here

Additional Plot:

    library(ggplot2)
    
    ggplot(my_data, aes(x=my_data$height, 
          group=my_data$basketball_status, fill = 
              my_data$basketball_status)) + 
        geom_density(adjust=2, alpha = 0.5) + xlab("Height") +
        ylab("Density") + ggtitle("Distribution of Heights for 
               Basketball Players vs. Non Basketball Players")

enter image description here

Suppose I have theory that says : "If you were to rank the entire population of people (e.g. USA) by height, ** and take the top 20% of these people : 65% of these (top 20%) people play basketball**".

Normally, I have seen these kinds of questions being answered using traditional statistical methods (e.g. hypothesis testing) that often rely on the distribution of the data. However, I had the following idea:

Question: Given this data that I have (assuming that it is "well representative" ), could I use a "cross-validation style approach" to test this idea that : in the top 20% of the population (by height - i.e. over the 80th percentile), at least (minimum) 60% of these people play basketball?

This would involve taking random samples (e.g. test, train) from the data I have and calculating the 80th percentile for the training data - and then seeing how accurately this 80th percentile predicts whether a person plays basketball from the test data. If I repeatedly take many such samples (i.e. many iterations) and take the average accuracy of how well the 80th percentile from the training data characterizes the test data for all iterations - is this a statistically valid approach?

I have illustrated this approach below:

    results <- list()
    for (i in 1:100) {
    
    my_data$row = 1:nrow(my_data)
train_i <- sample_frac(my_data, 0.7)
sid <- train_i$row
     
      quantiles = data.frame( train_i %>% summarise (quant_1 = 
           quantile(height, 0.80)))
          
      test_i$basketball_pred = as.character(ifelse(test_i$height 
        > quantiles$quant_1 , "basketball",   "not_basketball" ))
     
      test_i$accuracy = ifelse(test_i$basketball_pred == 
              test_i$basketball_status, 1, 0)
          
      results_tmp = data.frame(test_i %>%
                              
                               dplyr::summarize(Mean = 
          mean(accuracy, na.rm=TRUE)))
     
      results_tmp$iteration = i
     
      results_tmp$total_mean = mean(test_i$accuracy)
      results[[i]] <- results_tmp
    }
    
    results
    
    results_df <- do.call(rbind.data.frame, results)

A table containing the average results of each iteration can be seen below:

        head(results_df)
    
           Mean iteration total_mean
    1 0.6433333         1  0.6433333
    2 0.6433333         2  0.6433333
    3 0.6388889         3  0.6388889
    4 0.6388889         4  0.6388889
    5 0.6411111         5  0.6411111
    6 0.6511111         6  0.6511111

And the final result (i.e. the average of all iterations) can be seen here (64%):

    mean(results_df$total_mean)

    [1] 0.6449111

Extra: When we look the top 20% of the full data, Roughly 78% of people in the top 20% of the data play basketball (2361 / 2361 + 635) :

    sort_data <- my_data[order(-my_data$height), ]
    top_20 <- sort_data[ 1:nrow(my_data) *0.2,]
    
    table(top_20$basketball_status)
    
    
  

     basketball not_basketball 
          2361            635 

And the actual 80th Percentile of the full data is 209 cm (last row):

    tail(sort_data)
    
            height basketball_status
    603   209.0169        basketball
    603.1 209.0169        basketball
    603.2 209.0169        basketball
    603.3 209.0169        basketball
    603.4 209.0169        basketball
    880   209.0094        basketball

Question: Is the statistical methodology presented in this question reasonable? Based on these simulations (and assuming that the data we have is well-representative of the true population) - can we conclude that at least 64% of people that are taller than 209 cm (the 80th percentile of the full data) play basketball? In my opinion, the above cross validation procedure is somewhat able to recreate a variety of possible conditions in which you observe people of different heights and test your theory even when you observe data that is unfavorable to your theory (e.g. test samples that contain higher proportions of tall people that don't play basketball).

The following graphs can be used to build confidence intervals of your estimate :

    par(mfrow=c(1,2))
     hist(results_df$total_mean, breaks = 90)
    plot(density(results_df$total_mean))

enter image description here

And here are the same charts when the cross validation procedure is repeated 10,000 times:

enter image description here

To wrap everything up - technically, a "mean" can be considered as a statistical model: a single parameter statistical model. On the other hand, a neural network is a multi-parameter model. For the same data, the "mean" and a trained neural network estimate different things about this same data. In theory, the "80th percentile" (e.g. of a variable) can also be considered as a "model" - and regarding the issue of performance metrics and generalization, I do not see why this "model" (i.e. the 80th percentile) should be treated any differently than how we validate common statistical models like decision trees and regression models. Thus, I attempted to show (or disprove) that in general, the generalizability and the information provided to us from the "80th percentile" of a particular dataset - and to what extent do we have reason to believe that this "80th percentile" should describe "unseen data", as consistently as it described the "seen data".

Can someone please tell me if what I am proposing is statistically correct? I had initially thought of using hypothesis testing, but I was not sure how well standard hypothesis testing performs on non-normally distributed data. I am aware of non-parametric hypothesis testing methods, but these in general provide less information (and apparently wider bounds) compared to o parametric hypothesis tests.

$\endgroup$

1 Answer 1

1
$\begingroup$

Percentiling is meaningless in this context. That involves incomplete conditioning, e.g., computing things like P(Y=1 | X > c) instead of P(Y=1 | X = c), the latter being proper complete conditioning, i.e., full use of information. The task instead is to estimate P(Y=1 | X=c) as a function of height c. This is readily accomplished by fitting a binary logistic regression model with height X represented by a regression spline so as to relax the linearity assumption. See for example the RMS course notes.

$\endgroup$
3
  • $\begingroup$ Hello Dr. Harrell! Thank you for your reply! I agree with your answer - in most cases, a binary logistic regression would be more suitable for this type of question. However, in my question, I am specifically interested in answering the following question: "After having observed some data, is it reasonable to conclude that AT LEAST 65% of people taller than 210 cm play basketball?". Therefore, I am trying to answer "P(Y=1 | X > c)" instead of "P(Y=1 | X=c) ". $\endgroup$
    – stats_noob
    Commented Oct 16, 2021 at 20:42
  • $\begingroup$ I understand that this is a strange question - someone could just as easily ask "After having observed some data, is it reasonable to conclude that AT LEAST 21% of people taller than 202 cm play basketball?" . But in my case, I am specifically interested in knowing if AT LEAST 65% of people taller than 210 cm play basketball. I am not looking to make a regression model, but rather to ascertain a simple "fact" (at least 65% of people taller than 210 cm play basketball). If you have time, could you please explain why in this example, incomplete conditioning is meaningless? Thank you so much! $\endgroup$
    – stats_noob
    Commented Oct 16, 2021 at 20:46
  • $\begingroup$ These questions are not very sensible. Ask instead what is the probability that someone who is 215cm tall will play basketball. Then do the same for 211cm, 216, 220, 221, 190, ... A person in the population is not just "at least 210cm. They are exactly some height. $\endgroup$ Commented Oct 17, 2021 at 12:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.