2
$\begingroup$

I have 200 million features and 1 label (features and label have about 1 million observations). Features are binary, and each has an unknown but different amounts of True and False. Label is also binary, and has a distribution of 25% True, 75% False.

Testing all features against the labels, I have 3 values per feature how it relates to the label:

True positives count (Feature=True & Label=True) - mean 65295, std 61287

False positives count (Feature=True & Label=False) - mean 192631, std 185827

Accuracy (true_positives/(true_positives+false_positives)) - mean 0.264, std 0.047

My goal is to select the statistically significant features by accuracy. My current idea is to find the minimum true positives for each accuracy. Because there are 200 million features, some have an accuracy of 1 (100%) but only 3 true positives (and 0 false positives) and are not statistically significant. Other features have an accuracy of 0.51 (51%) with 50000 true positives and 48000 false positives, and are also unlikely statistically significant.

What is the correct way to capture the relationship between minimum true positives required for statistical significance of a given accuracy value?

Improve this question
$\endgroup$
6
  • $\begingroup$ Example: there are 750000 features with accuracy > 0.6 (average accuracy being around 0.26). However the majority of these selected features is unlikely statistically significant. $\endgroup$
    – Franc Weser
    Commented Feb 7, 2020 at 14:05
  • 2
    $\begingroup$ Why do you want to use accuracy? See stats.stackexchange.com/questions/359909/…, stats.stackexchange.com/questions/208529/…, stats.stackexchange.com/questions/145875/… $\endgroup$
    – kjetil b halvorsen ♦
    Commented Feb 7, 2020 at 15:32
  • $\begingroup$ It's just one idea, what the best alternative for the provided case? For my use case true positives and false positives are most important, true negatives and false negatives don't matter at all. $\endgroup$
    – Franc Weser
    Commented Feb 7, 2020 at 15:35
  • 1
    $\begingroup$ Can you please give the context? What is your goal? To what use will you put the predictions? What are all those variables, how are they measured? $\endgroup$
    – kjetil b halvorsen ♦
    Commented Feb 7, 2020 at 16:04
  • $\begingroup$ Goal is to pick the best features out of the 200 million by a metric. Basically feature selection for further processing. What is important for me right now is not the metric that is used (be it accuracy or not), but rather to try to find out how to conduct the hypothesis test to distinguish the feature that are statistically significant from the ones that are not. $\endgroup$
    – Franc Weser
    Commented Feb 7, 2020 at 16:55

0

Reset to default