I'm new here. My background is in signal processing including stochastic signals, so while I have a decent foundation in probability, my stats knowledge is weak.
I am evaluating the performance of an object recognition algorithm. I have about 30 test cases, each with a large number of objects (100k+ each, about 5 million total), and I am calculating the precision/recall and F1 score for each test case.
In addition to the performance metrics, I would like to figure out where the algorithm needs improvement. I have about 10 characteristics about each object (e.g. size, color), and I would like to determine which of these characteristics has a significant impact on the object recognition performance. For example, one data set might look like:
Object Color Size Intensity Detection 1 Red 10 0.5 True 2 Green 10 0.8 Miss 3 Red 20 0.6 False Alarm 4 Red 10 0.7 Miss ... 150,000 Yellow 15 0.8 True ------------------------------------------ F1 Score: 0.68 Precision: 0.6 Recall: 0.8
where the "detected" category defines whether the object is a correct (true) detection, a miss, or a false alarm.
My objective now is to determine if (for example) color has a significant effect on the detector performance.
What I have tried
The first thing I tried is multinomial logistic regression with the three "detection" categories as an outcome. I also tried grouping the miss and false alarm outcomes into good vs. bad and doing straight logistic regression. However, I'm not really sure how to interpret the results - I've read that the p-value is somewhat meaningless with my large sample size, and it's always really small no matter what predictor variables I use. I'm also not sure if it's important to balance my categories, so I've tried subsampling to get balanced sets, but at this point I'm just trying random things without really understanding the results.
A different approach that I have been considering is calculating the F1 score for different groups (e.g. all the reds, all with a size of 10, etc) and then trying to assess if the results are significantly different from random groupings. However, I don't really know how to assess this significance. The F1 score doesn't look to be normally distributed, so I don't think ANOVA would be the way to go.
Finally, I'm not confident that my variables are uncorrelated, so that just throws another wrench in the problem. Preferably I would like to take interaction into account in order to know that (for example) red objects under a certain size threshold are the biggest challenge for the algorithm.
How would you go about determining which object characteristics pose a problem for my object recognition algorithm? Am I on the right track with either of my proposed approaches? Do I even need to find statistical significance, or should I just look at the outcomes of all the different characteristic combinations? I appreciate any advice you might.