Suppose I have a bunch of people who did a few tests (let's call it tests A). Now I'll do a second test (test B) in order to categorize these people into two categories (that have a pre-defined "true" answer).
Note that score from test A can only predict the signal-to-noise ratio of test B. I.e you can't use score from test A to do the categorization itself.
My aim is to select a group of people that are as easily categorized into correct categories as possible (I.e actual score of test B does not matter), using the score of test A.
Now, I want to examine how test A (which consists of multiple variables A1, A2...) predicts the final outcome.
How could I do that? I thought about creating bunch of subgroups of a fixed sample size randomly and measure the mean of test score A and the percentage I got the categorization right for these subgroups, and then apply regression analysis on that. Would this be a valid approach? If not, what is the correct way of doing this?