For a certain problem, where we need to create a score on some malicious activities of user, I have created a custom scoring mechanism. The scores are generated for each user.

The problem statement is

  1. How do I validate that if I rank based on score (higher the score, lower the rank number i.e., one with maximum score is to be given rank 1), there is some statistical significance to it?
  2. How do I validate (quantitatively) the rank of such scores i.e., if I have predicted some user at rank 1, is that correct?

Which are the metrics, which would be effective or verifying both the above statements. I assume for problem statement 1, there should be some Hypothesis Testing needed, but can someone point out towards an appropriate test or some literature which refers to this. Also, how can I detect the appropriate value of sample size I should take for performing the test (for e.g., should I compare top-20 or top-30) and if so, should I compare them as top-k vs k random from all users or top-k vs k-random from top-n (for removing any selection bias possibility).

To clear myself more on Problem 1, I want to check that if I am giving these scores for user and using it to rank these users, does that order actually helps in identifying more malicious user or not? In ideal scenario, the scoring should do it, but I want to validate it through some metric or by statistical testing.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.