Assume the following most simple setup:

I want to perform a binary classification based on a single continuous variable $X$. The model is then a threshold $x$ so that the class of a data point $d$ is $1$ if $X_d >= x$, and $0$ otherwise.

I'm doing an exhaustive search for the optimal threshold $x$ by simply using each value of $X$ occurring in my training set. The optimal threshold is then identified by maximizing a specific performance metric like PPV, for instance.

Using this approach, my threshold will always be a value that I already observed in the training data.

Now I'm wondering if there was any benefit in not placing the thresholds onto the training data, but e.g. half way in between any two adjacent observations? I can also think of even more sophisticated strategies like weighted averages of adjacent observations with weights based on the distribution of $X$ in the whole training set.

The question is if such strategies make my model any better in terms of a more "unbiased" model when it is applied to new data?

Could you point me to any statistical theory or comparisons of different strategies?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.