# ML for specific classification problem

I have a training dataset for classification problem $X \rightarrow y$. Where $X$ is an $n$th dimensional real vector, $y$ is an integer number in $\{0, 1\}$.

I want to solve the next problem: predict as many lines where $y = 1$ as possible with error level no more than $\alpha = 10\%$. In other words, I want to split dataset ($N$ samples) in two sets: 1) $k$ - samples predicted to be $y = 1$, where $y = 0$ cases less than $k * \alpha$ 2) $N-k$ samples - I don't care about. I need to maximize $k$ with given $\alpha$ .

I have a lot of experience in classification with ML, but this problem is rather unusual to me. Please help me how to formalize this problem for ML. Which and how should I use ML algorithm for the problem? I will be happy to get advice about general idea.

3. Look at $k*\alpha$ on the test data for various scores.