# Analogue of F1 beta score when formulating a classification as a regression

I have a problem of binary classification with class imbalance for the positive label (about 10:1 odds), except that in doing the classification I want to rank the data by confidence of prediction.

In this classification I want the classifier to prefer recall over precision so that it's more aggressive at finding the rare positive examples, so the natural method to combine these is the $F1_\beta$ where say $\beta = 2$.

My question is whether of not the following line of reasoning for formulating an analogous metric for real-valued predictions makes sense, and if it doesn't where and why:

1) To get confidence ratings, formulate the task as regression instead of classification, using say, logistic regression or random forest regression to make predictions. The regressor will learn from examples with responses that are either 1's or 0's, and so predictions that are closer to 1's or 0's are more confident.

2) Now that it's regression, $F1$ doesn't apply. Since I care more about the model being correct with the rare positive examples, I want to give errors for positive examples more weight. Here the error is RMSE.

I suppose I could just use a weighted average of the errors, but alternatively I could keep two different error rates, call them $e_+$ and $e_-$, one for errors on positives and one for errors on negatives. Taking $1-e_+$ and $1-e_-$ gives me an estimate of accuracy for each class.

3) Given that I have two rates, where I want to give one rate more preference, the $\beta$-weighted harmonic mean ( identical in formula to the $F1$ score) seems appropriate.

Does this metric make sense?

Mathematically it would be:

Given $n$ labeled data samples $\mathbf{X} = \{(X_1, l_1),...,(X_n,l_n)\}$ with sets of indices $L_0 = \{i \| l_i = 0\}$ and $L_1 = \{i \| l_i = 1\}$

Learn some real-valued model $M:\mathbb{R}^D \rightarrow [0,1]$ where $D =|X_i|$ $\forall i$

Calculate RMSE error rates $e_0$ and $e_1$ where $e_0 = \sqrt{\frac{\Sigma_{i \in L_0}(M(X_i)-l_i)^2}{\|L_0\|}}$ and $e_1 = \sqrt{\frac{\Sigma_{i \in L_1}(M(X_i)-l_i)^2}{\|L_1\|}}$

Calculate accuracies $a_0 = 1 - e_0$ and $a_1 = 1 - e_1$

For given $\beta$ calculate:

$$Score(M, \mathbf{X}) = (1+\beta^2)\frac{a_0a_1}{\beta^2a_0 + a_1}$$