Model developed by optimizing one measure of performance, but assessed by another

Question

I'm trying to predict a binary outcome - let's call it "Cancer/no cancer" after this example here.

So, $a$ refers to cancer correctly diagnosed as such, and so on. Using logistic regression I can find the parameters $\theta$ that will minimize the cost function:

$J(\theta) = - \frac{1}{m} \displaystyle \sum_{i=1}^m [y^{(i)}\log (h_\theta (x^{(i)})) + (1 - y^{(i)})\log (1 - h_\theta(x^{(i)}))]$,

where $h_\theta (x^{(i)})$ is the model prediction, $y^{(i)})$ is the outcome, and $m$ is the number of training examples. Following Andrew Ng's Coursera subject on machine learning I have no problem writing a MATLAB function that finds appropriate parameters:

function [J, grad] = costFunction(theta, X, y)
%COSTFUNCTION Compute cost and gradient for logistic regression
%   J = COSTFUNCTION(theta, X, y) computes the cost of using theta as the
%   parameter for logistic regression and the gradient of the cost
%   w.r.t. to the parameters.

m = length(y); % number of training examples
J = 0;
grad = zeros(size(theta));

the_hypothesis = sigmoid(X*theta);
first_part = y'*log(the_hypothesis);
second_part = (1-y)'*log(1-the_hypothesis);
J = -(1 / m) * sum(first_part + second_part);

grad = (1/m) * ((the_hypothesis-y)'*X);

end

I'm now trying to apply this stuff to my own research area. However, ultimately my models won't be judged in relation to this cost function. Instead they'll be judged in relation to a "skill score" of a type traditionally applied in my research area. Here are some examples of skill scores:

What I really want is to find parameters that will maximize whatever skill score I need to deal with. So let's say I get told "you're going to be judged by the Peirce Skill Score", then I'd just edit my cost function and run my model, and it'd produce parameters that would maximze that particular skill score. However, it wasn't apparent to me how I could do that.

Daydreaming about this problem I came up with a lame solution. I was imagining some set-up where I split up the data into training/cross-validation/test sets, and then do the following:

1. Training phase. We find the best $\theta$ values according to the logistic regression cost function.
2. Cross-validation phase. We apply various threshold values to probabilities predicted by those $\theta$s and see how this affects performance on the relevant skill score. We might, for example, find that skill score performance is best when model-forecasted probabilities $\geq 0.45 $ are converted to $1$, while those $< 0.45$ are converted to $0$.
3. Test phase. We apply the $\theta$ and threshold values found in the previous two phases, and apply them to the test data, hopefully achieving good skill score performance.

However, what would be really cool if there was some way to just edit my cost function, and make it directly maximise the skill score I am assigned. Ideally I'd just edit my logistic regression cost function with a new cost function that matches the skill score I'm using, and which doesn't run into any computation problems (e.g. local optima).

EDIT: Thanks to @deltaiv's answer, I realise I misspoke here - I understand that I'd need to change the model as well as the cost function. Is there some model and cost function that 'matches' a skill score I might be interested in?

Is it realistic to think there's a model (and associated cost function) that would come up with parameters that directly maximize one of those skill scores I'm interested in? So maybe this hypothetical model would produce probabilities and a threshold, or maybe it would produce probabilities that optimize the skill score when some standard threshold like 0.5 is applied to it, or maybe it would just produce binary forecasts of 0 and 1. I'm completely open to the possibility that such a model is not feasible - but why is it not feasible?

If such a model is unfeasible (or wouldn't work well), what should I do?

Answer 1

I think there is a little misunderstanding here: logistic regression (LR), by itself, is not exactly a classifier, but a model for the conditional probability $P(Y=k|X=x)$, where $k=0,1$ maps to your class labels {No Cancer, Cancer}. In ML terminology, I think you would call it a discriminative model, since it models the conditional probability, as opposed to a generative model, such as for example LDA, which models the joint probability $P(Y=k,X=x)$. LR only becomes a classifier when you couple it with a decision rule, which usually is something like

if (phat > alpha) yhat = 1
else yhat = 0 

And then you set $\alpha$ by cross-validation to optimize your accuracy metric, which is what you're doing.

---

The cost function is already the "right one" for LR, because it's just maximum likelihood estimation, under the assumption that the training data is a random sample. To use another cost function would mean that either you're not estimating the $\theta$ by using MLE, or that you're using another model for the conditional probability of $Y$ given $X$ . As a matter of fact, (regularized) hinge loss, which is a different cost function, is the correct cost function for SVM, which is a different model with respect to LR. Thus you're left with basically two choices:

1. you keep using your cost function and your procedure to select $\alpha$. At most, you could add ($L_2-$ or $L_1-$) regularization to your cost function and see if that improves test error for the CV-selected value of the regularization parameter $\lambda$. I would expect this to make a improvement on your current procedure only if either classes are close to linearly separable, or if you have a number of predictors which is close to or larger than the number of training points, i.e., either when the estimated parameters of nonregularized model have large variance, or when the nonregularized model cannot be fitted at all. BTW, $L_2- $regularization corresponds to MAP estimate (instead than MLE) with a Gaussian uncorrelated prior for the parameters, while $L_1- $regularization corresponds to a Laplace uncorrelated prior.
2. you add new models to your pool of models. For example, you try fitting also LDA or SVM, and compare them to LR using cross-validation. These are different models, and using MLE you arrive at objective functions which are different from the log-loss you were using. However, you do it in a principled way. If you used different cost functions to estimate the LR model, you wouldn't be doing MLE anymore.

EDIT the OP was modified as a consequence of my answer, and it's now different. I try to take into account the new requests, but I don't think estimating the model parameters based on skill scores is smart. I'd still go with trying to estimate the conditional or joint probability as accurately as possible, and only then trying to find thresholds (by cross-validation) which optimize skill scores.

Let $u(x)$ denote the step function:

$$\begin{align} u(x) &= \left\{ \begin{array}{l} 1 \quad x\ge0 \\ 0 \quad x<0 \end{array} \right. \end{align}$$

If $\mathbf{X}=(1,x_1,\dots,x_d)$ is a vector of predictors (note that, with an abuse of notation, I included the constant $1$ in the vector of predictors) and $\boldsymbol{\beta}=(\beta_0,\dots,\beta_d)$ a vector of parameters, $u(\boldsymbol{\beta}^T\mathbf{X})$ is a linear classifier. We could find the $\boldsymbol{\beta}$ which minimize, for example, the FAR on the training set $\{(\mathbf{x}_i,y_i)\}_{i=1}^N$, i.e.

$$\min_{\boldsymbol{\beta}}\sum_{i=1}^N (1-y_i)u(\boldsymbol{\beta}^T\mathbf{x}_i)$$

Now, this objective function is obviously discontinuous, so optimization will be an hassle. Not only that, but there are infinitely many solutions - just consider that, whatever the training set, $\boldsymbol{\beta}=(\beta_0,0,\dots,0)$ minimizes FAR as long as $\beta_0<0$, because then $u(\boldsymbol{\beta}^T\mathbf{X})=u(\beta_0)=0$. And I haven't even touched the issue of the properties of this estimator: with MLE, we know that as long as the model is not misspecified, the estimator is consistent, asymptotically normal and efficient. But with this estimator? I have no idea. Still, you could try using it, compare to LR or SVM, and see what you get in terms of accuracy on the test set.

Of course, you could repeat similar considerations for other skill score-based loss functions: I don't think you would get much better results. In particular, I expect the issue of non-uniqueness to reappear. The reason why I expect that, is that skill scores are all discrete loss functions, i.e., they all take values in a finite set of size $N$. The parameter vector $\boldsymbol{\beta}$ instead is a continous variable: intuitively, if there's a $\boldsymbol{\beta}^*$ which minimizes the skill score, then there are infinitely many (for example, think of the values of $x$ which minimize $u(x)$). Thus, even if you consider a skill score for which there is not such a trivial solution as there is for FAR, you will still have to deal with discontinuity of the loss function, and non-uniqueness of the parameters estimate. You could try to fix the last issue with regularization.