I have a hybrid classification/regression problem.The predicted value can be assumed to be centred around 0. I want to penalize the predictor more, if the predicted value and actual value have opposite signs. The reason is some kind of demand prediction,for the same degree of offset from the actual demand, its better to predict the trend of demand correctly, as that reduces the loss. One idea I have is this

$$ \sum_{i=1}^n w_i \exp{\bigl[-y_i\widehat{g}(x_i)/\alpha\bigr]} \cdot \left[y_i - \widehat{g}(x_i)\right]^2 $$ where $$ \widehat{g}(x_i) = \beta^Tx_i $$ is just the linear predictor. We can think of $w_i$ as weights (e.g. for sampling), probably equalling $1/n$. My questions are: 1) Does someone know a similar problem in academic literature or actual use, (basically a hybrid mix of classification and regression, so that wrong sign error have higher weight).What kind of error function should I use, since the once I seemed to have chosen has bad convergence properties.

2) How do i optimize this kind of function. I havent checked but it probably is convex.

3) Does python or R have some library or package with this algorithm or, some kind of automatic minimzation algo.

4) How do i scale $\alpha$, as clearly its value needs to be determined

Finally I hope to get this done on large dataset so probably some efficient way to evaluate the model will be great

As far as why asymmetric treatment is done, its quite simply that even if you are wrong $y_i\ne\widehat{g}(x_i)$ , the trend is in the correct direction. I'd think that $\alpha$ is s.t. $\exp{\bigl[-y_i\widehat{g}(x_i)/\alpha\bigr]} \approx 1/n$ should be used, so that some points are not overly weighted, but want some ideas for that.

Update1: (partly to @StasK) I tried going by iterative LS, and seem to run into convergence issues even on a toy example. Is there some known things about ITLS convergence, or is it my issue. One thing which I do more is to snap some of the $\beta$ coefficients to 0, or maybe there is some other thing which i did wrong

Update2: On 3 different examples one quite realistic, I found that in the IRLS the β tends to get close to 2 different points and keeps jumping between them, likely somethign the weights keep changing drastically across those two. I tried to trim the exp number to a max value, and even then this seems to happen.So I am looking to redesign my loss now.

  • $\begingroup$ I would re-write it so that $\alpha=0$ corresponds to OLS/weighted LS. The way you've written it, OLS is $\alpha=\infty$ which isn't convenient at all. $\endgroup$
    – StasK
    Commented Jan 29, 2015 at 15:38

1 Answer 1


There is no shortage of optimizers in R. I will let a more knowledgeable R user recommend the most appropriate one(s).

A cheap solution is to iteratively reweight your data, treating $r_i^{(k)}=\exp\bigl[-\alpha^{(k)} y_i\hat{g}^{(k-1)}(x_i)\bigr] $ as a fixed weight within iteration $k$ (I took a liberty to move $\alpha$ from the denominator to the numerator), where $\hat{g}^{(k-1)}=x^T\hat\beta^{(k-1)}$ is the result of the weighted least squares from the previous iteration. So

  1. Set iteration counter $k=0$, $\alpha^{(k)}=0$, run OLS to get $\hat\beta^{(0)} \equiv \hat\beta_{\rm WLS}$
  2. Update $k \leftarrow k+1$, $\alpha^{(k)}=\alpha$, $r_i^{(k)}=\exp\bigl[-\alpha^{(k)} y_i\hat{g}^{(k-1)}(x_i)\bigr] $
  3. Run weighted least squares $\sum_{i=1}^n w_i r_i (y_i - x'\beta)^2 \to \min_\beta$
  4. Iterate 2--4 a few times until the coefficients and/or the weights $r_i$ stabilize.

As an option to improve stability a little bit, you can devise a trajectory for $\alpha^{(k)}$, so that for the first say 10 iterations, $\alpha^{(k)}=k/10 \cdot \alpha$ for your selected ultimate $\alpha$, and then you stick $\alpha^{(k)}=\alpha$ for the remaining $k=11,12,\ldots$

Given how quick regression is, I won't be surprised to see that this converges faster than the full non-linear optimization. The standard errors, however, will be too small, as they don't account for uncertainty in $r_i$ (due to uncertainty in $\hat\beta$ that the weights $r_i$ utilize internally). You can get the right standard errors from the sandwich variance formula.

Finding the "right" value of $\alpha$ must come from the same (substantive) considerations which dictate the need for asymmetric treatment of the regression error. Given that you did not explain what these are, I am not sure how to advise on that.

  • $\begingroup$ Thanks, this is almost perfect. I just didnt think that the iterative procedure would converge.I thought to implement gradient descent on it , but the gradient computation itself is a little problematic. $\endgroup$
    – ssj3892414
    Commented Jan 30, 2015 at 3:40
  • $\begingroup$ I tried this and it doesnt seem to converge even on small examples. I'll also try gradient descent to see if that works $\endgroup$
    – ssj3892414
    Commented Jan 30, 2015 at 15:19
  • $\begingroup$ Oh, bummer. If something this simple does not converge, then may be your problem isn't that well defined, and has unstable solutions. Your $\exp()$ weight function may be too extreme, so that instead of a version of least squares, you are getting the minimization of the largest residual that has a wrong sign... which is related to what you want, but may not be quite what you want. $\endgroup$
    – StasK
    Commented Jan 30, 2015 at 15:54
  • $\begingroup$ On 3 different examples one quite realistic, i found that the $\beta$ tends to oscillate close to 2 different points and keeps jumping between them, likely somethign similar to what you said keeps happening. I tried to trim the exp number to a max value, and even then this happens $\endgroup$
    – ssj3892414
    Commented Jan 31, 2015 at 2:53
  • $\begingroup$ any ideas suggestions for how to redesign the loss $\endgroup$
    – ssj3892414
    Commented Feb 1, 2015 at 7:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.