0
$\begingroup$

In the context of machine learning, most examples of regression (linear, ridge, elasticnet) tend to split along the 50/50 probability line.

In the chart below, suppose that a red event is 10x more costly than a blue event. Because of that it might be desirable to incorrectly classify some more blue events (as red events) in the interest of correctly classifying more red events.

Is there an easy way to weight the regression in favor of one dataset over another? One might imagine the regression in that case to be closer to the red line than the dashed line.

... or is that the wrong approach entirely?

example

$\endgroup$

1 Answer 1

0
$\begingroup$

For a situation as simple as you've indicated here, with your visualization, weighting the data is not usually necessary. Instead, the output of whatever classification algorithm is used typically produces a continuous value, and you can simply choose the "cut point" to optimize your classifier for the frequencies of the classes that you expect.

Two examples are linear discriminant analysis and logistic regression. In LDA, a hyperplane $\vec \beta \cdot \vec x = \beta_0$ is drawn through the parameter space -- corresponding to the red and dashed lines that you have shown here. The canonical LDA procedure produces a value for $\beta_0$ (based on certain assumptions that aren't important here). However, you are completely free to choose $\beta_0$ yourself. All this parameter sets is the location of the hyperplane along its normal, e.g. whether to the pick the red line or the dashed line in your graphic. Similarly, in logistic regression, the regression produces an estimate for the log-odds of membership in one of the two classes. Given a uniform prior, the line where the estimated log-odds is zero corresponds to the line where an event has a 50-50 chance of membership in either class, e.g. $\hat \pi = 0.5$. However, you are perfectly free to assume a different, non-uniform prior. In practice this ends up being the same as saying that you can pick a separation value other than $\hat \pi = 0.5$.

The same is more-or-less true for other classification methods that produce a continuous output. You can try to construct an appropriate Bayesian prior, but its usually easier to just toggle your cut point on the output.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.