3
$\begingroup$

There is many variants of type of solver in liblinear but I don't understand their differences.Which one I must choose?

Also why data must be scaled? duo to some numerical issues?

-s type : set type of solver (default 1)
  for multi-class classification
   0 -- L2-regularized logistic regression (primal)
   1 -- L2-regularized L2-loss support vector classification (dual)
   2 -- L2-regularized L2-loss support vector classification (primal)
   3 -- L2-regularized L1-loss support vector classification (dual)
   4 -- support vector classification by Crammer and Singer
   5 -- L1-regularized L2-loss support vector classification
   6 -- L1-regularized logistic regression
   7 -- L2-regularized logistic regression (dual)
$\endgroup$
4
  • 2
    $\begingroup$ Kernel methods like SVM are based on the notion of distance. To use such methods all input dimensions should have comparable ranges. For instance, if dimension 1 has a range of millions, while dimension 2 has a range of 1, the model will be completely based on dimension 1. Scaling prevents these issues. $\endgroup$ Nov 7, 2014 at 9:04
  • $\begingroup$ @MarcClaesen, this is old, but liblinear is for linear models; no kernels involved, unless you're doing embeddings of some kind $\endgroup$
    – Danica
    Oct 1, 2015 at 5:00
  • $\begingroup$ @Dougal linear SVM still belongs to the category of kernel SVM, albeit with a trivial one. My comment still applies nonetheless. $\endgroup$ Oct 1, 2015 at 10:23
  • 3
    $\begingroup$ @MarcClaesen As I'm sure you know, if you scale data points $x$ into $x' = S^{-1} x$ (where $S_{ii} = \sigma_i$ is the std dev of the $i$th feature), then a linear model $\beta^T x + b$ is equivalent to $(\beta')^T x' + b = (S \beta)^T (S^{-1} x) + b = \beta^T x + b$. The only difference scaling makes in liblinear, therefore – outside of numerical and conditioning issues – is in the regularization. If $\sigma_i$ is huge, then $\lVert S \beta \rVert$ will penalize a given value of $\beta_i$ much more than $\lVert \beta \rVert$, and so the effect is actually (at least typically) to discount $i$. $\endgroup$
    – Danica
    Oct 1, 2015 at 10:39

2 Answers 2

1
$\begingroup$

Here is an article about L1 and L2 loss function

http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/

L1-norm loss function is also known as least absolute deviations (LAD), least absolute errors

L2-norm loss function is also known as least squares error (LSE).

Also, programm will solve faster if you scale your data properly, but it is not necessary when your data amount is very large.

Here is a guide for liblinear.

$\endgroup$
3
  • 4
    $\begingroup$ Scaling has no effect on training time. $\endgroup$ Nov 7, 2014 at 9:01
  • $\begingroup$ May be I'm wrong, but I saw the video of machine learning on coursera, and there's one section showing that scaling the data can help finding best solution faster. $\endgroup$ Nov 7, 2014 at 9:04
  • $\begingroup$ @MarcClaesen Depends on your solver. Scaling can act as a form of preconditioning if the solver doesn't do that explicitly. $\endgroup$
    – Danica
    Oct 1, 2015 at 5:01
0
$\begingroup$

It depends on what you want to do, the L1 and L2 loss are different ways of measuring the loss between the targeted value (label) and the predicted value, with L1 loss being called as least absolute errors and L2 loss being called as least square errors. The regularization is to prevent overfitting. Adding a regularization term can prevent the coefficients to fit so perfectly thus to overfit. The difference between the L1 and L2 is just that L2 is the sum of the square of the weights, while L1 is just the sum of the weights.

The data scaling is neccessary becuase you want to make sure one variable in your model doesn't overcontribute the importance to the result. E.g variable1 ranges from [2000-5000], variable2 ranges from [0.2-0.5], to predict the value between [0,1] without feature scaling, variable1 will contirbute more than variable 2 given the nature of the range.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.