I run epsilon-svr on about 1000 samples (RBF kernel), and then calculate the in-sample training errors.
The hyperparameters are found via a search, ranging from 1e-5 to 1e2. model->eps is a couple orders of magnitude lesser than model->p.
From my understanding, the KKT conditions are:
1. sv_coef[i] == 0 <==> abs( predict(prob->x[i]) - y[i] ) <= model->param.p, 2. 0 < abs(sv_coef[i]) < C <==> abs( predict(prob->x[i]) - y[i] ) == model->param.p (or extremely close) 3. abs(sv_coef[i]) == C <==> abs( predict(prob->x[i]) - y[i] ) >= model->param.p 4. abs(sv_coef[i]) <= C
Conditions 1,2,3 break for me over many of the training vectors, e.g.:
Violated KKT conditions for RemainingSet: Weight/Cost: 0/700, label: 1.00288, predicted: 1.0025, Error/Eps: 0.000372128/0.0001, computed violation: 0.000272128
As printed, a non-support vector does not like inside the epsilon tube. The algorithm does not hit max iterations, so it claims that it has converged to the global optimum.
Does LibSVM output a broken system in epsilon-SVR mode? Is the error caused by numerical stability? How would one go about to stabilize an SVM with violated KKT conditions?