Hot answers tagged

2

Is there an ExtraTreesClassifier-like classifier that has decision boundary function like SVM? The SVM boundary arises directly from how SVMs are defined. It's not something that can be "tacked on" to an arbitrary classifier because non-SVM classifiers are not defined in the same way. There are alternative ways for decision trees to make predictions, but I ...


2

It's not going to be ok since the kernel that needs to be present is quadratic in terms of number of samples. Of course, implementation details of the library do also matter. sklearn doc suggests kernel approximation in such large datasets.


2

Maximizing the margin is not just "rhetoric". It is the essential feature of support vector machines and ensures that the trained classifier has the optimal generalization properties. More precisely, making the margin large maximizes the probability that the classification error on new data will be small. The theory behind it is called the Vapnik-...


1

Most of the theory behind the support vector machine assumes you are constructing a maximal margin classifier following a fixed transformation of the input space (via a kernel). The theory is less applicable if the fixed transformation has been learned from the data (as would be the case for the lower levels of the deep neural network). This means that ...


1

I think there are two things you should potentially be careful about: one is overfitting and the other is tuning the RBF kernel. More specifically: The RBF kernel (typically) has one free parameter often called the bandwidth. In order to get good performance, this parameter needs to be tuned. Typically, one tries out a (large) grid of candidate values for ...


1

For a given, finite data set it should always be possible—just let each data point have its own dimension! So, maybe a more interesting question would be for a stochastic model, generating a data set, such that for $n$ realizations, linear separability would require a dimension growing linearly with $n$?


1

The slides are about Perceptron algorithm not SVM (although it's quoted maybe mistakenly). First equation is about normal perceptron, and the second is about the kernel version of it. They're equivalent. However, perceptron algorithm only give you a solution, not necessarily the max-margin solution. For having max-margin solution, you need to look at the ...


1

The support vectors are just the points that are used to define your hyperplane. These are basically the points of the two classes that are the closest together, and that are used to define the margins. From a visual perspective, your support vectors are the points that sit on your margins. Indeed, if you add training data to an SVM, any point that does not ...


1

I think the confusion comes from the definition of support vectors : a point $i$ is defined as a support vector precisely if $y_i(w^Tx_i+b) = 1$. The inequality $y_i(w^Tx_i+b) \geq 1$ must hold for all data points, but the support vectors are the data points for which this constraint is attained the least. Support vectors satisfy $y_i(w^Tx_i+b) = 1$... ...


1

The argument goes like this. 1) the value that maximizes $1/||w||$ is the value that minimizes $||w||$ 2) the value that minimizes $||w||$ also minimizes $||w||^2$, 3) which also minimizes $1/2 ||w||^2$


1

That particular slide is about Perceptron algorithm, where initially $w$ is $0$ and you update it for each misclassified sample with the following update rule (there are slightly changed versions of this, but sticking with the slides): $$w\leftarrow w+y_ix_i$$ Because we start from $0$ and every update made is in terms of $y_ix_i$, the final version of the ...


1

Is my understanding correct? Yes, in general I would say that it is, and that your classifier might be using some different parameters to work better with your unbalanced classes. However, AUC curve with an SVM might not be completely "truthful", and this brings me to the next point Is SVM using a threshold of 0.5? First of all, I suggest reading this ...


1

Wikipedia explains nonlinear SVM classification with The original maximum-margin hyperplane algorithm proposed by Vapnik in 1963 constructed a linear classifier. However, in 1992, Bernhard E. Boser, Isabelle M. Guyon and Vladimir N. Vapnik suggested a way to create nonlinear classifiers by applying the kernel trick (originally proposed by Aizerman et al.)...


1

Do you use python? do the hyperparameter tuning by RandmizedSearchCV or GridSearchCV: https://scikit-learn.org/stable/modules/grid_search.html As you're doing Ridge regression https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html, Should be very straight forward. RandmizedSearchCV or GridSearchCV learn different models given ...


1

All the calculations are the same, wether you are doing it sequentially or in one shot, so yes. Remember, if you want to asses the out of sample error you should keep out an untouched test set and evaluate the error on that. The error that you get on the validation set with the picked w will underestimate the true out of sample error, as this is was chosen ...


1

Over-fitting, most probably. Your model might fit this particular set of data very well, but it could be unlikely to generalize well to another sample of data from the same population. With 150 patients total and a 2:1 class ratio, you only have about 50 in the minority class. With biomedical data such as you have, an unpenalized model is likely to be ...


Only top voted, non community-wiki answers of a minimum length are eligible