Why are SVMs hard to fit? I often hear the following complaint from people: "SVMs work really well WHEN they actually work." By "work" I mean that the algorithm will actually finish running. Are SVMs difficult to fit in practice versus other models, or are these people making a (I assume common) mistake while attempting to use SVMs?
 A: In the theory, to insure the convergence, your problem must be convex. Otherwise you can't know if the algorithm will find a solution or not.
Moreover SVMs' complexity are O(n^2) so not very scalable with large datasets.
A: I suspect the problem is to do with the hyper-parameter settings (the regularization parameter, $C$, and any kernel parameters).  You can end up with essentially all of the data being support vectors, at which point the optimisation procedure takes a long time because there are many "active" parameters.  For small to medium size problems (a few thousand training patterns) I tend to use least-squares support vector machines, as the fitting procedure only requires solving a single set of linear equations (or an eigen-decomposition of the kernel matrix if you want to tune the regularisation parameter as well).  
Getting good generalisation performance from an SVM depends on choosing a suitable kernel and careful tuning of the hyper-parameters (grid search minimising cross-validation error is a good method).
