Are linear classifiers (SVM, Logistic Regression) deterministic?

Question

I am just starting to learn about classification and have been playing around with some linear classifiers. I was wondering if linear classifiers are deterministic--given the same model parameters and training/testing data, should n runs of a classifier yield the same results on testing data? Any help would be much appreciated, thanks!

Answer 1

First, your definition of "deterministic" and "linear classifier" are not clear to me. For example, are you asking if the model building deterministic or model prediction deterministic? In addition, most people will think SVM is not a linear model but you treat it is linear.

I am trying to guess what you want to ask from now on.

Most models (not necessary to be "linear") are "deterministic" on prediction stage, and they should be. Intuitively we want that feeding the same input, we want to have the same output.

However, many models do have some randomness during when we build the model. This means that

• Given the same data, with different random seeds, you can have different models (see Random Forest as an example)

• After model building, during the prediction stage it is "deterministic", i.e., feeding same input will have same output.

Finally, in the "linear models" you mentioned logistic regression and SVM, they do not have a random seed during the training process. As mentioned in the other answers and comments, the reason is the objective function for logistic regression and SVN are convex, so we have the unique answer / global minima when we build the model.

Answer 2

I would think that any algorithm that can prove it reaches a global error minimum (linear/logistic regression, support vector machines) should stay the same, except for maybe a few trailing decimal places. Models that do not make this guarantee that could get stuck in a local minimum, like neural networks or random forests, will probably differ from training session to training session.