0
$\begingroup$

I have a problem when I run Logistic Regression on a dataset I have modelled. Every time I run the algorithm, it always gives me the same prediction accuracy. Does that make sense? Any help would be much appreciated.

Thank you in advance!

$\endgroup$
  • 3
    $\begingroup$ Why would you expect the results to be different? $\endgroup$ – Sycorax Feb 6 '18 at 22:32
  • $\begingroup$ To be honest this is my first implementation of a Machine Learning model and it is my final year project for my bachelor's degree. I have seen many tutorials online and when they run the algorithms they get different percentage in their accuracy in each run. Does that mean the algorithms do not converge in these examples and that in my model it converges? My data set has around 300 data entries with 92 features. $\endgroup$ – Anestis S. Feb 6 '18 at 23:02
  • 4
    $\begingroup$ Algorithms that get different results each run are non-deterministic. Is logistic regression deterministic? Why or why not? $\endgroup$ – Sycorax Feb 6 '18 at 23:06
1
$\begingroup$

tl;dr There is no randomness in any step of this procedure so you should always get the same result.

Longer version

To add a little more explanation (although @Sycorax hints at the reason), let's just understand what's going on from a high level. A little debugging can go a long way. Since this is tagged with sci-kit, I'll assume you're familiar with sci-kit idioms.

I imagine you have a model (clf) fitted on a training set (X_train, y_train) and are predicting some testing set (X_test). Then you compare your predicted values (y_pred) against a target (y_test).

In this case, if we assume that the training and testing set are held constant every time you run your model, then the only things that can change are clf and y_pred. So instead of questioning why the accuracy remains unchanged, we can go upstream a bit and ask why the model and its output don't change.

First of all, the logistic regression equation looks like this:

$$P(Y_i=1 | \mathbf{X}_{i}) = \text{logit}^{-1}(\bf{\beta} \cdot \mathbf{X}_{i})$$

The only way that the output $P(Y_i=1 | \mathbf{X}_{i})$ (aka y_pred) can remain the same from run to run is if $\beta$ remains the same from run to run, because we know that $\mathbf{X}_{i}$ (aka X_test) doesn't change. There is no randomness in this equation.

Therefore we just need to understanding if $\beta$ will remain the same from run to run, given the same training input X_train, y_train. Note that clf is basically just a set of $\beta$'s.

To solve for $\beta$, note that we have to solve for the $\beta$ that minimizes the likelihood function, which is a joint probability distribution (which is easily solvable since we assume every observation is IID). This procedure, which is common in statistics and machine learning, is known as Maximum Likelihood Estimation. This is what happens when you run clf.fit(X_train, y_train) and get a set of coef_ ($\beta$'s).

Since there is no closed form solution for $\beta$, we need to find an algorithm to do the trick. For this type of problem (Logistic Regression is an example of a Generalized Linear Model) we use an algorithm called Iteratively Re-weighted Least Squares. There is a lot of content on this site related to this topic. The form of IRWLS is:

$${\displaystyle \mathbf {w} _{k+1}=\left(\mathbf {X} ^{T}\mathbf {S} _{k}\mathbf {X} \right)^{-1}\mathbf {X} ^{T}\left(\mathbf {S} _{k}\mathbf {X} \mathbf {w} _{k}+\mathbf {y} -\mathbf {\boldsymbol {\mu }} _{k}\right)}$$

Each of the terms on the right hand side are constant, that is, not random. For more information on what they stand for, see here. The algorithm stops when some tolerance has been achieved. This could be a small change in $\mathbf{w}$ or a small change in your target.

As you can see, there is no reason why this algorithm shouldn't converge to the same value every time (save perhaps for potential overflow errors on some runs).

And that's why you keep getting the same prediction accuracy.

Finally, convince yourself this is what's expected using sklearn's sample code:

from sklearn import datasets, neighbors, linear_model

digits = datasets.load_digits()
X_digits = digits.data
y_digits = digits.target

n_samples = len(X_digits)

X_train = X_digits[:int(.9 * n_samples)]
y_train = y_digits[:int(.9 * n_samples)]
X_test = X_digits[int(.9 * n_samples):]
y_test = y_digits[int(.9 * n_samples):]
logistic = linear_model.LogisticRegression()

for i in range(10):
    print('LogisticRegression score: %f' % logistic.fit(X_train, y_train).score(X_test, y_test))
$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.