I am working with SGDClassifier from Python library scikit-learn, a function which implements linear classification with a Stochastic Gradient Descent (SGD) algorithm. The function can be tuned to mimic a Support Vector Machine (SVM) by setting a hinge loss function 'hinge' and a L2 penalty function 'l2'.

My train data set consists on approximately $n=100,000$ samples with $159$ features (some features are categorical and have been binarized, some other features are quantitative and have been scaled to interval $[0,1]$). The data has two possible labels $\{0,1\}$, but the data set is highly unbalanced $-$ approximately only $4.5\%$ of samples are labelled $0$. Hence I have introduced class weights into my SGDClassifier, with the weight assigned to class $i$ corresponding to the fraction of the data labelled with $j$, $(i,j) \in \{0,1\}, i \not= j$ $-$ so class $0$ has weight $95.5\%$.

I also mention that the learning rate of the SGD algorithm is set to 'optimal'.

Now, here is my issue:

  • I wanted to test the convergence of the algorithm, so I set verbose equal to '10' to obtain the Avg. loss which the algorithm computes at each iteration $-$ for information, the algorithm shuffles the data then goes through all of it, repeating this operation $n_{iter}$ times, such that it makes $n \times n_{iter}$ computations; the Avg. loss is equal to the accumulated average loss and is given at the end of each iteration or epoch.
  • Then, I train my machine through a $10$-fold cross-validation procedure, such that for each cross-validation fold I have a final Avg. loss computed by SGDClassifier $-$ which as explained corresponds to an average of $n \times n_{iter}$ losses.
  • Simultaneously, for each cross-validation fold I compute the score (accuracy rate) of my classifier on the training data.

I have done this for $n_{iter}=10,100,500,1000$, I show results above:

Table of convergence

Manifestly, the algorithm converges with respect to the hinge loss, which is $\max[0,1-y\,f(\mathbf{x})]$ where $f$ corresponds to the trained machine, however the accuracy rate, which is $1_{\{sign(f(\mathbf{x}))=sign(y)\}}$, seems to have a rather erratic trajectory with respect to $n_{iter}$.

My question is: is this normal, or at least possible? Intuitively, I would say that it is mathematically possible to have these results, however I do not know if this is normal or if it indicates that something is going on with my code.

Has someone already experienced this phenomenon? Could someone shed more light on this?

P.S.: I know accuracy rates are poor, however this is due to the quality of the data.


The above computations have been undertaken for some value $\alpha^*$ of parameter alpha that I had already found to be optimal by cross-validation, setting $n_{iter}=100$. I wanted to test the sensibility of the training accuracy rate for $\alpha^*$ to the number of SGD's iterations $n_{iter}$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.