9
$\begingroup$

I've been toying around with logistic regression with various batch optimization algorithms (conjugate gradient, newton-raphson, and various quasinewton methods). One thing I've noticed is that sometimes, adding more data to a model can actually make training the model take much less time. Each iteration requires looking at more data points, but the total number of iterations required can drop significantly when adding more data. Of course, this only happens on certain data sets, and at some point adding more data will cause the optimization to slow back down.

Is this a well studied phenomenon? Where can I find more information about why/when this might happen?

Improve this question
$\endgroup$
5
  • 6
    $\begingroup$ This is an interesting observation. That the number of iterations can decrease with more data is intuitive: except with complete separation, having more data implies greater precision even in rough starting estimates of the solution. With fewer data a broader initial search, with small gradients, may need to occur. Analysis of the information matrix in a neighborhood of the true parameter values would make this intuition quantitative. $\endgroup$
    – whuber ♦
    Commented Apr 2, 2014 at 22:21
  • 5
    $\begingroup$ Besides the things @whuber mentions, adding data can make the likelihood surface "nicer", which means typical algorithms should converge much more quickly. In small samples convergence for GLMs may sometimes be slow because the surface isn't a nice, nearly-quadratic-in-the-parameters thing. As sample sizes get larger - especially if you have a canonical link function, so the likelihood is just a function of some simple sufficient statistics - it may be quicker not just in iterates, but possibly even in time. $\endgroup$
    – Glen_b
    Commented Apr 2, 2014 at 22:32
  • 1
    $\begingroup$ I understand the intuition that you both mention, but I'm curious if this can be quantified a bit more somehow. For example, maybe some experimental results showing how much speed improvement can possibly be gained by more data. $\endgroup$
    – Mike Izbicki
    Commented Apr 2, 2014 at 22:39
  • 1
    $\begingroup$ Lots of odd things can affect processing speed. See the most upvoted question on Stack Overflow for instance. $\endgroup$
    – Nick Stauner
    Commented May 13, 2014 at 20:54
  • $\begingroup$ Can you provide one case that show this? If you could make it "typical" for your experience, and show that how a subset of otherwise "healthy" data has slow convergence, but the set of data itself has faster convergence, that might help with a better answer. I think that I just paraphrased Mike Izbicki. $\endgroup$
    – EngrStudent
    Commented May 13, 2014 at 21:21

1 Answer 1

Reset to default
4
$\begingroup$

With less amounts of data, spurious correlation between regression inputs is often high, since you only have so much data. When regression variables are correlated, the likelihood surface is relatively flat, and it becomes harder for an optimizer, especially one that doesn't use the full Hessian (e.g. Newton Raphson), to find the minimum.

There are some nice graphs here and more explanation, with how various algorithms perform against data with different amounts of correlation, here: http://fa.bianp.net/blog/2013/numerical-optimizers-for-logistic-regression/

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.