4

I've seen this question and answer: Why does Newton's method work?

It gives some geometric intuition as to what is going on when applying Newton's method, but what I really need to know is why it works that way.

How can you prove that each approximation is closer to the correct value than the previous one?

And also, if the following is indeed true, how can you prove that the "gap" between each two subsequent approximations gets smaller and smaller as you keep going up in the sequence of approximations?

Community
  • 1
jeremy radcliff
  • 4,933
  • 1
    If you start close enough to a non degenerate root $r$, you can show that the iterates satisfy $|x_{n+1}-r| \le C |x_n-r|^2$. Hence if you start with $C|x_0-r| <1$, then you have the desired conclusion. This is true in higher dimensions as well. It is not too surprising in that each iteration solves the linearised problem, so what is left over is second order. – copper.hat May 27 '16 at 19:13

1 Answers1

5

Proof that Newton's method converges is a standard and elementary result that can be found in any textbook on numerical analysis, as well as in any number of places online (including Wikipedia or Pete Clark's excellent notes).

The real interesting observation, though, is that Newton's method work a lot better in practice than is guaranteed by the theory. Many times Newton's method will converge, and converge quickly, even when starting far from the root, or when the function Hessian is replaced by a crude approximation (a property that quasi-Newton methods like BFGS take advantage of). Closing the gap between how well Newton-type methods work in practice, and what is proven theoretically, is an active research topic.

Community
  • 1
user7530
  • 50,625
  • Thanks a lot for the link to Pete's notes. About your second paragraph, is the idea that when you say "guaranteed by the theory" you're essentially talking about an upper bound on the rate of convergence? And if not, how is it that we don't know yet exactly why there is such a gap? – jeremy radcliff May 27 '16 at 19:41
  • 1
    @jeremyradcliff I believe what he meant was that the theory predicts a convergence that is at least as fast as blabla. But in practice, Newton's method converges much faster than blabla. (Which is allowed by the theory. But this makes us wonder why there's such a difference, and if there's a way to prove that the method converges this fast, maybe with some additional assumptions) – Kitegi May 27 '16 at 19:47
  • 1
    @jeremyradcliff To be a bit more precise than Farnight's comment: Newton's method's theory says if you use an exact derivative (rather than a numerical approximation) and start close enough to a non-degenerate root, then you get quadratic convergence. You essentially never get better than that except for linear problems. But you often get something pretty good when you use a numerical approximation to the derivative and/or start far away from the root. For example, Newton's method for calculating $\sqrt{x}$ is the iteration $y_{k+1}=\frac{y_k+x/y_k}{2}$. This converges whenever $x_0>0$. – Ian May 27 '16 at 19:49