Steepest descent method and how to find the weighting factor

Question

I am trying to understand the numerical method of finding a minimum, the steepest descent method. \begin{equation} x_1 = x_0 - \alpha \frac{df}{dx} \end{equation} I understand the idea behind it and how to set up the equation, but I do not get:

How to find the "right" weighting factor $\alpha$.
Why the next search has to occur orthogonal to the previous search direction.

Thanks in advance for the answers.

Your second question is not correct; there is no such requirement. (Are you thinking perhaps of conjugate gradients?) For your first, do a Google search for "backtracking line search" and go from there. — Michael Grant, Apr 04 '15 at 14:41
Actually, I think the search directions are orthogonal if $f$ is quadratic and you are doing an exact line search. — Michael Grant, Apr 04 '15 at 14:48
Yes, the equation is quadratic... however I dont know how that has an importance. I thought one could apply this to all "reasonable" functions. — Nenunathel, Apr 06 '15 at 15:03
You can, but the orthogonality claim is not valid except for quadratics. — Michael Grant, Apr 06 '15 at 15:03

Hans · Answer 1 · 2015-04-04T14:52:29.160

1

For the second part, you can see the book "Nonlinear Programming:Theory and Algorithms",written by Mokhtar S.Bazaraa and H.D.Sherali. 3rd edtion. Page389. It is zigzagging .

edited Apr 04 '15 at 14:52

answered Apr 04 '15 at 14:46

Hans

346

For step length, people usually don't search for the best step length, because it usually takes too much effort and is not necessary. We just look for a pretty good result under some conditions. For example, the Wolfe conditions and Goldstein conditions. You can find them in the book "Numerical Optimization", by Jorge Nocedal and another author. – Hans Apr 04 '15 at 14:51

Steepest descent method and how to find the weighting factor

1 Answers1